Audio content generation and classification

Abstract

Some disclosed methods involve receiving audio data of at least a first audio data type and a second audio data type including an audio signal and associated spatial data, determining at least a first feature type from the audio data, and applying a positional encoding process to the audio data to generate encoded audio data. The encoded audio data may include a representation of at least the spatial data and the first feature type in a first embedding vector of an embedding dimension. Some methods may involve training a neural network based on the encoded audio data to transform the audio data from an input audio data type having an input spatial data type to a transformed audio data type having a transformed spatial data type. Some methods may involve training a neural network to identify the input audio data type.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the following priority applications: U.S. Provisional Application No. 63 / 277,217, filed November 9, 2021 (Docket No. D21107USP1), and U.S. Provisional Application No. 63 / 374,702, filed September 6, 2022 (Docket No. D21107USP2), all of which are incorporated by reference in their entireties.

[0002] Technical Field The present disclosure relates to devices, systems, and methods for generating new audio content based on existing audio content, and devices, systems, and methods for classifying audio content. [Background technology]

[0003] Although existing methods of classifying audio content can provide adequate results in some contexts, more advanced methods are desirable.

[0004] Notation and Nomenclature Throughout this disclosure, including the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used interchangeably to refer to any sound emitting transducer (or set of transducers) driven by a single speaker feed. A typical headphone set includes two speakers. A speaker may be implemented to include multiple transducers (e.g., woofers and tweeters) that may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker signal may undergo different processing in different circuit branches coupled to different transducers.

[0005] Throughout this disclosure, including the claims, the phrase performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to a signal or data) is used broadly to refer to performing an operation on the signal or data directly, or on a processed version of the signal or data (e.g., on a version of the signal that has been subjected to pre-filtering or pre-processing before performing the operation).

[0006] Throughout this disclosure, including the claims, the term "system" is used broadly to denote a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0007] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0008] Throughout this disclosure, including the claims, the terms "couple" or "coupled" are used to mean either a direct connection or an indirect connection. Thus, if a first device couples to a second device, the connection may be through a direct connection or through an indirect connection via other devices and connections.

[0009] As used herein, a "smart device" is generally an electronic device that is configured to communicate with one or more other devices (or networks) via various wireless protocols, such as Bluetooth, Zigbee, near field communications, Wi-Fi, light fidelity (Li-Fi), 3G, 4G, 5G, etc., and can operate interactively and / or autonomously to some degree. Some notable types of smart devices are smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smart watches, smart bands, smart keychains, and smart audio devices. The term "smart device" may also refer to devices that exhibit some characteristics of ubiquitous computing, such as artificial intelligence.

[0010] The term "smart audio device" is used herein to denote a smart device that is either a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of a virtual assistant function). A single-purpose audio device is a device (e.g., a television (TV)) that includes or is coupled to at least one microphone (and, optionally, includes or is coupled to at least one speaker and / or at least one camera) and is designed primarily or primarily to achieve a single purpose. For example, a TV can typically play (and is considered capable of playing) audio from program material, but in most cases, modern TVs run some operating system on which applications run locally, including applications for watching television. In this sense, a single-purpose audio device having a speaker and microphone is often configured to run local applications and / or services to directly use the speaker and microphone. Several single-purpose audio devices may be configured to be grouped together to achieve playback of audio over a zone or user-configured area.

[0011] One common type of multipurpose audio device is a smart audio device, such as a "smart speaker," that implements at least some aspects of the virtual assistant functionality, although other aspects of the virtual assistant functionality may be implemented by one or more other devices, such as one or more servers with which the multipurpose audio device is configured to communicate. Such multipurpose audio devices may be referred to herein as "virtual assistants." A virtual assistant is a device (e.g., a smart speaker or a voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide the ability to utilize multiple devices (different from the virtual assistant) for applications that are in some sense cloud-enabled or otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality, e.g., speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant can communicate over a network, such as the Internet. Virtual assistants may sometimes work together, for example in a discrete, conditionally defined manner. For example, two or more virtual assistants may cooperate in the sense that one of them, for example the one that is most confident that it heard the wake word, responds to the wake word. Connected virtual assistants may, in some implementations, form a kind of constellation, which may be managed by one main application that may be (or may implement) a virtual assistant.

[0012] Here, "wake word" is used broadly to mean any sound (e.g., a word spoken by a human being, or any other sound) that the smart audio device is configured to wake up in response to detecting ("listening to") that sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "awakening" refers to the device entering a state in which it waits for (i.e., listens for) a voice command. In some instances, what may be referred to herein as a "wake word" may include multiple words, e.g., a phrase.

[0013] Here, the term "wake word detector" refers to a device (or software including instructions for configuring a device) configured to continuously look for alignment between real-time audio (e.g., speech) features and a trained model. Typically, a wake word event is triggered whenever the wake word detector determines that the probability that the wake word has been detected exceeds a predetermined threshold. For example, the threshold may be a predetermined threshold tuned to provide a reasonable compromise between false accept and false reject rates. Following the wake word event, the device may enter a state (which may be referred to as an "awake" or "attentive" state) in which it listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0014] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and possibly video signals, that are intended to be listened to at least in part together. Examples include selections of music, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from a smart assistant, etc. In some instances, a content stream may contain multiple versions of at least a portion of an audio signal, e.g., the same dialogue in multiple languages. In such instances, only one version of the audio data or portion thereof is intended to be played at a time (e.g., the version corresponding to a single language). Summary of the Invention [Means for solving the problem]

[0015] At least some aspects of the present disclosure may be implemented via one or more audio processing methods. In some examples, the methods may be implemented, at least in part, by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some such methods involve receiving, by the control system, first audio data of a first audio data type including one or more first audio signals and associated first spatial data. The first spatial data may indicate an intended perceived spatial location of the one or more first audio signals. In some examples, the methods may involve determining, by the control system, at least a first feature type from the first audio data. The methods may involve applying, by the control system, a positional encoding process to the first audio data to generate first encoded audio data. In some examples, the first encoded audio data may include a representation of at least the first spatial data and the first feature type in a first embedding vector of an embedding dimension.

[0016] In some examples, the method may involve receiving, by the control system, second audio data of a second audio data type including one or more second audio signals and associated second spatial data. According to some examples, the second audio data type may be different from the first audio data type. In some examples, the second spatial data may indicate an intended perceived spatial location for the one or more second audio signals. In some examples, the method may involve determining, by the control system, at least a first feature type from the second audio data.

[0017] According to some examples, the method may involve applying, by the control system, a positional encoding process to the second audio data to generate second encoded audio data. In some examples, the second encoded audio data may include a representation of at least the second spatial data and the first feature type in a second embedding vector of an embedding dimension.

[0018] In some examples, the method may involve training a neural network implemented by the control system to convert audio data from an input audio data type having an input spatial data type to a converted audio data type having a converted spatial data type. In some examples, the training may be based at least in part on the first encoded audio data and the second encoded audio data.

[0019] According to some examples, the method may involve receiving first through Nth audio data of first through Nth input audio data types including first through Nth audio signals and associated first through Nth spatial data, where N is an integer greater than 2. In some such examples, the method may involve determining, by a control system, at least the first feature type from the first through Nth input audio data types, applying, by the control system, a positional encoding process to the first through Nth audio data to generate the first through Nth encoded audio data, and training a neural network based at least in part on the first through Nth encoded audio data.

[0020] In some examples, the neural network may be or include an attention-based neural network, hi some examples, the neural network may include a multi-head attention module.

[0021] According to some examples, training the neural network may involve training the neural network to transform the first audio data into a first region of a latent space and to transform the second audio data into a second region of the latent space. In some examples, the second region may be at least partially separate from the first region.

[0022] In some examples, the intended perceived spatial location corresponds to at least one of channel or location metadata of a channel-based audio format.

[0023] According to some examples, the input spatial data type may correspond to a first audio data format and the converted audio data type may correspond to a second audio data format. In some examples, the input spatial data type may correspond to a first number of channels and the converted audio data type may correspond to a second number of channels. In some examples, the first feature type may correspond to a frequency domain representation of the audio data.

[0024] In some examples, the method may involve determining, by the control system, at least a second feature type from the first audio data and the second audio data. In some such examples, the positional encoding process may involve representing the second feature type in an embedding dimension.

[0025] According to some examples, a method may involve receiving, by a control system, audio data of an input audio data type. In some such examples, a method may involve converting the audio data of the input audio data type to a converted audio data type.

[0026] Some alternative methods may involve receiving, by a control system, audio data of an input audio data type having an input spatial data type, and converting, by the control system, the audio data of the input audio data type to audio data of a converted audio data type having a converted spatial data type. According to some examples, the converting may involve implementing, by the control system, a neural network trained to convert the audio data from the input audio data type to the converted audio data type. In some such examples, the neural network may be trained based at least in part on encoded audio data resulting from a position encoding process. The encoded audio data may include a representation of at least first spatial data and a first feature type in a first embedding vector of an embedding dimension. The first spatial data may indicate an intended perceived spatial location for the reproduced audio signal. According to some examples, the input spatial data type may correspond to a first audio data format and the converted audio data type may correspond to a second audio data format.

[0027] Some alternative methods may involve receiving, by a control system, first audio data of a first audio data type including one or more first audio signals and associated first spatial data. The first spatial data may indicate an intended perceived spatial location for the one or more first audio signals. In some examples, the method may involve determining, by the control system, at least a first feature type from the first audio data. According to some examples, the method may involve applying, by the control system, a position encoding process to the first audio data to generate first encoded audio data. In some examples, the first encoded audio data may include a representation of at least the first spatial data and the first feature type in a first embedding vector of an embedding dimension.

[0028] According to some examples, the method may involve receiving, by a control system, second audio data of a second audio data type including one or more second audio signals and associated second spatial data. The second audio data type may, in some examples, be different than the first audio data type. In some examples, the second spatial data may indicate an intended perceived spatial location for the one or more second audio signals.

[0029] In some examples, the method may involve determining, by a control system, at least the first feature type from the second audio data. According to some examples, the method may involve applying, by the control system, a positional encoding process to the second audio data to generate second encoded audio data. In some examples, the second encoded audio data may include a representation of at least the second spatial data and the first feature type in a second embedding vector of an embedding dimension. In some examples, the method may involve training a neural network implemented by the control system to identify an input audio data type of the input audio data. According to some examples, the training may be based at least in part on the first encoded audio data and the second encoded audio data.

[0030] In some examples, identifying the input audio data type may involve identifying a content type of the input audio data. According to some examples, identifying the input audio data type may involve determining whether the input audio data may correspond to a podcast, dialogue from a movie or television program, or music. In some examples, the method may involve training a neural network to generate new content of a selected content type.

[0031] Some or all of the operations, functions, and / or methods described herein may be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0032] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices (e.g., a system including one or more devices) may be capable of at least partially performing the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to implement some or all of the methods disclosed herein.

[0033] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following figures may not be drawn to scale. [Brief description of the drawings]

[0034] Like reference numbers and designations in the various drawings indicate like elements. [Figure 1]1 illustrates example blocks that may be involved in training a neural network, according to some disclosed implementations. [Figure 2A] 1 shows, according to some examples, examples of blocks that may be involved in training a neural network. [Figure 2B] FIG. 1 is a block diagram illustrating example components of an apparatus capable of implementing various aspects of the disclosure. [Figure 3A] 1 illustrates a representation of a position encoding matrix according to an example. [Figure 3B] An example of mapping loudspeaker positions onto a unit circle is shown. [Figure 4] 13 illustrates another example representation of a position encoding matrix. [Diagram 5] 1 illustrates a block diagram for generating new audio content according to an example. [Figure 6] FIG. 1 is a flow diagram outlining an example of the disclosed method. [Figure 7] FIG. 11 is a flow diagram outlining another example of the disclosed method. [Figure 8] FIG. 1 is a flow diagram outlining an example of an additional disclosed method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0035] Generalizations can be made about how audio content was created, and conventions are commonly followed among audio content of the same class. For example, in Dolby 5.1 and 7.1 channel audio content corresponding to a movie or television program, the main dialogue utterances are typically positioned in the front center speaker, and the momentary special effects are typically positioned in the rear. Similarly, in Dolby 5.1 and 7.1 channel audio content corresponding to music, the audio corresponding to the lead singer is generally positioned in the front center speaker.

[0036] In the context of multichannel audio content, complex signal relationships need to be modeled and understood. As used herein, the term "multichannel audio content" can refer to any audio with multiple channels, for example, single channel dialogue and podcasts, stereo music and movie content, multichannel movie soundtracks and dialogue, and multichannel music content. As used herein, the term "multichannel audio content" is intended to apply to object-based audio formats such as Dolby Atmos™, even though such audio formats are not literally channel-based.

[0037] This disclosure describes methods, devices, and systems for modeling complex multi-dimensional relationships between channels in audio content. Some disclosed examples involve identifying latent space variables corresponding to a body of audio content that arise after reinforcement learning and describe the principal variance within the body of audio content.

[0038] In some disclosed examples, the capabilities of neural networks are leveraged to identify such latent spatial variables. Some disclosed examples are configured to identify an audio content type based on the nature of these latent variables. Alternatively or additionally, some disclosed examples are configured for optional generation of new audio content based on the nature of these latent variables. The new audio content may represent a desired audio content class, such as, for example, movie dialogue, podcast, music, etc. Some such examples may involve identifying a transformation function that may be applied to convert audio data from having an input spatial data type into audio data having a transformed spatial data type. According to some such examples, the input spatial data type may include a smaller number of channels than the transformed spatial data type. Some examples disclose novel methods of positional encoding for training neural networks.

[0039] FIG. 1 illustrates an example of blocks that may be involved in training a neural network, according to some disclosed implementations. As with other figures provided herein, the types, numbers, and arrangements of elements shown in FIG. 1 are provided merely as examples. Other implementations may include more, fewer, and / or different types, numbers, and / or arrangements of elements. In this example, the blocks of FIG. 1 are implemented by a control system 160, an example of which is described in more detail with reference to FIG. 2B.

[0040] According to this example, the elements of FIG.

[0041] 101: A training content database including audio data for training a neural network. In some examples, the training content database may include multiple classes of audio content. For example, one class of audio data may correspond to movie dialogue, another class of audio data may correspond to podcasts, another class of audio data may correspond to music, etc. In some examples, the training content database 101 may include audio data in multiple audio formats for one or more classes of audio content. The audio formats may include standard audio formats such as, in some examples, Dolby 5.1 format, Dolby 6.1 format, Dolby 7.1 format, Dolby 9.1 format, Dolby Atmos™ format, etc. As used herein, an audio format referred to as the X.1 format generally refers to X.1, X.2, etc. For example, a reference to the Dolby 9.1 format also applies to the Dolby 9.2 format unless otherwise noted. According to the example shown in Figure 1, the audio data in the training content database 101 includes or represents an audio signal and corresponding spatial data. The spatial data indicates an intended perceived spatial location for the audio signal. The intended perceived spatial location may correspond to a channel in a channel-based audio format or position metadata in an object-based audio format.

[0042] 102A: Digital samples of audio data from the training content database 101 after transformation from time domain to frequency domain.

[0043] 102B: A label identifying a class of audio content, such as movie dialogue, podcast, music, etc., corresponding to digital sample 102A.

[0044] 103: an input transformation block configured to extract features from digital samples of audio data 102A and generate input feature data 123, where the input feature data 123 includes one or more (and in some examples multiple) types of audio features representing a short (in time) context window. A short context window in time may be a time interval less than 1 second, such as, for example, a 100 ms time interval, a 200 ms time interval, a 300 ms time interval, a 400 ms time interval, a 500 ms time interval, a 600 ms time interval, a 700 ms time interval, a 800 ms time interval, etc. In some examples, the input feature data 123 may be represented as X(T,C,B), where T represents a time frame, C represents a content channel, and B represents a discrete Fourier bin.

[0045] 104: A position encoding signal generator block configured to output position encoding information 124. The position encoding information 124 may correspond to a position encoding matrix in some examples.

[0046] 105: Encoder Neural Network configured to convert Encoder Network Input Data 129 into encoded audio data 113. The encoded audio data 113 may be or include a latent space vector of a multi-dimensional latent space. Thus, in some instances, the encoded audio data 113 may be referred to as a latent space vector 113. The multi-dimensional latent space may also be referred to herein as an encoding space or encoding dimension. In some examples, the encoder neural network 105 may be or include a multi-head attention neural network.

[0047] 107: A statistical model of the latent space vectors 113 for all audio content in the training content database 101.

[0048] 108: A decoder neural network configured to decode the encoded audio data 113. In this example, decoding the encoded audio data 113 involves converting the latent space vectors 113 from latent space coordinates to “real world” feature data 109. In this example, the output feature data 109 includes the same types of features as the input feature data 123.

[0049] 109: Feature data output by the decoder neural network 108 and provided to the loss function block 121. In this example, the loss function data 142 is based at least in part on a comparison of the input feature data 123 and the output feature data 109.

[0050] 110: Graph showing a representation of the statistical model 107 of the latent space variables in two dimensions. In this example, the graph 110 is a two-dimensional representation of a high-dimensional latent space.

[0051] 111: Two-dimensional representation of audio content class A with an example.

[0052] 112: Two-dimensional representation of audio content class B with an example.

[0053] 113A: Example of a latent space variable, also represented as Y in Figure 1.

[0054] 113B: The position of the latent space variable 113A in the two-dimensional space of the graph 110.

[0055] 114A: Sampled points from within the representation of content class A in statistical model 107.

[0056] 115: A Fourier transform block configured to transform the audio data in the content database 101 from the time domain to the frequency domain. In some examples, the Fourier transform block 115 also applies an absolute operation to provide only the absolute value of the input data. However, in some alternative examples, the Fourier transform block 115 may provide complex Fourier information.

[0057] 120: An input dimension transformation block configured to map features 123 to an encoding dimension D of higher dimensional spatial features 125. The encoding dimension D may also be referred to herein as hidden dimension D. In this example, the number of dimensions of higher dimensional spatial features 125 (in other words, the number of dimensions of D) should match the number of dimensions of position encoding information 124 output by position encoding signal generator block 104.

[0058] 121: A loss function block configured to output loss function data 142 that may be used as feedback for training the input dimension transformation block, the position encoding signal generator block 104, the encoder network 105, and / or the decoder network 108. The loss function data 142 may include, for example, prediction and target inputs, as well as weight updates.

[0059] 123: Features output by the input transformation block 103.

[0060] 124: Position encoding information 124 output by the position encoding signal generation block 104.

[0061] 125: High-dimensional spatial features output by the dimensional transformation block 120.

[0062] 127: Encoder network. A summing block configured to combine the high-dimensional spatial features 125 with the position encoding information 124 to generate input data 129.

[0063] 129: Encoder network input data generated by summation block 127 and provided to encoder network 105.

[0064] The training data stored in the training content database 101 should include a wide range of content, ideally "all digitized content in existence," but realistically what is accessible, complies with legal licensing requirements, and can be fed to the neural network model and trained in a reasonable length of time. The number of audio channels in the training data should be diverse, and where appropriate, data in higher channel surround sound formats should also be presented downmixed in lower channel formats to best illustrate the variance in the data across a range of surround sound channel formats. In some examples, the data may be generated in a guided manner, for example by using empirical estimates of typical variances that the content may illustrate, such as by synthesizing multiple pieces of content in a surround sound format domain, such as Dolby Atmos™, where the audio channels are represented as objects that can occupy any position in space.

[0065] In some examples, noise may be added to the data in order to obfuscate some of the input data from the network during training and perturb the optimization to find a solution that can best fulfill the desired task. Similarly, in some examples, data may sometimes be omitted, particularly (a) in the channel dimension (in which case data may be omitted entirely on selected channels) or (b) in time, in small time segments. The latter may encourage the network to best model the data.

[0066] The neural network may be trained by offline training (eg, before deployment by an end user), online training (eg, during deployment by an end user), or a combination of both offline and online training.

[0067] The loss function block 121 may implement one or more cost functions used to train the neural network. In some examples, the cost function may be selected by a human, such as a system designer. The cost function may be selected in some examples in a manner that attempts to cause the encoder-decoder network to best find a globally optimal fit for the input and output training data in an unsupervised neural network optimization setting. The definition of "globally optimal" depends on the application and may be selected, for example, by the system designer. In some examples, the cost function(s) may be selected to optimize for one or more of the following: Generate application-specific maximum distances between distributions of class embedding / latent space vectors in order to best use said vectors in classification and clustering tasks. The task is to minimize noise or discontinuities in the latent space vectors being sampled from a distribution function to generate novel audio data that satisfies certain properties about the distribution of the latent space being sampled. Generate meaningful, subjectively desirable new content from existing data in the audio channel that the network is tasked with interpolating or generating. Provide a globally optimal fit of the underlying function in the latent space that can appropriately interpolate between data, e.g. content formats, to transform the identified data from one content class to another via a linear transformation in the latent space. Any combination of the above.

[0068] According to some examples, the loss function block 121 may implement one or more cost functions based at least in part on an L1 norm calculated as the sum of the absolute values ​​of the vectors, an L2 norm calculated as the square root of the sum of the squares of the vector values, or a combination thereof. The L1 and L2 loss functions may be used, for example, to perturb the model in training to converge to a solution that best matches the decoder output with the encoder input audio. The use of rudimentary loss functions is to support unsupervised training procedures. According to some such examples, there may be no need for more complex loss functions due to the aforementioned additive noise, as well as masking in both channel and time.

[0069] In addition to using one or more loss functions, in some examples, the neural network training procedure may be perturbed in the latent space, for example, by quantizing, Gaussian fitting and sampling, or otherwise interpolating and modifying the latent vectors, to generate a latent space that is best suited for the task of audio data generation or audio data classification.

[0070] FIG. 2A illustrates example blocks that may be involved in training a neural network, according to some examples. As with other figures provided herein, the types and number of elements shown in FIG. 2A are provided merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements. In this example, the blocks of FIG. 2A are implemented by a control system 160, an example of which is described in detail with reference to FIG. 2B.

[0071] Figure 2A shows an example in which the encoder neural network 105 of Figure 1 is a multi-head attention neural network with N heads. According to this example, the elements of Figure 2A that have not yet been described with reference to Figure 1 are as follows:

[0072] 201: Input dimension coordinates describing the discrete units of the real world from which the input features 103 are sampled.

[0073] 202: a query (Q) transformation block, which is configured to apply first through N different linear transformations (which may be learned linear transformations) to the encoder network input data 129 generated by the summation block 127 to generate queries 212a through 212n corresponding to the N heads of the multi-head attention process.

[0074] 203: Key (K) transformation block, which is configured to apply first through Nth different linear transformations (which may be learned linear transformations) to the encoder network input data 129 to generate keys 213a through 213n.

[0075] 204: A value (V) transformation block, which is configured to apply first through N different linear transformations (which may be learned linear transformations) to the encoder network input data 129 to generate values ​​214a through 214n.

[0076] 205a-205n: Scaled dot product modules, one for each of the N heads of a multi-head attention process. Each module is configured to implement a multi-head attention process in which an attention function is performed in parallel on queries 212a-212n, keys 213a-213n, and values ​​214a-214n to generate output values ​​222a-222n. Related scaled dot product attention processes and multi-head attention processes are described in "Scaled Dot Product Attention Processes," in "Computer-Based Multi-Head ... [Non-Patent Document 1] A. Vaswani et al, “Attention Is All You Need”, 31st Conference on Neural Information Processing Systems (NIPS 2017), especially Section 3, p.2-6

[0077] 206: Multiple (N) transducer heads shown as a parallel network.

[0078] 207: A concatenation block configured to combine the scaled dot product output values ​​222a-222n of each head.

[0079] 208: A final linear transformation block configured to project and combine the output of each of the heads to generate encoded audio data 113, sometimes referred to herein as latent space vectors 113.

[0080] FIG. 2B is a block diagram illustrating examples of components of a device capable of implementing various aspects of the disclosure. As with other figures provided herein, the types and numbers of elements shown in FIG. 2B are provided merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements. According to some examples, device 250 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 250 may be or include one or more components of an audio system. For example, device 250 may be an audio device, such as a smart audio device, in some implementations. In other examples, device 250 may be a mobile device (such as a mobile phone), a laptop computer, a tablet device, a television, or another type of device.

[0081] According to some alternative implementations, apparatus 250 may be or include a server. In some such examples, apparatus 250 may be or include an encoder. Thus, in some cases, apparatus 250 may be a device configured for use in an audio environment, such as a home audio environment, and in other cases, apparatus 250 may be a device configured for use in the "cloud," e.g., a server.

[0082] In this example, the device 250 includes an interface system 255 and a control system 160. The interface system 255, in some implementations, may be configured to communicate with one or more other devices of the audio environment. The audio environment may, in some examples, be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. The interface system 255, in some implementations, may be configured to exchange control information and associated data with audio devices of the audio environment. The control information and associated data, in some examples, may relate to one or more software applications that the device 250 is executing.

[0083] The interface system 255 may be configured to receive or provide a content stream in some implementations. The content stream may include audio data. The audio data may include, but is not limited to, an audio signal. In some instances, the audio data may include spatial data, such as channel data and / or spatial metadata. The metadata may have been provided, for example, by what may be referred to herein as an “encoder.” In some instances, the content stream may include video data and audio data corresponding to the video data.

[0084] Interface system 255 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some implementations, interface system 255 may include one or more wireless interfaces. Interface system 255 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. Thus, although some such devices are depicted separately in FIG. 2B, such devices may correspond to aspects of interface system 255 in some examples.

[0085] In some examples, interface system 255 may include one or more interfaces between control system 160 and a memory system, such as optional memory system 265 shown in FIG. 2B. However, control system 160 may include a memory system in some cases. Interface system 255, in some implementations, may be configured to receive input from one or more microphones in the environment.

[0086] The control system 160 may include, for example, a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, logical gate or transistor logic, and / or discrete hardware components.

[0087] In some implementations, the control system 160 may reside in more than one device. For example, in some implementations, a portion of the control system 160 may reside in a device in one of the environments illustrated herein, and another portion of the control system 160 may reside in a device that is external to the environment, such as a server, a mobile device (e.g., a smartphone or tablet computer), etc. In other examples, a portion of the control system 160 may reside in a device in one of the environments illustrated herein, and another portion of the control system 160 may reside in one or more other devices of the environment. For example, the control system functionality may be distributed across multiple smart audio devices of the environment, or may be shared by a leader device (such as what may be referred to herein as a smart home hub) and one or more other devices of the environment. In other examples, a portion of the control system 160 may reside in a device implementing a cloud-based service, such as a server, and another portion of the control system 160 may reside in another device implementing a cloud-based service, such as another server, a memory device, etc. The interface system 255 may also reside in more than one device, in some examples.

[0088] In some implementations, the control system 160 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 160 may be configured to receive audio data of at least a first audio data type and a second audio data type, the audio data including an audio signal and associated spatial data indicative of an intended perceived spatial location for the audio signal. In some such examples, the control system 160 may be configured to determine at least a first feature type from the audio data and apply a positional encoding process to the audio data to generate encoded audio data. The encoded audio data may include a representation of at least the spatial data and the first feature type in a first embedding vector of an embedding dimension. The spatial data may be considered, in some examples, as another type of feature in addition to the "first feature type."

[0089] According to some examples, the control system 160 may be configured to implement a trained or untrained neural network. In some such examples, the neural network may be or may include an attention-based neural network.

[0090] Some disclosed methods may involve training a neural network implemented by control system 160 based at least in part on the encoded audio data to convert audio data from an input audio data type having an input spatial data type to a converted audio data type having a converted spatial data type. For example, the trained neural network implemented by control system 160 may be configured to convert a movie soundtrack from a Dolby 5.1 format to a Dolby 7.1 format even if there is no existing version of the movie soundtrack in the Dolby 7.1 format. In some such instances, there may not have been a higher channel version of the movie soundtrack than the Dolby 5.1 version.

[0091] Some disclosed methods may involve training a neural network implemented by control system 160 to identify an input audio data type. For example, the trained neural network implemented by control system 160 may be configured to determine whether the input audio data corresponds to music, a podcast, a movie soundtrack, a television program soundtrack, etc.

[0092] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored in one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, and the like. The one or more non-transitory media may be present, for example, in optional memory system 265 and / or control system 160 shown in FIG. 2B. Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. The software may include, for example, instructions for controlling at least one device to perform some or all of the methods disclosed herein. The software may be executable by one or more components of a control system, such as, for example, control system 160 of FIG. 2B.

[0093] In some implementations, the device 250 may include an optional sensor system 270 as shown in FIG. 2B. The optional sensor system 270 may include one or more touch sensors, gesture sensors, motion detectors, and the like. According to some implementations, the optional sensor system 270 may include one or more cameras. In some implementations, the camera may be a freestanding camera. In some examples, the one or more cameras of the optional sensor system 270 may be present in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, the one or more cameras of the optional sensor system 270 may be present in a television, a mobile phone, or a smart speaker. In some examples, the device 250 may not include the sensor system 270. However, in some such implementations, the device 250 may still be configured to receive sensor data for one or more sensors in the audio environment via the interface system 160.

[0094] In some examples, optional sensor system 270 includes an optional microphone system. The optional microphone system may include one or more microphones. According to some examples, the optional microphone system may include an array of microphones. In some examples, the array of microphones may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, e.g., according to instructions from control system 160. The array of microphones may be configured for receiver beamforming, in some cases, e.g., according to instructions from control system 160. In some implementations, one or more of the microphones may be part of or associated with another device, such as a loudspeaker of loudspeaker system 275, a smart audio device, etc.

[0095] In some examples, device 250 may not include a microphone system. However, in some such implementations, device 250 may still be configured to receive microphone data for one or more microphones in the audio environment via interface system 160. In some such implementations, a cloud-based implementation of device 250 may be configured to receive microphone data, or data corresponding to the microphone data, from one or more microphones in the audio environment via interface system 160.

[0096] According to some implementations, the device 250 may include an optional loudspeaker system 275, shown in FIG. 2B. The optional loudspeaker system 275 may include one or more loudspeakers, which may also be referred to herein as a "speaker" or, more generally, an "audio reproduction transducer." In some examples (e.g., cloud-based implementations), the device 250 may not include the loudspeaker system 275.

[0097] In some implementations, the device 250 may include an optional display system 280 as shown in FIG. 2B. The optional display system 280 may include one or more displays, such as one or more light emitting diode (LED) displays. In some examples, the optional display system 280 may include one or more organic light emitting diode (OLED) displays. In some examples, the optional display system 280 may include one or more displays of a smart audio device. In other examples, the optional display system 280 may include a television display, a laptop display, a mobile device display, or another type of display. In some examples in which the device 250 includes a display system 280, the sensor system 280 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of the display system 280. According to some such implementations, the control system 160 may be configured to control the display system 280 to present one or more graphical user interfaces (GUIs).

[0098] According to some such examples, device 250 may be or include a smart audio device, such as a smart speaker. In some such implementations, device 250 may be or include a wake word detector. For example, device 250 may be configured to implement (at least in part) a virtual assistant.

[0099] 3A shows a representation of a position encoding matrix according to an example. This example shows the relationship between multi-channel format regions 302A and 302B of position encoding matrix 301 and physical presentation formats 303A and 303B, respectively. According to this example, the elements of FIG. 3A are as follows:

[0100] 301: Example of a three-dimensional representation of a four-dimensional positional encoding matrix. In this example, the positional encoding matrix 301 is represented as PE(T,C,B,D), where T represents a time frame, C represents an audio content channel, B represents a discrete Fourier bin, D represents an encoding dimension not shown in the three-dimensional representation of FIG. 3A, and PE represents positional encoding. An example of the D dimension of a four-dimensional positional encoding matrix is ​​shown in FIG. 4 and described below.

[0101] 302A: A multi-channel format region of the position encoding matrix 301 corresponding to audio data in the Dolby 5.1 audio format. 302B: A multi-channel format region of the position encoding matrix 301 corresponding to audio data in the Dolby 7.1 audio format. 303A: Representation of loudspeaker positions corresponding to the Dolby 5.1 audio format. 303B: Representation of loudspeaker positions corresponding to the Dolby 7.1 audio format.

[0102] FIG. 3B shows an example of mapping loudspeaker positions onto a unit circle. In this example, FIG. 3B shows a mapping of loudspeaker positions corresponding to the Dolby 7.1 audio format onto a unit circle 304. Instead of Cartesian or (x,y) coordinate positions as suggested by the representation of loudspeaker positions 303B in FIG. 3A, in FIG. 3B the loudspeaker positions corresponding to the Dolby 7.1 audio format are shown as angular coordinates, which in this example correspond to values ​​of the angle theta (Θ). Although only loudspeaker positions corresponding to the Dolby 7.1 audio format are shown in FIG. 3B, loudspeaker positions corresponding to other audio formats may also be represented as angular coordinates in the same manner.

[0103] As described elsewhere herein, some disclosed examples involve receiving, by a control system, audio data of a first audio data type including one or more first audio signals and associated first spatial data. The first spatial data may indicate an intended perceived spatial location for the one or more first audio signals. Some disclosed examples involve receiving, by a control system, audio data of a second audio data type including one or more second audio signals and associated second spatial data. The first spatial data may indicate an intended perceived spatial location for the one or more first audio signals, and the second spatial data may indicate an intended perceived spatial location for the one or more second audio signals. Some disclosed examples involve receiving, by a control system, first through Nth audio data of first through Nth input audio data types including first through Nth audio signals and associated first through Nth spatial data, where N is an integer greater than 2.

[0104] The adjectives "first," "second," and "Nth" in the context of audio data types, one or more audio signals, and associated spatial data are used merely to distinguish one audio data type from another and do not necessarily denote a temporal sequence. In other words, audio data of the first audio data type does not necessarily be received prior to the time that audio data of the second audio data type or audio data of the Nth audio data type is received.

[0105] The Dolby 5.1 audio format depicted in FIG. 3A is an example of the "first audio data type" (or second audio data type or Nth audio data type) discussed above. If the first audio data type is the Dolby 5.1 audio format, the "first spatial data" may be the intended perceived spatial positions (loudspeaker positions) shown in representation 303A of FIG. 3A, a corresponding angular representation, or any other suitable representation. Similarly, if the second audio data type is the Dolby 7.1 audio format, the "second spatial data" may be the loudspeaker positions shown in representation 303B of FIG. 3A, a corresponding angular representation shown in FIG. 3B, or any other suitable representation.

[0106] Some disclosed examples may involve determining, by a control system, at least a first feature type from the input audio data. In some examples, the control system may be configured to determine two or more feature types from the input audio data. The feature type may be or may include one or more types of frequency domain representations of the audio samples, such as, for example, a representation of energy or power in each of a plurality of frequency bands. As noted elsewhere herein, spatial data may be considered a type of feature in some examples.

[0107] Some disclosed examples involve applying, by a control system, a positional encoding process to audio data to generate encoded audio data. The encoded audio data may, in some examples, include a representation of at least first spatial data and a first feature type(s) in a first embedding vector of an embedding dimension. Some examples are shown in FIG. 3A and FIG. 3B. In the example of FIG. 3A, the positional encoding process involves applying a positional encoding process to audio data in a Dolby 5.1 audio format to generate a multi-channel format region 302A of the positional encoding matrix 301. According to the example of FIG. 3A, the positional encoding process involves applying a positional encoding process to audio data in a Dolby 7.1 audio format to generate a multi-channel format region 302B of the positional encoding matrix 301. The "real world" loudspeaker positions of the audio data in the Dolby 5.1 and 7.1 audio formats may initially be represented as shown in representations 303A and 303B of FIG. 3A (e.g., expressed via (x,y) coordinates), expressed via angular coordinates as shown in FIG. 3B, or expressed in any other convenient manner.

[0108] Another example representation of the position encoding matrix and vector is shown in Figure 4. In this example, the elements in Figure 4 are as follows:

[0109] 401: Another example of a 3D representation of a 4D positional encoding matrix. In this example, the positional encoding matrix 301 is represented as PE(T,C,B,D), where T represents a time frame not shown in the 3D representation of FIG. 4, C represents an audio content channel, B represents a discrete Fourier bin, and D represents the encoding dimension. 402: Example of a position encoding vector in the encoding dimension D of the position encoding matrix 401. 402A: A portion of vector 402 showing an example of a transformed input feature, in this case transformed via function f. 402B: A portion of vector 402 showing an example of transformed spatial data, in this case transformed via function g. 404: An example of a position encoding matrix transformation function f with respect to index i (up to N) and coordinates in dimension B. 405: An example of a position encoding matrix transformation function g with respect to index j (up to M) and coordinates in dimension C.

[0110] In the example shown in Figure 4, each vector in the encoding dimension D is a concatenation of contributions from each of the input feature dimensions, in this case channel C and bin B. The portion of each vector in dimension D that each dimension contributes (in this example, portions 404 and 405) may or may not be equal. In other words, N and M are not necessarily equal, and in many cases they are not equal. The size of N and M may vary depending on how many samples each feature dimension contains and how much resolution is desired across the feature dimension coordinates.

[0111] The actual number of bins (B) may not be N or (i+N). N may be less than, equal to, or greater than the number of bins B. Making N smaller than the number of bins may reduce computational complexity. However, making N larger than the number of bins B for a particular data set may be useful if there are other data sets with a larger number of bins. For example, there may be input audio data with known coordinates corresponding to the bins and other input audio data with different coordinates corresponding to the bins. For example, in one case there may be 512 bins between 0 and 24 kilohertz and in another case there may be 256 bins between 0 and 22.5 kilohertz. Making N larger than the number of bins B for any data set allows input data with different numbers of bins to be mapped to appropriate locations in the encoding dimension D.

[0112] Similarly, for the dimension C, M may not correspond to the actual number of channels in the dataset being converted. Typically, M is larger than C. Making M larger than the number of channels C for a particular dataset may be useful (or necessary) if there are other data formats with a larger number of channels. In the example shown in FIG. 3B, M corresponds to the granularity on the unit circle mapping after conversion to the encoding dimension D. Therefore, it may be advantageous to use a large value of M compared to the actual number of channels in a given audio format to ensure that the trained neural network is able to create new content corresponding to any channel, even if it is a channel not represented in any input dataset used during the training process.

[0113] When parameters for the positional encoding matrix are created, it may be beneficial to ensure that there is a unique distance metric that results when the function used to determine the vector portion of the encoding dimension D is applied, so that bins that were close to each other before the transformation to the embedding dimension remain close to each other after the transformation, and bins that were farther apart before the transformation to the embedding dimension remain farther apart after the transformation. The choice of N relative to B may depend, for example, on whether there are vectors with very similar but slightly different bin spacings. A function that is desirable to choose is one that can transform B into the encoding dimension D and ensures that there is a unique solution for, for example, a 48 kilohertz sample rate example that may have a bin at 1 kilohertz, and another sample rate example (e.g., a 44.1 kilohertz sample rate example) that may not have a bin at 1 kilohertz.

[0114] The valid position encoding matrix can map discrete points of input coordinates to vectors in the embedding dimension that can represent a set of discrete coordinates that may be important during the training process. An example of a valid position encoding matrix is ​​one that transforms audio data corresponding to different audio content classes into distinguishably different regions of the embedding space, such as that shown in graph 110 of FIG.

[0115] In some examples, the disclosed position encoding matrices accept the discrete coordinates underlying the content data 102 of FIG. 1. There are multiple examples of reasons to improve upon the previously disclosed coordinate-free encoding matrices, such as in Equations 1, 2, and 3 below. First, for a control system prepared for a range of spatial audio channel formats, the modifications and addition of coordinates to the previously disclosed position encoding formulations serve two purposes. First, to provide a means to assign physical location meanings to the audio channels, and second, to allow for degrees of freedom in the ordering of channels in the input data 103 and the intended physical presentation locations (spatial formats). This is to best encourage the control system under training to converge the latent space such that the presentation of a set of content with different spatial formats, or arbitrary spatial locations and channel numbers such as in Atmos, does not perturb the latent space. It can be seen below that in Equation 3, the spatial format location θ of audio channel c is accepted in the position encoding formulation. A continuous function that transforms spatial format channel locations into encodings that can be compared across a continuous range of locations may be desirable in some cases. In others, selecting a discrete set of arbitrary encoding vectors that evidences a maximum distance measure across a discrete set of spatial format locations may be used instead. Importantly, the disclosed encoding formulations may be implemented in a variety of ways, as long as the encodings accept spatial coordinates and provide tractable and uniform results in the distance measures and in the encoded distance comparisons.

[0116] The second reason for the disclosed position encoding method can be seen in Equations 1 and 2, where time t is normalized by the maximum time T and frequency bin b is normalized by the maximum frequency bin B used in the position encoding formulation. In applications, there are multiple discrete Fourier transform formulation possibilities, such that the update rate at t and the frequency interval at b can vary. This may be due to implementation constraints. Here, a pre-trained control system may be expected to be robust to these conditions, and the acceptance of coordinates in these dimensions may provide a unique variance in the sampling of data during training, which allows the final solution to which the network converges to generalize better and find a more representative latent space.

[0117] This disclosure includes other examples of applying a positional encoding process to audio data to generate encoded audio data that includes a representation of spatial data and one or more other feature types in an embedding vector of an embedding dimension. In the following examples, there are three input feature types, each of which corresponds to a feature dimension. In these examples, 1 / 3 of the encoding dimension D is dedicated to each of the three input feature dimensions. In these examples, the following Equations 1, 2, and 3 are each used to convert the input feature types into portions of the embedding dimension vector:

number

[0118] In the above equation, w (omega) represents the contribution of the alternating sin and cos encodings to the equation for the hidden dimension D. This is constructed to give a unique value for each d and is scaled by f to allow a numerically significant difference (closer to 1.0 than 0 or ± infinity) in the final value across the dimension d of the positional encoding matrix PE. The important, underlying goal is to make distinctions that indicate physical distance, for example, that PE(t = 1, c = 1, b = 1, 1 < d < D) is close to PE(t = 2, c = 1, b = 1, 1 < d < D) in a Euclidean sense, but within the features in any of the dimensions c, b, and t, it is ensured that it is far from PE(t = T, c = 1, b = 1, 1 < d < D) across their respective d encoding dimensions. In the above equation, f represents any hyperparameter, which is selected to be numerically useful to avoid encoding a dimension D where any range of the PE matrix equation degenerates to less than measurable precision for the computer. A wide range of numbers can be appropriate values for f.

[0119] In these examples, as before, the positional encoding matrix PE may be expressed as a function of T, C, B, and D. Here, T represents the time frame, C represents the audio content channel, B represents the discrete Fourier bin, and D represents the encoding dimension. In equations 1 - 3, t represents the time instance or index of the time frame T, c represents the audio content channel index, b represents the discrete frequency domain feature, for example, the Fourier bin or Mel band (the index corresponding to B) among B frequency domain bins, and d represents the index of the encoding dimension D. According to some examples, f may be many times larger than D.

[0120] Equation 1 and Equation 2 may be normalized in some examples, e.g., to maintain similar properties for discrete time sampling intervals before performing a Discrete Fourier Transform (DFT) and for DFT bin frequencies. This may be advantageous when different data sets have different sample rates. In some cases, different features may be used for those data sets, potentially corresponding to different feature extraction processes. Being able to specify the sample rate in Equation 1 and Equation 2 may be beneficial in such examples.

[0121] For example, Equation 1 specifies the sign of the t index divided by the total T. The capital T represents the number of time instances in the time dimension T. In one example, suppose there are 100 time instances t in the time dimension T. One may choose to use data sampled at twice that sampling rate, and in such an example one would want to take into account the fact that T is not an index, but rather an absolute time that one has decided to normalize to. If, for example, these 100 samples represent 1 second of data, then it would be advantageous to have a constant of 1 second somewhere in Equation 1 so that Equation 1 is agnostic to the actual time sampling frequency of the input data.

[0122] FIG. 5 illustrates blocks for generating new audio content according to an example. According to this example, the blocks of FIG. 5 are implemented by an instance of the control system 160 of FIG. 2B. In this example, the control system 160 is configured to convert audio data from an input audio data type having an input spatial data type to audio data of a converted audio data type having a converted spatial data type. According to this example, the input spatial data type includes fewer channels than the converted spatial data type. In some examples, the input spatial data type may correspond to a Dolby 5.1 audio format, and the converted spatial data type may correspond to a Dolby 6.1 audio format, a Dolby 7.1 audio format, a Dolby 9.1 audio format, a Dolby Atmos™ format, or the like.

[0123] 501: An encoder neural network configured to transform input audio data from a feature space into a lower dimensional ("bottlenecked") latent space.

[0124] 502: A latent space vector representing a point inside area 507, also denoted as “y” in FIG.

[0125] 503: A previously derived transformation h (e.g., by training a neural network as disclosed herein) for generating new audio content of a desired type, sometimes referred to as a transformed space data type. In this example, h is configured to transform latent space vectors corresponding to audio data of the input space data type into latent space vectors corresponding to audio data of the transformed space data type.

[0126] 504: The transformed latent space vector representing the point in area 508, also shown as “ŷ” in FIG. 5 .

[0127] 505: A decoder neural network configured to transform vectors in latent space to feature space.

[0128] 506: A two-dimensional representation of the latent space, which will generally have more than two dimensions.

[0129] 507: Sampling of latent space vectors corresponding to audio data of input spatial data type.

[0130] 508: Sampling of latent space vectors corresponding to audio data in the transformed spatial data type, which in this example contains more channels than the audio data in the input spatial data type.

[0131] Aspects of the underlying model formed by training a neural network that generally describes the differences between lower channel count content and higher channel count content can be understood when the model is implemented to transform latent space vectors. For example, if the network is trained on multiple examples of the same type of audio content (e.g., podcasts, music, TV show dialogue, movie dialogue, etc.) with different spatial data types (e.g., lower channel count versions and higher channel count versions), the neural network can derive one or more models in the latent space of what it means for a particular type of audio content to exist in some spatial data type.

[0132] By sampling the latent vectors for each spatial data type of the content, the neural network can derive a linear or nonlinear mapping of audio content from one spatial data type to another spatial data type in the multi-dimensional latent vector space. For a database of similar audio content (e.g., movie soundtracks) with different spatial data types, the neural network can be trained to derive a general mapping h (503). The general mapping can then be used to convert latent space vectors for audio data having one spatial data type (input spatial data type) to audio data having another spatial data type (transformed spatial data type) with a higher channel count format. The conversion is by generating new samples of the content in the higher channel count format. In some cases, the audio data in the transformed spatial data type may not have existed before in that particular spatial data type. For example, the audio data may have previously been present only in a lower channel count version of the input audio data type, such as the Dolby 5.1 format, but may not have previously been present in the Dolby 7.1 format or any higher channel count format.

[0133] 6 is a flow diagram outlining an example of the disclosed method. The blocks of method 600, as well as other methods described herein, are not necessarily performed in the order shown. Additionally, such methods may include more or fewer blocks than shown and / or described.

[0134] Method 600 may be performed by an apparatus or system, such as apparatus 250, one or more components thereof, shown in FIGURE 2B and described above. In some such examples, apparatus 250 configured to perform method 600 may include at least control system 160 shown in FIGURES 1 and 2 and described above.

[0135] In this example, block 605 involves receiving, by the control system, first audio data of a first audio data type including one or more first audio signals and associated first spatial data, where the first spatial data indicates an intended perceived spatial location of the one or more first audio signals. According to some examples, the intended perceived spatial location may correspond to a channel of a channel-based audio format, and in other examples, the intended perceived spatial location may correspond to and / or be indicated by position metadata of an audio object-based audio format, such as Dolby Atmos™. In some examples, the first audio data type may correspond to an audio format, such as a Dolby 5.1 audio format, a Dolby 6.1 audio format, a Dolby 7.1 audio format, a Dolby 9.1 audio format, a Dolby Atmos™ audio format, or the like. The first spatial data may be or include information corresponding to loudspeaker locations corresponding to the audio format of the first audio data type. Some relevant examples of spatial data are disclosed herein, such as in the discussion of Figures 3A and 3B.

[0136] According to this example, block 610 involves determining, by the control system, at least a first feature type from the first audio data, which may be or include one or more frequency domain representations of the audio sample, such as, for example, a representation of energy or power in each of a plurality of frequency bands.

[0137] In this example, block 615 involves applying, by the control system, a positional encoding process to the first audio data to generate first encoded audio data. According to this example, the first encoded audio data includes at least first spatial data and a representation of a first feature type in a first embedding vector of an embedding dimension. Several relevant examples of applying a positional encoding process are disclosed herein, including the discussion of Figures 3A, 3B, and 4, as well as the discussion of Equation 1, Equation 2, and Equation 3.

[0138] According to this example, block 620 involves receiving, by the control system, second audio data of a second audio data type including one or more second audio signals and associated second spatial data. In this example, the second audio data type is different from the first audio data type. For example, if the first audio data type corresponds to a Dolby 5.1 audio format, the second audio data type may correspond to another audio format, such as a Dolby 6.1 audio format, a Dolby 7.1 audio format, a Dolby 9.1 audio format, a Dolby Atmos™ audio format, etc. Here, the second spatial data indicates an intended perceived spatial location of the one or more second audio signals. In this example, block 625 involves determining, by the control system, at least a first feature type from the second audio data.

[0139] In this example, block 630 involves applying, by the control system, a positional encoding process to the second audio data to generate second encoded audio data, in this instance including at least the second spatial data and a representation of the first feature type in a second embedding vector of an embedding dimension.

[0140] According to this example, block 635 involves training a neural network implemented by the control system to convert audio data from an input audio data type having an input spatial data type to a converted audio data type having a converted spatial data type. In some examples, the input spatial data type may correspond to a first audio data format and the converted audio data type may correspond to a second (and different) audio data format. For example, the input spatial data type may correspond to a first number of channels and the converted audio data type may correspond to a second number of channels. In this example, the training is based at least in part on the first encoded audio data and the second encoded audio data. Various examples of suitable neural network training are disclosed herein, such as in the description of Figures 1 and 2A.

[0141] In some examples, training the neural network may involve training the neural network to transform the first audio data into a first region of a latent space and to transform the second audio data into a second region of the latent space. According to some examples, the second region may be at least partially separate from the first region, for example, as shown in FIGs. 1 and 5. In some examples, the neural network may be or include an attention-based neural network. According to some such examples, the neural network may include a multi-head attention module.

[0142] In some examples, method 600 may involve receiving, by a control system, first through Nth audio data of first through Nth input audio data types including first through Nth audio signals and associated first through Nth spatial data, where N is an integer greater than 2. In some such examples, method 600 may involve determining, by the control system, at least a first feature type from the first through Nth input audio data types. According to some such examples, method 600 may involve applying, by the control system, a positional encoding process to the first through Nth audio data to generate the first through Nth encoded audio data. In some such examples, method 600 may involve training a neural network based at least in part on the first through Nth encoded audio data.

[0143] According to some examples, method 600 may involve determining, by the control system, at least a second feature type from the first audio data and the second audio data, and in some such examples, the positional encoding process may involve representing the second feature type in an embedding dimension.

[0144] In some examples, method 600 may involve receiving, by a control system, audio data of an input audio data type. In some such examples, method 600 may involve converting, by the control system, the audio data of the input audio data type to audio data of a convert audio data type.

[0145] FIG. 7 is a flow diagram outlining an example of another disclosed method. The blocks of method 700, as well as other methods described herein, are not necessarily performed in the order shown. Moreover, such methods may include more or fewer blocks than shown and / or described. Method 700 may be performed by an apparatus or system, such as apparatus 250, one or more components thereof, shown in FIG. 2B and described above. In some such examples, apparatus 250 configured to perform method 700 includes at least control system 160, shown in FIG. 2B and described above.

[0146] According to this example, method 700 involves implementing a neural network trained to convert audio data from an input audio data type to a converted audio data type. In some examples, the neural network may have been trained as described herein, for example, with reference to FIG. 1, FIG. 2A, or FIG. 6.

[0147] In this example, block 705 involves receiving, by the control system, audio data of an input audio data type having an input spatial data type, which may correspond to an audio format such as, for example, a Dolby 5.1 audio format, a Dolby 6.1 audio format, etc.

[0148] According to this example, block 710 involves converting, by the control system, audio data of an input audio data type to audio data of a transformed audio data type having a transformed spatial data type. In this example, converting involves implementing, by the control system, a neural network trained to convert audio data from said input audio data type to said transformed audio data type. According to this example, the neural network has been trained based at least in part on encoded audio data resulting from a position encoding process, where the encoded audio data included at least first spatial data and a representation of a first feature type in a first embedding vector of an embedding dimension. The first spatial data indicates an intended perceived spatial location for the reproduced audio signal.

[0149] In some examples, the input spatial data type may correspond to a first audio data format and the transform audio data type may correspond to a second audio data format. For example, if the input spatial data type corresponds to a Dolby 5.1 audio format, the transform audio data type may correspond to a Dolby 6.1 audio format, a Dolby 7.1 audio format, a Dolby 9.1 audio format, a Dolby Atmos format, etc.

[0150] In some examples, the neural network may be trained to determine a transformation h configured to transform latent space vectors corresponding to audio data of the input spatial data type into latent space vectors corresponding to audio data of the transformed spatial data type, e.g., as described with reference to FIG. 5.

[0151] 8 is a flow diagram outlining an example of an additional disclosed method. The blocks of method 800, as well as other methods described herein, may not necessarily be performed in the order shown. Additionally, such methods may include more or fewer blocks than shown and / or described.

[0152] Method 800 may be performed by an apparatus or system, such as apparatus 250, one or more components thereof, shown in FIGURE 2B and described above. In some such examples, apparatus 250 configured to perform method 800 includes at least control system 160, shown in FIGURE 2B and described above.

[0153] In this example, blocks 805-830 are similar to blocks 605-630 described in detail with reference to Figure 6. Therefore, the description of blocks 805-830 will not be repeated here.

[0154] However, according to this example, block 835 involves training a neural network implemented by the control system to identify an input audio data type of the input audio data. In this example, the training is based at least in part on the first encoded audio data and the second encoded audio data. Various examples of suitable neural network training are disclosed herein, such as those described in Figures 1 and 2A.

[0155] In some examples, identifying the input audio data type may involve identifying a content type of the input audio data. According to some such examples, identifying the content type may involve determining whether the input audio data corresponds to a podcast, dialogue in a movie or television show, or music. In some such examples, identifying the content type may involve determining a region of a multidimensional latent space that corresponds to a particular content type, e.g., as shown in FIG. 1.

[0156] Some aspects of the disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor that is programmed and / or configured with software or firmware to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to asserted data.

[0157] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform the necessary processing on the audio signal(s), including performing one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include input devices and memory) that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the inventive system are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, the system also including other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and / or a keyboard), a memory, and a display device.

[0158] Another aspect of the disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code (e.g., executable code) for performing one or more examples of the disclosed methods or steps thereof.

[0159] While specific embodiments of and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many variations on the embodiments and applications described herein are possible without departing from the scope of the present disclosure as described and claimed herein. Although certain forms of the present disclosure have been illustrated and described, it will be understood that the present disclosure should not be limited to the specific embodiments described and illustrated, or to the specific manner described.

Claims

1. receiving, by a control system, first audio data of a first audio data type including one or more first audio signals and associated first spatial data, the first spatial data indicating an intended perceived spatial location for the one or more first audio signals; determining, by the control system, at least a first feature type from the first audio data; applying, by the control system, a positional encoding process to the first audio data to generate first encoded audio data, the first encoded audio data including a representation of at least the first spatial data and the first feature type in a first embedding vector of an embedding dimension; receiving, by the control system, second audio data of a second audio data type including one or more second audio signals and associated second spatial data, the second audio data type being different from the first audio data type, the second spatial data indicating an intended perceived spatial location for the one or more second audio signals; determining, by the control system, at least the first feature type from the second audio data; applying, by the control system, the position encoding process to the second audio data to generate second encoded audio data, the second encoded audio data including a representation of at least the second spatial data and the first feature type in a second embedding vector of the embedding dimension; training a neural network implemented by the control system to convert audio data from an input audio data type having an input spatial data type to a transformed audio data type having a transformed spatial data type, the training being based at least in part on the first encoded audio data and the second encoded audio data. method.

2. The method comprises: receiving first through Nth audio data of first through Nth input audio data types including first through Nth audio signals and associated first through Nth spatial data, where N is an integer greater than 2; determining, by the control system, at least the first feature type from the first through Nth input audio data types; applying, by the control system, the position encoding process to the first through Nth audio data to generate first through Nth encoded audio data; training the neural network based at least in part on the first through Nth encoded audio data. The method of claim 1.

3. The method of claim 1 or 2, wherein the neural network is or includes an attention-based neural network.

4. The method of claim 1 , wherein the neural network includes a multi-head attention module.

5. 5. The method of claim 1, wherein training the neural network involves training the neural network to transform the first audio data into a first region of a latent space and to transform the second audio data into a second region of the latent space, the second region being at least partially distinct from the first region.

6. The method of claim 1 , wherein the intended perceived spatial location corresponds to at least one of channel or location metadata of a channel-based audio format.

7. 7. The method of claim 1, wherein the input spatial data type corresponds to a first audio data format and the converted audio data type corresponds to a second audio data format.

8. 8. The method of claim 1, wherein the input spatial data type corresponds to a first number of channels and the transformed audio data type corresponds to a second number of channels.

9. The method of claim 1 , wherein the first feature type corresponds to a frequency domain representation of audio data.

10. 10. The method of claim 1, further comprising determining, by the control system, at least a second feature type from the first audio data and the second audio data, and wherein the positional encoding process involves representing the second feature type in the embedding dimension.

11. receiving, by the control system, audio data of the input audio data type; and converting the audio data of the input audio data type to the converted audio data type.

11. The method according to any one of claims 1 to 10.

12. A neural network trained according to the method of any one of claims 1 to 11.

13. One or more non-transitory media having stored thereon software including instructions for implementing the neural network of claim 12.

14. receiving, by a control system, audio data of an input audio data type having an input spatial data type; and converting, by the control system, the audio data of the input audio data type into audio data of a transform audio data type having a transform space data type, the converting involves implementing, by the control system, a neural network trained to convert audio data from the input audio data type to the converted audio data type, the neural network being trained based at least in part on encoded audio data resulting from a position encoding process, the encoded audio data including a representation of at least first spatial data and a first feature type in a first embedding vector of an embedding dimension, the first spatial data indicating an intended perceived spatial location for a reproduced audio signal; method.

15. 15. The method of claim 14, wherein the input spatial data type corresponds to a first audio data format and the converted audio data type corresponds to a second audio data format.

16. receiving, by a control system, first audio data of a first audio data type including one or more first audio signals and associated first spatial data, the first spatial data indicating an intended perceived spatial location for the one or more first audio signals; determining, by the control system, at least a first feature type from the first audio data; applying, by the control system, a positional encoding process to the first audio data to generate first encoded audio data, the first encoded audio data including a representation of at least the first spatial data and the first feature type in a first embedding vector of an embedding dimension; receiving, by the control system, second audio data of a second audio data type including one or more second audio signals and associated second spatial data, the second audio data type being different from the first audio data type, and the second spatial data indicating an intended perceived spatial location for the one or more second audio signals; determining, by the control system, at least the first feature type from the second audio data; applying, by the control system, the position encoding process to the second audio data to generate second encoded audio data, the second encoded audio data including a representation of at least the second spatial data and the first feature type in a second embedding vector of the embedding dimension; training a neural network implemented by the control system to identify an input audio data type of input audio data, the training being based at least in part on the first encoded audio data and the second encoded audio data. method.

17. 17. The method of claim 16, wherein identifying the input audio data type involves identifying a content type of the input audio data.

18. 18. The method of claim 16 or 17, wherein identifying the input audio data type involves determining whether the input audio data corresponds to a podcast, dialogue in a movie or television program, or music.

19. 19. The method of any one of claims 16 to 18, further comprising training the neural network to generate new content of a selected content type.

20. Apparatus configured to carry out the method of any one of claims 1 to 19.

21. A system configured to carry out the method of any one of claims 1 to 19.

22. 20. One or more non-transitory media having software stored thereon, the software comprising instructions for controlling one or more devices to perform the method of any one of claims 1 to 19.

Images