Audio signal enhancement

Abstract

An apparatus includes a processor configured to perform signal enhancement of an input audio signal to generate an enhanced monaural audio signal. The processor is further configured to mix the first audio signal and the second audio signal to generate a stereo audio signal. The first audio signal is based on the enhanced monaural audio signal.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority from commonly owned U.S. non-provisional patent application No. 17 / 814,783, filed on July 25, 2022, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] Generally speaking, the present disclosure relates to audio signal enhancement. Background Art

[0004] Technological advances have led to smaller and more powerful computing devices. For example, there are currently various portable personal computing devices, including small, lightweight and easily carried by users of wireless phones (such as mobile and smart phones, tablet devices and laptop computers). These devices can transmit voice and data packets on wireless networks. In addition, many such devices incorporate additional functions, such as digital cameras, digital video cameras, digital recorders and audio file players. In addition, such devices can process executable instructions, including software applications (such as web browser applications) that can be used to access the Internet. Therefore, these devices may include critical computing capabilities.

[0005] Such computing devices typically incorporate functionality for processing audio signals. For example, an audio signal may represent sound captured by one or more microphones, or correspond to decoded audio data. Such devices may perform signal enhancement (such as noise suppression) to generate an enhanced audio signal. Signal enhancement (e.g., noise suppression) may remove context from the enhanced audio signal and introduce artifacts that reduce audio quality. Summary of the invention

[0006] According to an implementation of the present disclosure, a device includes a processor configured to perform signal enhancement on an input audio signal to generate an enhanced mono audio signal. The processor is further configured to mix a first audio signal and a second audio signal to generate a stereo audio signal. The first audio signal is based on the enhanced mono audio signal.

[0007] According to another implementation of the present disclosure, a method includes: performing signal enhancement on an input audio signal at a device to generate an enhanced mono audio signal. The method also includes: mixing a first audio signal and a second audio signal at the device to generate a stereo audio signal. The first audio signal is based on the enhanced mono audio signal.

[0008] According to another implementation of the present disclosure, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: perform signal enhancement on an input audio signal to generate an enhanced mono audio signal. The instructions, when executed by one or more processors, further cause the one or more processors to perform the following operations: mix a first audio signal and a second audio signal to generate a stereo audio signal. The first audio signal is based on the enhanced mono audio signal.

[0009] According to another implementation of the present disclosure, an apparatus includes: a unit for performing signal enhancement on an input audio signal to generate an enhanced mono audio signal. The apparatus also includes: a unit for mixing a first audio signal and a second audio signal to generate a stereo audio signal. The first audio signal is based on the enhanced mono audio signal.

[0010] Other aspects, advantages, and features of the present disclosure will become apparent after reading the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and Claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A is a block diagram of certain illustrative aspects of a system operable to perform audio signal enhancement according to some examples of the present disclosure.

[0012] Figure 1B According to some examples of the present disclosure Figure 1A A schematic diagram of an illustrative implementation of a signal enhancer of a system operable to perform noise suppression.

[0013] Figure 1C According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of a signal enhancer of a system operable to perform audio scaling.

[0014] Figure 1D According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of a signal booster of a system operable to perform beamforming.

[0015] Figure 1E According to some examples of the present disclosure Figure 1A A schematic diagram of an illustrative implementation of a signal enhancer of a system that is operable to perform dereverberation.

[0016] Figure 1F According to some examples of the present disclosure Figure 1AA schematic diagram of another illustrative implementation of a signal enhancer of a system operable to perform source separation.

[0017] Figure 1G According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of a signal enhancer of a system that is operable to perform bass adjustment.

[0018] Figure 1H According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of a signal enhancer of a system that is operable to perform equalization.

[0019] Figure 2A According to some examples of the present disclosure Figure 1A A schematic diagram of an illustrative implementation of an audio mixer for a system.

[0020] Figure 2B According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0021] Figure 2C According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0022] Figure 3A According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0023] Figure 3B According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0024] Figure 3C According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0025] Figure 4A According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0026] Figure 4B According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0027] Figure 4C According to some examples of the present disclosure Figure 1AA schematic diagram of another illustrative implementation of an audio mixer for a system.

[0028] Figure 5A According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0029] Figure 5B According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0030] Figure 5C According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0031] Fig. 6A According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0032] Figure 6B According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0033] Figure 6C According to some examples of the present disclosure Figure 1A A schematic diagram of another illustrative implementation of an audio mixer for a system.

[0034] Figure 7 An example of a device operable to perform audio signal enhancement of an audio signal based on encoded data received from another device according to some examples of the present disclosure is shown.

[0035] Figure 8 An example of a device operable to transmit encoded data based on an enhanced audio signal according to some examples of the present disclosure is shown.

[0036] Fig. 9 An example of an integrated circuit operable to perform audio signal enhancement according to some examples of the present disclosure is shown.

[0037] Fig.10 is a schematic diagram of a mobile device operable to perform audio signal enhancement according to some examples of the present disclosure.

[0038] Fig.11 is a schematic diagram of a headset operable to perform audio signal enhancement according to some examples of the present disclosure.

[0039] Fig.12is a schematic diagram of a wearable electronic device operable to perform audio signal enhancement according to some examples of the present disclosure.

[0040] Fig.13 is a schematic diagram of a voice-controlled speaker system operable to perform audio signal enhancement according to some examples of the present disclosure.

[0041] Fig.14 is a schematic diagram of a camera operable to perform audio signal enhancement according to some examples of the present disclosure.

[0042] Fig.15 is a schematic diagram of a headset (such as a virtual reality, mixed reality, or augmented reality headset) operable to perform audio signal enhancement according to some examples of the present disclosure.

[0043] Fig.16 is a schematic diagram of a first example of a vehicle operable to perform audio signal enhancement according to some examples of the present disclosure.

[0044] Fig.17 is a schematic diagram of a second example of a vehicle operable to perform audio signal enhancement according to some examples of the present disclosure.

[0045] Fig.18 Some examples according to the present disclosure may be Figure 1A A schematic diagram of a specific implementation of a method for audio signal enhancement performed by a device.

[0046] Fig.19 is a block diagram of a specific illustrative example of a device operable to perform audio signal enhancement according to some examples of the present disclosure. DETAILED DESCRIPTION

[0047] Various devices perform signal enhancement (such as noise suppression) to generate an enhanced audio signal. Signal enhancement (some examples include noise suppression, audio scaling, beamforming, de-reverberation, source separation, bass adjustment, equalization) may remove audio context from the enhanced audio signal. The audio context in the present disclosure may generally refer to one or more audio signals or signal components that provide audible space and / or environmental information for the enhanced audio signal. For example, signal enhancement may be performed on an audio signal captured via a microphone of a device during a call to generate an enhanced audio signal without background sounds (e.g., as a result of noise suppression of the audio signal). In the absence of background sounds, a listener of the enhanced audio signal cannot determine whether the speaker is in a busy market or in an office. Signal enhancement (e.g., noise suppression) may also introduce artifacts that reduce audio quality. For example, the speech in the enhanced audio signal may sound incoherent.

[0048] Recently, signal enhancement using one or more generative networks has been introduced. Specifically, so-called generative adversarial networks (GANs) can be used to generate audio signals (such as speech signals) with improved signal quality (e.g., with an increased signal-to-noise ratio or even without any background sounds). In a GAN, a generative network can generate candidates for data (such as elements (e.g., words, phonemes, etc.) of a speech signal), and a discriminative network evaluates these candidates. Signal enhancement in the present disclosure can use at least one generative network (e.g., a GAN) to process one or more input audio signals to generate one or more enhanced monophonic audio signals.

[0049] Specifically, one or more input signals may be audio signals captured by one or more microphones in a soundscape that includes a source of a primary (target) audio signal, such as a speech signal (e.g., uttered by a person), and one or more sources of secondary (unwanted) audio signals (e.g., other speech signals, directional noise, diffuse noise, etc.). Signal enhancement in the present disclosure may refer to at least partially removing secondary audio signals from the input audio signal. As described above, in some examples, one or more generation networks may be used to remove secondary audio signals.

[0050] Alternatively or in addition, when performing signal enhancement, noise suppression by signal filtering, audio scaling, beamforming, de-reverberation, source separation, bass adjustment and / or equalization can be applied. For example, signal enhancement in the present disclosure can refer to increasing the gain of a target signal (e.g., a speech signal), reducing the gain of an unwanted audio signal, or both to perform an audio scaling operation. As described above, in some examples, the gain of the target signal can be increased based on the use of one or more generation networks. In addition, as described above, in some examples, the gain of an unwanted audio signal can be reduced based on the use of one or more generation networks.

[0051] One way to perform an audio scaling operation is to perform a beamforming operation, which includes generating a virtual audio beam formed by two or more microphones in the direction of a primary (target) audio signal and / or generating a null beam in the direction of a secondary (undesired) audio signal. Therefore, signal enhancement in the present disclosure may also refer to at least performing an audio scaling operation. As described above, in some examples, the scaling operation for increasing the perceptibility of the target signal can be based on the use of one or more generation networks.

[0052] In addition, signal enhancement in the present disclosure may also refer to generating a virtual audio beam in the direction of a target signal and / or generating a null beam in the direction of an unwanted sound signal. As described in the present disclosure, in some examples, beamforming may use one or more generation networks to focus on the target signal. In other examples within the present disclosure, unwanted signals may be removed based on using one or more generation networks.

[0053] In another example, the mixture of audio signals includes different types of sounds (e.g., speech signals, directional noise, diffuse noise, non-stationary noise, speech from multiple speakers, etc.). Signal enhancement in the present disclosure may also refer to source separation, wherein the mixture of audio signals is separated from each other. In other examples within the present disclosure, the source separation of the mixture of audio signals may be based on the use of one or more generation networks.

[0054] In some examples, an audio signal (such as music audio) includes various frequency components. Signal enhancement in the present disclosure may refer to equalization, wherein the balance of the frequency components is adjusted. In other examples within the present disclosure, the equalization of the frequency components of the audio signal may be based on the use of one or more generation networks.

[0055] In addition, generative audio techniques can be used to generate an enhanced mono audio signal. In some examples, the enhanced mono audio signal can be a noise-suppressed speech signal (e.g., a speech signal captured by one or more microphones or decoded from encoded audio data), where noise has been partially or completely removed from the corresponding one or more input audio signals. As described above, noise suppression can involve one or more generative networks (e.g., GANs), such as those described with respect to Figure 1A-1B Thus, the enhanced mono audio signal may be a mono speech signal with noise suppressed by applying one or more speech generation networks.

[0056] In some examples, the enhanced mono audio signal can be an audio scaled signal (e.g., a target signal captured by one or more microphones or decoded from encoded audio data), wherein the gain of the unwanted audio signal has been reduced, the gain of the target signal has been increased, or both. As described above, audio scaling can involve one or more generative networks (e.g., GANs), such as those described with respect to Figure 1A and 1C Thus, the enhanced mono audio signal may be a mono speech signal having audio scaled by applying one or more speech generation networks.

[0057] In some examples, the enhanced mono audio signal can be a beamformed signal (e.g., a target signal captured by one or more microphones or decoded from encoded audio data), wherein a virtual audio beam is formed by two or more microphones in the direction of a primary (target) audio signal, and / or a null beam is formed in the direction of a secondary (undesired) audio signal. As described above, beamforming can involve one or more generative networks (e.g., GANs), such as those described with respect to Figure 1A and 1D Thus, the enhanced mono audio signal may be a mono speech signal with audio beamformed by applying one or more speech generation networks.

[0058] In some examples, the enhanced mono audio signal may be a dereverberated signal (e.g., a speech signal captured by one or more microphones or decoded from encoded audio data), where reverberation has been partially or completely removed from the corresponding one or more input audio signals. As described above, dereverberation may involve one or more generative networks (e.g., GANs), such as those described with respect to Figure 1A and 1E As further described, the enhanced mono audio signal may be a mono speech signal with dereverberation by applying one or more speech generation networks.

[0059] In some examples, the enhanced mono audio signal can be a source separated signal (e.g., a target signal from a particular audio source captured by one or more microphones or decoded from encoded audio data), wherein unwanted audio signals have been partially or completely removed from the corresponding one or more input audio signals. As described above, source separation can involve one or more generative networks (e.g., GANs), such as those described with respect to Figure 1A and 1F Thus, the enhanced mono audio signal may be a mono speech signal having audio source separated by applying one or more speech generation networks.

[0060] In some examples, the enhanced mono audio signal can be a bass-adjusted signal (e.g., a music signal captured by one or more microphones or decoded from encoded audio data), where bass has been added or reduced from the corresponding one or more input audio signals. As described above, bass adjustment can involve one or more generative networks (e.g., GANs), such as those described with respect to Figure 1A and 1G Thus, the enhanced mono audio signal may be a mono speech signal with a bass adjusted by applying one or more speech generation networks.

[0061] In some examples, the enhanced mono audio signal can be an equalized signal (e.g., a music signal captured by one or more microphones or decoded from encoded audio data), wherein the balance of different frequency components is adjusted according to the corresponding one or more input audio signals. As described above, equalization can involve one or more generative networks (e.g., GANs), such as those described with respect to Figure 1A and 1H Thus, the enhanced mono audio signal may be a mono speech signal with equalization by applying one or more speech generation networks.

[0062] A system and method for audio signal enhancement is disclosed. In an illustrative example, a signal enhancer performs signal enhancement on an input audio signal to generate an enhanced mono audio signal. As described above, the enhanced mono audio signal can be an enhanced speech signal. The enhanced speech signal can be associated with a single / specific speaker. The enhanced speech signal can be a mono audio signal generated from one or more input audio signals. One or more microphones can be used to capture the one or more input audio signals, as described in more detail below. The enhanced mono audio signal is a single channel audio signal.

[0063] As described above, signal enhancement may include at least one of noise suppression, audio scaling, beamforming, de-reverberation, source separation, bass adjustment or equalization. The audio context in the present disclosure may refer to an auxiliary or secondary audio signal and / or signal component, wherein the enhanced mono audio signal represents a primary audio signal or signal component (such as a speech signal associated with an individual / specific speaker). As described above, the enhanced mono audio signal may be a processed input audio signal (e.g., filtered or beamformed) or a synthesized audio signal (e.g., generated using a generation network based on the input audio signal). In some examples, the primary audio signal or signal component refers to an audio signal (such as a speech signal) received from a specific audio source (such as an individual / specific speaker) on a direct path from an audio source to one or more microphones (e.g., without reverberation or ambient noise). On the other hand, the audio context may refer to an audio signal and / or signal component other than an audio signal directly received from a specific audio source. In some examples, the audio context can include reverberant / reflected speech signals originating from an individual / specific speaker, speech signals from speakers other than the specific speaker, diffuse background noise, localized or directional noise (e.g., from a moving vehicle), or a combination thereof.

[0064] The audio mixer generates a first audio signal based on an enhanced monophonic audio signal. In some examples, the first audio signal is identical to the enhanced monophonic audio signal. In other examples, the audio mixer can perform additional processing (such as panning or binauralization) to generate the first audio signal to the enhanced monophonic audio signal. In some aspects, the first audio signal corresponds to an enhanced audio signal with a reduced audio context. In order to add such context, the audio mixer generates a second audio signal based on at least one of a directional audio signal, a background audio signal or a reverberation signal. The audio mixer mixes the first audio signal and the second audio signal to generate a stereo audio signal. The stereo audio signal therefore balances the signal enhancement included in the first audio signal with the audio context included in the second audio signal.

[0065] Certain aspects of the present disclosure are described below with reference to the accompanying drawings. In this description, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some implementations and plural in other implementations. For example, Figure 1A Depicts a system including one or more processors ( Figure 1A 190), indicating that in some implementations, the device 102 includes a single processor 190, while in other implementations, the device 102 includes multiple processors 190. For ease of reference, such features are generally introduced as "one or more" features and are subsequently referred to in the singular unless aspects related to multiple features are being described.

[0066] In some of the figures, multiple instances of a particular type of feature are used. Although the features are physically and / or logically distinct, the same reference numeral is used for each feature, and different instances are distinguished by adding a letter to the reference numeral. When features are referred to herein as a group or type, for example, when no reference is made to a specific one of the features, the reference numeral is used without a distinguishing letter. However, when a specific feature of multiple features of the same type is referred to herein, the reference numeral is used with a distinguishing letter. For example, when reference is made to Figure 1A, a plurality of directional (direct.) audio signals are shown and are associated with reference numerals 165A and 165B. When referring to a specific one of these directional audio signals (such as directional audio signal 165A), a distinguishing letter "A" is used. However, when referring to any one of these directional audio signals or referring to these directional audio signals as a group, reference numeral 165 is used without a distinguishing letter.

[0067] As used herein, the terms "comprise", "comprises" and "comprising" may be used interchangeably with "include", "includes" or "including". In addition, the term "wherein" may be used interchangeably with "wherein". As used herein, "exemplary" indicates examples, implementations and / or aspects, and should not be interpreted as limiting or indicating preferences or preferred implementations. As used herein, ordinal numbers (e.g., "first", "second", "third", etc.) used to modify elements such as structures, components, operations, etc. do not themselves indicate any priority or order of the element relative to another element, but merely distinguish the element from another element with the same name (but without the use of an ordinal number). As used herein, the term "set" refers to one or more of a particular element, and the term "plurality" refers to multiple (e.g., two or more) of a particular element. As used herein, "A" and / or "B" may mean "A and B" or "A or B", or both "A and B" and "A or B" are applicable or acceptable.

[0068] As used herein, "coupling" may include "communicatively coupled," "electrically coupled," or "physically coupled," and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) via one or more other devices, components, lines, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device or different devices, and may be connected via electronics, one or more connectors, or inductive coupling. In some implementations, two devices (or components) that are communicatively coupled (such as in electronic communication) may send and receive signals (e.g., digital signals or analog signals) directly or indirectly (via one or more lines, buses, networks, etc.). As used herein, "directly coupled" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without an intermediate component.

[0069] In the present disclosure, terms such as "determine," "calculate," "estimate," "shift," "adjust," and the like may be used to describe how to perform one or more operations. It should be noted that such terms should not be construed as limiting, and other techniques may be utilized to perform similar operations. Additionally, as referenced herein, "generate," "calculate," "estimate," "use," "select," "access," and "determine" may be used interchangeably. For example, "generating," "calculating," "estimating," or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating, or determining a parameter (or signal), or may refer to using, selecting, or accessing a parameter (or signal) that has already been generated, such as by another component or device.

[0070] refer to Figure 1A , particular illustrative aspects of a system configured to perform audio signal enhancement are disclosed and generally designated 100. System 100 includes a device 102, which includes one or more processors 190. In some implementations, device 102 is coupled to one or more microphones 120, one or more cameras 130, or a combination thereof. One or more microphones 120 may include a one-dimensional or two-dimensional microphone array for performing beamforming. One or more processors 190 include an audio analyzer 140, which is configured to process one or more input audio signals 125 to generate one or more stereo audio signals 149, as described herein.

[0071] The audio analyzer 140 is configured to obtain one or more input audio signals 125 representing a soundscape of sounds 185 having one or more audio sources 184. In some implementations, the one or more input audio signals 125 correspond to microphone outputs of one or more microphones 120, decoded audio data, an audio stream, or a combination thereof. For example, the audio analyzer 140 is configured to receive a first input audio signal 125 from a first microphone of the one or more microphones 120, and receive a second input audio signal 125 from a second microphone of the one or more microphones 120. In another example, the first input audio signal 125 corresponds to a first audio channel of the decoded audio data (or audio stream), and the second input audio signal 125 corresponds to a second audio channel of the decoded audio data (or audio stream).

[0072] In certain aspects, audio analyzer 140 is configured to obtain image data 127 representing a visual scene associated with (e.g., including) one or more audio sources 184. In some examples, image data 127 is based on camera output, decoded image data, stored image data, a graphical visual stream, or a combination thereof.

[0073] The audio analyzer 140 includes a signal enhancer 142 coupled to an audio mixer 148. The audio analyzer 140 also includes a directional analyzer 144 coupled to the signal enhancer 142, the audio mixer 148, or both. In some implementations, the audio analyzer 140 includes a context analyzer 146 coupled to the directional analyzer 144, the audio mixer 148, or both. In some implementations, the audio analyzer 140 includes a position sensor 162 coupled to the context analyzer 146.

[0074] The context analyzer 146 is configured to process the image data 127 to generate a visual context 147 of the one or more input audio signals 125. The visual context 147 may include any information associated with the one or more input audio signals 125 other than the input audio signals 125 themselves and may be derived based on the image data 127. Such information may include, but is not limited to, the (relative) position of the audio source in the visual scene, the (relative) position of the microphone in the visual scene, the acoustic characteristics of the soundscape (such as open space, confined / enclosed space, room geometry, etc.). In some examples, the visual context 147 indicates the position (e.g., elevation and azimuth) of the audio source 184A (e.g., a person) in the one or more audio sources 184 in the visual scene represented by the image data 127. In some examples, the visual context 147 indicates the environment of the visual scene represented by the image data 127. For example, the visual context 147 is based on the surface of the acoustic environment, the room geometry, or both. In some implementations, the context analyzer 146 is configured to process at least the image data 127 using the neural network 156 to generate the visual context 147. In some examples, one or more operations described herein as being performed by a neural network may be performed using a machine learning network, such as an artificial neural network (ANN), other types of machine learning networks (e.g., based on fuzzy logic, evolutionary programming, and / or genetic algorithms), and the like.

[0075] The context analyzer 146 is configured to obtain location data 163 indicating the location of a soundscape represented by the one or more input audio signals 125, a visual scene represented by the image data 127, or both. In some implementations, a location sensor 162 (e.g., a global positioning system (GPS) sensor) is configured to generate the location data 163. In other implementations, the location data 163 is generated by an application, received from another device, or both.

[0076] The context analyzer 146 is configured to process the position data 163 to generate a position context 137 for the one or more input audio signals 125. For example, the position context 137 indicates a position, a type of position, or both of a soundscape represented by the one or more input audio signals 125, a visual scene represented by the image data 127, or both. In some implementations, the context analyzer 146 processes at least the position data 163 to generate the position context 137 using the neural network 156.

[0077] The directional analyzer 144 is configured to perform directional audio coding (DirAC) on the one or more input audio signals 125 to generate one or more directional audio signals 165, background audio signals 167, or both. The one or more directional audio signals 165 correspond to directional sounds (e.g., speech or cars), while the background audio signals 167 correspond to diffuse noise in the soundscape (e.g., wind noise or background traffic). In some implementations, the directional analyzer 144 is configured to use the neural network 154 to perform DirAC on the one or more input audio signals 125.

[0078] The signal enhancer 142 is configured to perform signal enhancement to generate one or more enhanced mono audio signals 143. Signal enhancement is performed on one or more input audio signals 125, one or more directional audio signals 165, or a combination thereof to generate one or more enhanced mono audio signals 143. For example, the signal enhancer 142 performs signal enhancement on the first input audio signal 125 to generate a first enhanced mono audio signal 143, and performs signal enhancement on the second input audio signal 125 to generate a second enhanced mono audio signal 143. As another example, the signal enhancer 142 performs signal enhancement on the directional audio signal 165A to generate a first enhanced mono audio signal 143, and performs signal enhancement on the directional audio signal 165B to generate a second enhanced mono audio signal 143. Signal enhancement includes at least one of noise suppression, audio scaling, beamforming, dereverberation, source separation, bass adjustment, or equalization. In a specific implementation, the signal enhancer 142 is configured to perform signal enhancement using a neural network 152. For example, the signal enhancer 142 may be configured to use a neural network 152 (e.g., a GAN) to perform a generative audio technique to generate one or more enhanced mono audio signals 143. By way of example, the signal enhancer 142 may use the neural network 152 to partially or completely remove noise for noise suppression, adjust signal gain for audio scaling, perform beamforming for audio focusing, perform dereverberation to remove the effects of reverberation, partially or completely separate audio for source separation, perform bass adjustment to increase or decrease bass, perform equalization to adjust the balance of different frequency components, or a combination thereof.

[0079] The audio mixer 148 is configured to generate one or more enhanced audio signals 151 based on the one or more enhanced monophonic audio signals 143, and to generate one or more audio signals 155 based on the one or more directional audio signals 165, the background audio signals 167, the position context 137, the visual context 147, or a combination thereof, as described in reference to FIG. Figure 2A , 3A , 4A, 5A and 6A further described. The audio mixer 148 is configured to mix the one or more enhanced audio signals 151 and the one or more audio signals 155 to generate one or more stereo audio signals 149, as shown in FIG. Figure 2A , 3A , 4A, 5A and 6A further described. In some implementations, the audio mixer 148 is configured to generate one or more stereo audio signals 149 using a neural network, as described in reference Figure 2B , 2C , 3B, 3C, 4B, 4C, 5B, 5C, 6B and 6C. The one or more stereo audio signals 149 include signal enhancements of the one or more enhanced audio signals 151 and audio contexts of the one or more audio signals 155.

[0080] In some implementations, device 102 corresponds to or is included in one of various types of devices. In the illustrative example, one or more processors 190 are integrated into a computer such as a reference computer. Fig.11 In other examples, the one or more processors 190 are integrated into at least one of the following: Fig.10 A mobile phone or tablet computer device as described, such as reference Fig.12 The wearable electronic device described in reference Fig.13 The voice-controlled speaker system described in reference Fig.14 The camera device described in the reference Fig.15 In another illustrative example, one or more processors 190 are integrated into a virtual reality, mixed reality, or augmented reality headset such as the one described in the reference Fig.16 and Fig.17 In the vehicle further described.

[0081] During operation, the audio analyzer 140 obtains one or more input audio signals 125. In one example, the one or more input audio signals 125 correspond to sounds 185 of one or more audio sources 184 captured by one or more microphones 120. In another example, the one or more processors 190 are configured to decode the encoded audio data to generate the one or more input audio signals 125, as shown in FIG. Figure 7In the illustrative example, the encoded audio data corresponds to audio received during a call with a second user of the second device, and the audio analyzer 140 provides one or more stereo audio signals 149 to one or more speakers for playback to the user 101, as described in reference to Figure 7 Further described. In some examples, the one or more input audio signals 125 are based on an audio stream. For example, the one or more input audio signals 125 are generated by at least one of the following: an audio player, a game application, a communication application, a video player, an augmented reality application, or another application of the one or more processors 190.

[0082] In one example, one or more audio sources 184 include audio source 184A, audio source 184B, audio source 184C, one or more additional audio sources, or a combination thereof. By way of example, sound 185 includes speech from audio source 184A (e.g., a person), directional noise from audio source 184B (e.g., a passing car), diffuse noise from audio source 184C (e.g., leaves moving in the wind), or a combination thereof. In some examples, audio source 184C may be invisible, such as wind. In some examples, audio source 184C may correspond to multiple audio sources that together correspond to diffuse noise, such as traffic or leaves. In some examples, the sound from audio source 184C may be non-directional or omnidirectional.

[0083] In certain aspects, the audio analyzer 140 obtains image data 127 representing a visual scene associated with (e.g., including) one or more audio sources 184. In one example, the audio analyzer 140 is configured to receive the image data 127 from the one or more cameras 130 concurrently with receiving the one or more input audio signals 125 from the one or more microphones 120. In another example, the one or more processors 190 are configured to receive encoded data from another device and decode the encoded data to generate the image data 127, as described in reference to FIG. Figure 7 In some implementations, the image data 127 is based on a graphical visual stream received from an application of the one or more processors 190 , such as the same application that generated the one or more input audio signals 125 .

[0084] In some implementations, the context analyzer 146 determines a visual context 147 of the one or more input audio signals 125 based on the image data 127. In an illustrative example, the context analyzer 146 performs audio source detection (e.g., face detection) on the image data 127 to generate a visual context 147 that indicates a location (e.g., elevation and azimuth) of an audio source 184A (e.g., a person) in one or more audio sources 184 in a visual scene represented by the image data 127. In a particular aspect, the visual context 147 indicates a location of the audio source 184A relative to a location of an assumed listener (e.g., represented by one or more microphones 120) in the visual scene.

[0085] In certain aspects, visual context 147 is based on user input 103 indicating a posture (e.g., position, orientation, tilt, or a combination thereof) of user 101 (e.g., the head of user 101). By way of example, user input 103 may correspond to sensor data indicating movement of a headset, camera output representing an image of user 101, or both. In some implementations, visual context 147 indicates the environment of the visual scene represented by image data 127. For example, visual context 147 is based on surfaces of the environment, room geometry, or both.

[0086] In some implementations, the context analyzer 146 obtains location data 163 indicating the location of a soundscape represented by one or more input audio signals 125, a visual scene represented by the image data 127, or both. In some implementations, the context analyzer 146 processes the image data 127 based on the location data 163 to determine the visual context 147. In the illustrative example, the context analyzer 146 processes the image data 127 and detects a plurality of faces. The context analyzer 146, in response to determining that the location data 163 instructs the art gallery to perform an analysis of the image data 127 to distinguish between depicted faces and actual people to generate a visual context 147 indicating the location of a person as the audio source 184A.

[0087] In some aspects, the audio analyzer 140 determines a location context 137 of the one or more input audio signals 125 based on the location data 163. The location context 137 indicates a specific location indicated by the location data 163, a location type of the specific location, or both. The location indicated by the location data 163 may correspond to a geographic location, a virtual location, or both. The location type may indicate an open space or an enclosed or confined space. Non-limiting examples of location types may be indoors, outdoors, an office, a playground, a park, inside an airplane, inside a vehicle, etc.

[0088] In some examples, the location data 163 corresponds to GPS data indicating the location of the device 102, the one or more microphones 120, another device for generating the one or more input audio signals 125, or a combination thereof. The context analyzer 146 generates a location context 137 indicating a location associated with the GPS data, a type of location, or both. As another example, an application (e.g., a gaming application) of the one or more processors 190 generates location data 163 indicating a virtual location of a soundscape represented by the one or more input audio signals 125. The context analyzer 146 generates a location context 137 indicating a virtual location (e.g., a “training hall”), a type of virtual location (e.g., a large room), or both.

[0089] In some implementations, the location data 163 corresponds to user data (e.g., calendar data, user login data, etc.) indicating that the user 101 of the device 102 was at a particular location when one or more microphones 120 of the device 102 generated one or more input audio signals 125. In a particular aspect, the location data 163 corresponds to user data (e.g., calendar data, user login data, etc.) indicating that the second user was at a particular location when one or more input audio signals 125 were received from a second device of the second user (e.g., audio source 184A). The context analyzer 146 generates a location context 137 indicating a particular location (e.g., a large bazaar), a type of particular location (e.g., a covered market), or both.

[0090] In some examples, context analyzer 146 processes location data 163 based on image analysis of image data 127, audio analysis of one or more input audio signals 125, or both to generate location context 137. As an illustrative example, context analyzer 146 generates location context 137 indicating a location type corresponding to the interior of a vehicle on a highway in response to determining that location data 163 indicates a highway and image data 127 indicates the interior of a vehicle.

[0091] Directional analyzer 144 performs DirAC on one or more input audio signals 125 to generate one or more directional audio signals 165, background audio signals 167, or both. In some aspects, one or more directional audio signals 165 correspond to directional sounds, while background audio signals 167 correspond to diffuse noise. In an illustrative example, directional audio signal 165A represents speech from audio source 184A (e.g., a person), directional audio signal 165B represents directional noise from audio source 184B (e.g., a passing car), and background audio signal 167 represents diffuse noise from audio source 184C (e.g., leaves moving in the wind). In some aspects, the specific sound direction of the sound represented by directional audio signal 165A can change over time. In an illustrative example, the direction of the sound represented by directional audio signal 165A changes as audio source 184B (e.g., a car) moves relative to the assumed listener in the soundscape represented by one or more input audio signals 125.

[0092] In some implementations, the directional analyzer 144 generates the directional audio signal 165A based on audio source detection data (e.g., facial detection data) indicated by the visual context 147. For example, the visual context 147 indicates an estimated position (e.g., an absolute position or a relative position) of the audio source 184A in the visual scene represented by the image data 127, and the directional analyzer 144 generates the directional audio signal 165A based on the sound corresponding to the estimated position. As another example, the directional analyzer 144 performs an analysis (e.g., speech separation) corresponding to a specific audio source type in response to determining that the visual context 147 indicates a specific audio source type (e.g., a face) to generate the directional audio signal 165A from one or more input audio signals 125. As another example, the directional analyzer 144 performs spatial filtering or beamforming of multiple input audio signals captured by multiple microphones 120 in response to determining the (relative) position of the audio source (e.g., the audio source 184A) to spatially isolate the audio signal from the audio source. In a particular example, the directional analyzer 144 performs gain adjustments on multiple input audio signals captured by multiple microphones 120 in response to determining the (relative) position of an audio source (eg, audio source 184A) to perform audio scaling of audio signals from the audio source.

[0093] In some implementations, the directional analyzer 144 provides one or more directional audio signals 165 to the signal enhancer 142 and provides the background audio signal 167 to the audio mixer 148. In these implementations, the signal enhancer 142 performs signal enhancement on the one or more directional audio signals 165 to generate one or more enhanced mono audio signals 143. The audio mixer 148 generates one or more stereo audio signals 149 based on the one or more enhanced mono audio signals 143 and the background audio signal 167.

[0094] In some implementations, the directional analyzer 144 provides one or more directional audio signals 165, background audio signals 167, or a combination thereof to the audio mixer 148. In these implementations, the signal enhancer 142 performs signal enhancement on the one or more input audio signals 125 to generate one or more enhanced mono audio signals 143. The audio mixer 148 generates one or more stereo audio signals 149 based on the one or more enhanced mono audio signals 143 and further based on the one or more directional audio signals 165, background audio signals 167, or a combination thereof.

[0095] The signal enhancer 142 performs signal enhancement to generate one or more enhanced mono audio signals 143. The signal enhancement includes at least one of noise suppression, audio scaling, beamforming, dereverberation, source separation, bass adjustment, or equalization. In some examples, the signal enhancer 142 selects signal enhancement from at least one of noise suppression, audio scaling, beamforming, dereverberation, source separation, bass adjustment, or equalization, and performs the selected signal enhancement to generate one or more enhanced mono audio signals 143. The signal enhancer 142 may select the signal enhancement based on the user input 103 received from the user 101, a configuration setting, default data, or a combination thereof.

[0096] The audio mixer 148 receives one or more enhanced mono audio signals 143 from the signal enhancer 142. The audio mixer 148 also receives at least one of the following: one or more directional audio signals 165, background audio signals 167, location context 137, visual context 147, or a combination thereof. The audio mixer 148 generates one or more enhanced audio signals 151 based on the one or more enhanced mono audio signals 143, and generates one or more audio signals 155 based on the one or more directional audio signals 165, background audio signals 167, visual context 147, or a combination thereof. In some implementations, the one or more enhanced audio signals 151 are the same as the one or more enhanced mono audio signals 143, as described in reference to FIG. Figure 2A and 3AIn some implementations, the one or more enhanced audio signals 151 correspond to binauralization applied to the one or more enhanced monophonic audio signals 143, as described in reference to Figure 4A In some implementations, the one or more enhanced audio signals 151 correspond to panning applied to the one or more enhanced monophonic audio signals 143, as described in reference to Figure 5A and 6A As further described, the one or more enhanced audio signals 151 are based on the one or more enhanced mono audio signals 143 and include signal enhancement performed by the signal enhancer 142 .

[0097] In some implementations, the one or more audio signals 155 correspond to the delay and attenuation applied to the background audio signal 167, as described in reference to FIG. Figure 2A In some implementations, the one or more audio signals 155 correspond to delays and panning applied to the one or more directional audio signals 165, as described in reference to Figure 3A In some implementations, one or more audio signals 155 correspond to delays applied to one or more directional audio signals 165, as described in reference to Figure 4A In some implementations, the one or more audio signals 155 correspond to a reverberation signal based on one or more directional audio signals 165, background audio signals 167, or a combination thereof, as described in reference to Figure 5A In some implementations, the one or more audio signals 155 correspond to synthesized reverberation signals based on the location context 137, the visual context 147, or both, as described in reference to Fig. 6A As further described, the one or more audio signals 155 include an audio context associated with the one or more input audio signals 125 .

[0098] The audio mixer 148 mixes the one or more enhanced audio signals 151 with the one or more audio signals 155 to generate one or more stereo audio signals 149. In a specific implementation, the audio mixer 148 receives the enhanced mono audio signal 143A (e.g., enhanced first microphone sound, such as noise-suppressed speech), the enhanced mono audio signal 143B (e.g., enhanced second microphone sound, such as noise-suppressed speech), the directional audio signal 165A (e.g., speech), the directional audio signal 165B (e.g., directional noise), the background audio signal 167 (e.g., diffuse noise), or a combination thereof. In the illustrative example of this implementation, the audio mixer 148 mixes the one or more enhanced audio signals 151 corresponding to the noise-suppressed speech with the one or more audio signals 155 corresponding to the directional noise or corresponding to the reverberation signal based on the directional noise or the diffuse noise. The one or more stereo audio signals 149 include less noise (e.g., no diffuse noise) than the one or more input audio signals 125 while providing more audio context (e.g., directional noise, reverberation based on background noise, etc.) than the one or more enhanced mono audio signals 143 (e.g., noise suppressed speech).

[0099] In an alternative implementation, the signal enhancer 142 performs signal enhancement on the directional audio signal 165A and the directional audio signal 165B, respectively, to generate an enhanced mono audio signal 143A (e.g., enhanced speech) and an enhanced mono audio signal 143B (e.g., signal-enhanced directional noise, such as noise-suppressed silence). The audio mixer 148 receives the enhanced mono audio signal 143A (e.g., enhanced speech), the enhanced mono audio signal 143B (e.g., noise-suppressed silence), and the background audio signal 167 (e.g., diffuse noise). In the illustrative example of this implementation, the audio mixer 148 receives one or more enhanced audio signals 151 corresponding to noise-suppressed speech and silence, and mixes them with one or more audio signals 155 corresponding to a reverberation signal based on diffuse noise. The stereo audio signal 149 includes less noise (eg, no background noise) than the input audio signal 125 while providing more audio context (eg, diffuse noise-based reverberation) than the one or more enhanced mono audio signals 143 (eg, noise suppressed speech).

[0100] Thus, the system 100 balances the signal enhancement performed by the signal enhancer 142 and the audio context associated with the one or more input audio signals 125 when generating the one or more stereo audio signals 149. For example, the one or more stereo audio signals 149 may include directional noise or reverberation that provides audio context to the listener while removing at least some of the background noise (e.g., diffuse noise or all background noise).

[0101] Optionally, in some implementations, the signal enhancer 142 selects one or more of the signal enhancements (e.g., noise suppression, audio scaling, beamforming, de-reverberation, source separation, bass adjustment, or equalization), and the second signal enhancer performs one or more of the remaining signal enhancements. In certain aspects, the second signal enhancer is a component external to the signal enhancer 142 (because the directional analyzer 144 is external to the signal enhancer 144). In these implementations, a particular signal enhancement performed by the signal enhancer 142 may be performed before or after other signal enhancements performed by the second signal enhancer. By way of example, the input to the signal enhancer 142 may be based on the output of the second signal enhancer, the input to the second signal enhancer may be based on the output of the signal enhancer 142, or both. For example, the second signal enhancer performs specific signal enhancement on the one or more input audio signals 125 or the directional audio signal 165 to generate one or more second enhanced mono audio signals, and the signal enhancer 142 performs other signal enhancement on the one or more second enhanced mono audio signals to generate one or more enhanced mono audio signals 143. In another example, the second signal enhancer performs additional signal enhancement on the one or more enhanced mono audio signals 143 to generate one or more second enhanced mono audio signals, and the audio mixer 148 generates one or more enhanced audio signals 151 based on the one or more second enhanced mono audio signals.

[0102] Although the one or more microphones 120 and the one or more cameras 130 are shown as being external to the device 102, in other implementations, at least one of the one or more microphones 120 or the one or more cameras 130 may be integrated into the device 102. Although the one or more input audio signals 125 are shown as corresponding to microphone outputs of the one or more microphones 120, in other implementations, the one or more input audio signals 125 may correspond to decoded audio data, an audio stream, stored audio data, or a combination thereof. Although the image data 127 is shown as corresponding to camera outputs of the one or more cameras 130, in other implementations, the image data 127 may correspond to decoded image data, an image stream, stored image data, or a combination thereof.

[0103] Although the audio analyzer 140 is shown as being included in a single device (e.g., 140), two or more components of the audio analyzer 140 may be distributed across multiple devices. For example, the signal booster 142, the directional analyzer 144, the context analyzer 146, the location sensor 162, or a combination thereof may be integrated in a first device (e.g., a user playback device), and the audio mixer 148 may be integrated in a second device (e.g., a headset).

[0104] Optionally, one or more operations described herein with reference to the components of the audio analyzer 140 may be performed by a neural network. In one such example, the signal enhancer 142 uses a neural network 152 (e.g., a speech generation network) to perform signal enhancement to generate an enhanced mono audio signal 143A representing an enhanced version of the speech of the audio source 184A. In another example, the directional analyzer 144 uses a neural network 154 to process one or more input audio signals 125 to generate one or more directional audio signals 165, background audio signals 167, or a combination thereof. In yet another example, the context analyzer 146 uses a neural network 156 to process the image data 127, the position data 163, or both to generate a visual context 147, a position context 137, or both. In some examples, the neural network 156 includes a first neural network and a second neural network to process the image data 127, the position data 163, or both, respectively, to generate the visual context 147 and the position context 137. In one example, the audio mixer 148 uses a neural network to generate one or more stereo audio signals 149, as described with reference to Figure 2B , 3B , 4B, 5B, 6B and reference Figure 2C , 3C , 4C, 5C, and 6C. In some implementations, two or more of the neural networks described herein can be combined into a single neural network.

[0105] refer to Figure 1B , shows an illustrative implementation of the signal enhancer 142. The signal enhancer 142 is configured to perform noise suppression 132 on one or more input audio signals 115A to generate one or more enhanced audio signals 133A.

[0106] The signal enhancer 142 performs noise suppression 132 to partially or completely remove noise from the one or more input audio signals 115A to generate one or more enhanced audio signals 133A (e.g., enhanced mono audio signals). The one or more input audio signals 115A are based on the one or more input audio signals 125 or the one or more directional audio signals 165. For example, the one or more input audio signals 115A are the one or more input audio signals 125. As another example, the one or more input audio signals 115A are the one or more directional audio signals 165. In yet another example, the one or more input audio signals 115A include the one or more enhanced audio signals generated by the signal enhancer 142, as shown in FIG. Figure 1C-1H In a specific example, the one or more input audio signals 115A include one or more enhanced audio signals generated by a second signal enhancer external to the signal enhancer 142 .

[0107] In one example, the one or more enhanced audio signals 133A correspond to one or more noise suppressed speech signals. Optionally, in some implementations, the signal enhancer 142 uses the neural network 152A to perform noise suppression 132. Thus, the one or more enhanced audio signals 133A may be one or more monophonic speech signals with noise suppressed by applying the neural network 152A. The one or more enhanced monophonic audio signals 143 are based on the one or more enhanced audio signals 133A.

[0108] refer to Figure 1C , shows an illustrative implementation of the signal enhancer 142. The signal enhancer 142 is configured to perform audio scaling 134 on the input audio signal 115B to generate an enhanced audio signal 133B.

[0109] The signal enhancer 142 performs audio scaling 134 to reduce the gain of unwanted audio signals in the input audio signal 115B, increase the gain of target audio signals in the input audio signal 115B, or both, to generate an enhanced audio signal 133B (e.g., an enhanced mono audio signal). The enhanced audio signal 133B corresponds to the audio scaled signal. The input audio signal 115B is based on the input audio signal 125 or the directional audio signal 165. For example, the input audio signal 115B is the input audio signal 125. As another example, the input audio signal 115B is the directional audio signal 165. In yet another example, the input audio signal 115B includes the enhanced audio signal generated by the signal enhancer 142, as shown in FIG. Figure 1B , 1DIn a specific example, the input audio signal 115B includes an enhanced audio signal generated by a second signal enhancer external to the signal enhancer 142 .

[0110] In one example, the enhanced audio signal 133B corresponds to a scaled speech signal. Optionally, in some implementations, the signal enhancer 142 uses a neural network 152B to perform audio scaling 134. Thus, the enhanced audio signal 133B may be a mono speech signal with scaled audio generated by applying the neural network 152B. One or more enhanced mono audio signals 143 are based on the enhanced audio signal 133B.

[0111] refer to Figure 1D , shows an illustrative implementation of the signal enhancer 142. The signal enhancer 142 is configured to perform beamforming 136 on the input audio signal 115C to generate an enhanced audio signal 133C.

[0112] The signal enhancer 142 performs beamforming 136 to form a virtual beam in the direction of a primary (e.g., target) audio source and / or to form a null beam in the direction of a secondary (e.g., unwanted) audio source to generate an enhanced audio signal 133C (e.g., an enhanced mono audio signal). The enhanced audio signal 133C corresponds to the beamformed audio signal. The input audio signal 115C is based on the input audio signal 125 or the directional audio signal 165. For example, the input audio signal 115C is the input audio signal 125. As another example, the input audio signal 115B is the directional audio signal 165. In yet another example, the input audio signal 115C includes an enhanced audio signal generated by the signal enhancer 142, as shown in FIG. Figures 1A-1C In certain examples, the input audio signal 115B includes one or more enhanced audio signals generated by a second signal enhancer external to the signal enhancer 142 .

[0113] In one example, the enhanced audio signal 133C corresponds to a beamformed speech signal. Optionally, in some implementations, the signal enhancer 142 uses a neural network 152C to perform beamforming 136. Thus, the enhanced audio signal 133B may be a monophonic speech signal with beamformed audio generated by applying the neural network 152C. One or more enhanced monophonic audio signals 143 are based on the enhanced audio signal 133B.

[0114] refer to Figure 1E, shows an illustrative implementation of the signal enhancer 142. The signal enhancer 142 is configured to perform dereverberation 138 on one or more input audio signals 115D to generate one or more enhanced audio signals 133D.

[0115] The signal enhancer 142 performs dereverberation 138 to partially or completely remove reverberation from the one or more input audio signals 115D to generate one or more enhanced audio signals 133D (e.g., enhanced mono audio signals). The one or more input audio signals 115D are based on the one or more input audio signals 125 or the one or more directional audio signals 165. For example, the one or more input audio signals 115D are the one or more input audio signals 125. As another example, the one or more input audio signals 115D are the one or more directional audio signals 165. In yet another example, the one or more input audio signals 115D include the one or more enhanced audio signals generated by the signal enhancer 142, as shown in FIG. Figure 1B-1D In certain examples, the one or more input audio signals 115D include one or more enhanced audio signals generated by a second signal enhancer external to the signal enhancer 142 .

[0116] In one example, the one or more enhanced audio signals 133D correspond to one or more dereverberated speech signals. Optionally, in some implementations, the signal enhancer 142 uses a neural network 152D to perform dereverberation 138. Thus, the one or more enhanced audio signals 133D may be one or more monophonic speech signals with dereverberation generated by applying the neural network 152D. The one or more enhanced monophonic audio signals 143 are based on the one or more enhanced audio signals 133D.

[0117] refer to Figure 1F , shows an illustrative implementation of the signal enhancer 142. The signal enhancer 142 is configured to perform source separation 150 on one or more input audio signals 115E to generate one or more enhanced audio signals 133E.

[0118] The signal enhancer 142 performs source separation 150 to partially or completely remove sounds of secondary (e.g., unwanted) audio sources from the one or more input audio signals 115E to generate one or more enhanced audio signals 133E (e.g., enhanced mono audio signals). The one or more input audio signals 115E are based on the one or more input audio signals 125 or the one or more directional audio signals 165. For example, the one or more input audio signals 115E are the one or more input audio signals 125. As another example, the one or more input audio signals 115E are the one or more directional audio signals 165. In yet another example, the one or more input audio signals 115E include the one or more enhanced audio signals generated by the signal enhancer 142, as shown in FIG. Figure 1B-1E In certain examples, the one or more input audio signals 115E include one or more enhanced audio signals generated by a second signal enhancer external to the signal enhancer 142.

[0119] In one example, the one or more enhanced audio signals 133E correspond to a source-separated speech signal. Optionally, in some implementations, the signal enhancer 142 uses a neural network 152E to perform source separation 150. Thus, the one or more enhanced audio signals 133E may be one or more monophonic speech signals with source-separated audio generated by applying the neural network 152E. The one or more enhanced monophonic audio signals 143 are based on the one or more enhanced audio signals 133E.

[0120] refer to Figure 1G , shows an illustrative implementation of the signal enhancer 142. The signal enhancer 142 is configured to perform bass adjustment 158 ​​on one or more input audio signals 115F to generate one or more enhanced audio signals 133F.

[0121] The signal enhancer 142 performs the bass adjustment 158 ​​to increase or decrease the bass in the one or more input audio signals 115F to generate one or more enhanced audio signals 133F (e.g., enhanced mono audio signals). The one or more input audio signals 115F are based on the one or more input audio signals 125 or the one or more directional audio signals 165. For example, the one or more input audio signals 115F are the one or more input audio signals 125. As another example, the one or more input audio signals 115F are the one or more directional audio signals 165. In yet another example, the one or more input audio signals 115F include the one or more enhanced audio signals generated by the signal enhancer 142, as shown in FIG. Figures 1B-1FIn a specific example, the one or more input audio signals 115F include one or more enhanced audio signals generated by a second signal enhancer external to the signal enhancer 142 .

[0122] In one example, the one or more enhanced audio signals 133F correspond to one or more bass-adjusted speech signals. Optionally, in some implementations, the signal enhancer 142 uses the neural network 152F to perform the bass adjustment 158. Thus, the one or more enhanced audio signals 133F may be one or more monophonic speech signals with bass adjusted by applying the neural network 152F. The one or more enhanced monophonic audio signals 143 are based on the one or more enhanced audio signals 133F.

[0123] refer to Figure 1H , shows an illustrative implementation of the signal enhancer 142. The signal enhancer 142 is configured to perform equalization 160 on one or more input audio signals 115G to generate one or more enhanced audio signals 133G.

[0124] The signal enhancer 142 performs equalization 160 to adjust the balance of various frequency components of the one or more input audio signals 115G to generate one or more enhanced audio signals 133G (e.g., enhanced mono audio signals). The one or more input audio signals 115G are based on the one or more input audio signals 125 or the one or more directional audio signals 165. For example, the one or more input audio signals 115G are the one or more input audio signals 125. As another example, the one or more input audio signals 115G are the one or more directional audio signals 165. In yet another example, the one or more input audio signals 115G include one or more enhanced audio signals generated by the signal enhancer 142, as shown in FIG. Figure 1B-1G In a specific example, the one or more input audio signals 115G include one or more enhanced audio signals generated by a second signal enhancer external to the signal enhancer 142 .

[0125] In one example, the one or more enhanced audio signals 133G correspond to one or more equalized signals (e.g., music audio). Optionally, in some implementations, the signal enhancer 142 uses the neural network 152G to perform the equalization 160. Thus, the one or more enhanced audio signals 133G may be one or more monophonic speech signals with equalization performed by applying the neural network 152G. The one or more enhanced monophonic audio signals 143 are based on the one or more enhanced audio signals 133G.

[0126] Figure 2A-6CAspects of various illustrative implementations of audio mixer 148 are shown. Figure 2B , 3B , 4B, 5B and 6B show aspects of an audio mixer 148B that includes a neural network trained to generate one or more stereo audio signals 149 that are respectively generated by Figure 2A , 3A , 4A, 5A and 6A are similar in that the one or more stereo audio signals 149 generated by the audio mixer 148A of FIG. Figure 2C , 3C , 4C, 5C and 6C show examples of neural networks trained to generate one or more stereo audio signals 149, one or more stereo audio signals 149 and one or more stereo audio signals 149 respectively represented by Figure 2A , 3A , 4A, 5A and 6A are similar in that the one or more stereo audio signals 149 generated by the audio mixer 148A of FIG.

[0127] refer to Figure 2A , a schematic diagram showing illustrative aspects of audio mixer 148A. In certain aspects, audio mixer 148A corresponds to Figure 1A Implementation of the audio mixer 148 .

[0128] exist Figure 2A In the example shown in , one or more enhanced audio signals 151 are identical to one or more enhanced mono audio signals 143. For example, enhanced audio signal 151A corresponds to enhanced mono audio signal 143A, enhanced audio signal 151B corresponds to enhanced mono audio signal 143B, and so on.

[0129] The audio mixer 148A generates the audio signal 155 based on the background audio signal 167. For example, the audio mixer 148A applies the delay 202 to the background audio signal 167 to generate the delayed audio signal 203. In some aspects, the amount of the delay 202 is based on Figure 1A user input 103, configuration settings, default delays, or a combination thereof.

[0130] Additionally or alternatively, the audio mixer 148A performs an attenuation operation 216, which includes applying an attenuation factor 215 to the (delayed) audio signal 203 to generate an attenuated audio signal 217. In some aspects, the attenuation factor 215 is based on Figure 1A103, a configuration setting, a default value, or a combination thereof. In other aspects, the attenuation factor generator 226 of the audio mixer 148A determines the attenuation factor 215 based on an audio context indicated by or derived from the one or more input audio signals 125. For example, the attenuation factor generator 226 generates an attenuation factor 215 corresponding to a higher attenuation (e.g., greater sound damping) of the (delayed) audio signal 203 (e.g., corresponding to delayed diffuse noise) in response to determining that the audio context indicates a quieter environment.

[0131] The audio mixer 148A mixes the attenuated audio signal 217 with each of the one or more enhanced audio signals 151 to generate one or more stereo audio signals 149. For example, the audio mixer 148A mixes the attenuated audio signal 217 with each of the enhanced mono audio signal 143A and the enhanced mono audio signal 143B to generate stereo audio signals 149A and 149B. The enhanced mono audio signal 143A may correspond to an enhanced left channel of a stereo audio signal such as a stereo speech signal, and the enhanced mono audio signal 143B may correspond to an enhanced right channel of a stereo audio signal such as a stereo speech signal. The one or more enhanced mono audio signals 143 include a signal-enhanced sound (e.g., a noise-suppressed speech), and the audio signal 155 is based on a background audio signal 167 (e.g., diffuse noise from leaves), rather than based on one or more directional audio signals 165 (e.g., car noise). In this example, the one or more stereo audio signals 149 include speech and diffuse noise (eg, from foliage), but not directional noise (eg, car noise).

[0132] refer to Figure 2B , a schematic diagram showing illustrative aspects of audio mixer 148B is shown. In certain aspects, audio mixer 148B corresponds to Figure 1A Implementation of the audio mixer 148 .

[0133] The audio mixer 148B includes a neural network 258 configured to process Figure 2A The one or more inputs of the audio mixer 148A of the training neural network 258 to generate one or more stereo audio signals 149. In certain aspects, the training neural network 258 is used to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 generated by the training neural network 258. Figure 2AThe audio mixer 148A of the embodiment of the present invention approximates one or more stereo audio signals 149 generated by the audio mixer 148A of the present invention. For example, the audio mixer 148B generates one or more input feature values ​​based on one or more enhanced audio signals 151 (e.g., one or more enhanced mono audio signals 143), the background audio signal 167, one or more input audio signals 125, or a combination thereof, and provides the one or more input feature values ​​to the neural network 258. Audio feature extraction (not shown) can be performed on the one or more input audio signals to extract the one or more input feature values. Exemplary features include, but are not limited to, the amplitude envelope, the root mean square energy, and the zero crossing rate of the signal. The neural network 258 processes the one or more input feature values ​​to generate one or more output feature values ​​of the one or more stereo audio signals 149.

[0134] In certain aspects, the neural network 258 includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. In some implementations, the one or more hidden layers include a fully connected layer, one or more convolutional layers, or both. In some implementations, the one or more hidden layers include at least one recurrent layer, such as a long short-term memory (LSTM) layer, a gated recurrent unit (GRU) layer, or another recurrent neural network structure.

[0135] In a particular implementation, the input layer of the neural network 258 includes at least one input node for each signal input into the neural network 258. For example, the input layer of the neural network 258 may include at least one input node for feature values ​​derived from the enhanced mono audio signal 143A, at least one input node for feature values ​​derived from the enhanced mono audio signal 143B, at least one input node for feature values ​​derived from the background audio signal 167, and optionally at least one input node for feature values ​​derived from each of the input audio signals 125.

[0136] In a specific implementation, the output layer of the neural network 258 includes two nodes, which correspond to a right channel stereo output node and a left channel stereo output node. For example, the left channel stereo output node can output a stereo audio signal 149A, and the right channel stereo output node can output a stereo audio signal 149B.

[0137] During the training of the neural network 258, the audio mixer 148B (or another training component of the device) generates a signal based on the Figure 2AThe loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the present invention with one or more stereo audio signals 149 generated by the neural network 258, and iteratively updating the neural network 258 (e.g., its weights and biases) to reduce the loss metric. In some aspects, the neural network 258 is considered trained when the loss metric satisfies a loss threshold.

[0138] refer to Figure 2C , showing a system configured and trained to perform Figure 1A Schematic diagram of an illustrative neural network 258 for the operation of the audio mixer 148. In some examples, the Figure 2C The neural network 258 is implemented Figure 2A Audio mixer 148A or Figure 2B In some alternative examples, audio mixer 148A or audio mixer 148B may be implemented using conventional audio mixing techniques.

[0139] The neural network 258 is configured to process Figure 2A The neural network 258 is trained to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 by Figure 2A The one or more stereo audio signals 149 generated by the audio mixer 148A are approximated. For example, the neural network 258 may be included in Figure 2B within audio mixer 148B.

[0140] exist Figure 2C In the example shown in , the neural network 258 includes an input layer 270, an output layer 276, and one or more hidden layers 274 coupled between the input layer 270 and the output layer 276. For example, in Figure 2C , hidden layer 274 includes hidden layer 274A coupled to input layer 270 and hidden layer 274B, and includes hidden layer 274B coupled to hidden layer 274A and output layer 276. Figure 2C Two hidden layers 274 are shown in , but the neural network 258 may include less than two hidden layers 274 (e.g., a single hidden layer 274) or more than two hidden layers 274.

[0141] exist Figure 2C , the input layer 270 includes at least one input node for each signal to be input to the neural network 258. Specifically, Figure 2C The input layer 270 includes at least two input nodes to receive the enhanced audio signals 151A and 151B, respectively. Figure 2CIn the example shown in , for example, based on audio signals captured by different subgroups of the one or more microphones 120, the enhanced audio signal 151A corresponds to the enhanced mono audio signal 143A, and the enhanced audio signal 151B corresponds to the enhanced mono audio signal 143B. In addition, Figure 2C The input layer 270 of includes at least one input node for feature values ​​derived from the background audio signal 167 , and optionally includes at least one input node for feature values ​​derived from each of the input audio signals 125 .

[0142] exist Figure 2C , the ellipsis between the nodes of the input layer 270 indicates that although four nodes are shown (one node for each signal to be input to the neural network 258), the input layer 270 may include more than four nodes. For example, one or more of the audio signals 143A, 143B, 167, or 125 may be encoded into a multi-bit feature vector for input to the neural network 258. As an example, the background audio signal 167 may be sampled to generate time window samples. Each time window sample may be converted to a frequency domain signal. Each frequency domain signal may be encoded as a feature vector of N bits (e.g., 16 bits), where N is a non-zero power of 2. In this example, each sample of the background audio signal 167 is represented by 16 bits, and the input layer 270 may include 16 nodes to receive 16 bits of features of the background audio signal 167. In other examples, feature vectors greater than or less than 16 bits are used to represent one or more of the audio signals 143A, 143B, 167, or 125. Furthermore, the feature vectors used to represent each of the audio signals 143A, 143B, 167, or 125 need not be of the same size. For example, the enhanced audio signal 151 may be represented with a higher fidelity (and a correspondingly larger number of bits) than the background audio signal 167. In other words, the neural network 258 may include a larger number of input nodes per input signal for one or more enhanced monophonic audio signals than for other signals (e.g., background audio signals).

[0143] exist Figure 2CIn the example of , hidden layer 274A is fully connected to input layer 270 and hidden layer 274B, and each node of hidden layer 274A is associated with a corresponding one of multiple biases 272A. In some aspects, the bias value of a node tends to shift the output of the node independently of the input of the node. Similarly, hidden layer 274B is fully connected to hidden layer 274A and output layer 276, and each node of hidden layer 274B is optionally associated with one of biases 272B. In other implementations, hidden layer 274 includes other types of interconnection schemes, such as convolutional layer interconnection schemes. Optionally, one or more of hidden layer 274 is a loop layer including feedback connection 278.

[0144] The output layer 276 of the neural network 258 includes at least two nodes corresponding to the output nodes for the features of the stereo audio signal 149A and the stereo audio signal 149B. Optionally, each of the nodes of the output layer 276 may be associated with a corresponding one of the biases 272C. Figure 2C , the ellipsis between the nodes of the output layer 276 indicates that although two nodes are shown (one node for each signal output by the neural network 258), the output layer 276 may include more than two nodes. For example, each of the stereo audio signals 149 may be represented as a multi-bit feature vector in the output of the neural network 258. In this example, the output layer 276 may include at least one node for each bit of the multi-bit feature vector for each stereo audio signal 149.

[0145] During the training of the neural network 258, the audio mixer 148B (or another training component of the device) is based on the Figure 2A The loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the neural network 258 with one or more stereo audio signals 149 generated by the neural network 258, and link weights between the various layers 270, 274, 276 and / or the bias 272 of the neural network 258 are iteratively updated to reduce the loss metric. In some aspects, the neural network 258 is considered trained when the loss metric satisfies a loss threshold.

[0146] refer to Figure 3A , shows another schematic diagram of illustrative aspects of audio mixer 148A. In certain aspects, audio mixer 148A corresponds to Figure 1A Implementation of the audio mixer 148 .

[0147] exist Figure 3AIn the example shown in , one or more enhanced audio signals 151 are identical to one or more enhanced mono audio signals 143. For example, enhanced audio signal 151A corresponds to enhanced mono audio signal 143A, enhanced audio signal 151B corresponds to enhanced mono audio signal 143B, and so on.

[0148] The audio mixer 148A generates the one or more audio signals 155 based on the one or more directional audio signals 165. For example, the audio mixer 148A applies the delay 302 to the one or more directional audio signals 165 to generate the one or more delayed audio signals 303. In some aspects, the amount of the delay 302 is based on Figure 1A user input 103, configuration settings, default delays, or a combination thereof.

[0149] Additionally or alternatively, the audio mixer 148A applies one or more panning operations 316 to the one or more (delayed) audio signals 303 to generate one or more panned audio signals 317. The one or more panned audio signals 317 correspond to the one or more audio signals 155. For example, the panned audio signal 317A and the panned audio signal 317B correspond to the audio signal 155A and the audio signal 155B, respectively. In one example, the panning factor generator 326 of the audio mixer 148A determines the one or more panning factors 315 (e.g., gain, delay, or both) based on the visual context 147, the source direction selection 347, or both. As an example, the visual context 147 indicates that the audio source 184A is located in the direction selected by Figure 1A The source direction selection 347 indicates a selection of a specific location (e.g., an absolute location, a relative location, or both) in the visual scene represented by the image data 127 of the audio source 184A. In another example, the source direction selection 347 indicates a selection of a specific location (e.g., an absolute location, a relative location, or both) of the audio source 184A in the soundscape, the visual scene, or both. By way of example, the source direction selection 347 indicates an audio source direction, an audio source distance, or both relative to an assumed listener in the soundscape, the visual scene, or both.

[0150] In some examples, audio analyzer 140 determines source direction selection 347 based on gesture detection, head tracking, eye gaze direction, user interface input, or a combination thereof. In certain aspects, source direction selection 347 corresponds to Figure 1A The user input 103 indicates a selection of a particular location of the audio source 184A in the soundscape, the visual scene, or both. By way of example, the user input 103 may correspond to at least one of a mouse input, a keyboard input, an eye gaze direction, a head direction, a gesture, a touch screen input, or another type of user selection of a particular location.

[0151] In certain aspects, the particular location indicated by the visual context 147, the source direction selection 347, or both corresponds to an estimated location of the audio source 184A, a target (e.g., desired) location of the audio source 184A, or both. The panning factor generator 326 generates panning factor 315A having a lower gain value than panning factor 315B in response to determining that the particular location (indicated by the visual context 147, the source direction selection 347, or both) corresponds to the left side of the center in the visual scene. The panning factor 315A (e.g., the lower gain value) is used to generate the stereo audio signal 149B (e.g., the right channel signal), and the panning factor 315B (e.g., the higher gain value) is used to generate the stereo audio signal 149A (e.g., the left channel signal). The audio mixer 148A performs a panning operation 316A on a specific delayed audio signal of the one or more delayed audio signals 303 based on the panning factor 315A to generate a panned audio signal 317A. For example, the specific delayed audio signal is based on the directional audio signal 165A representing the voice of the audio source 184A, and the audio mixer 148A performs the panning operation 316A on the specific delayed audio signal to generate the panned audio signal 317A. The audio mixer 148A mixes the enhanced mono audio signal 143B (e.g., the enhanced second microphone sound) with the panned audio signal 317A to generate a stereo audio signal 149B (e.g., the right channel signal).

[0152] In addition, the audio mixer 148A performs a panning operation 316B on the specific delayed audio signal based on the panning factor 315B to generate a panned audio signal 317B. The audio mixer 148A mixes the enhanced mono audio signal 143A with the panned audio signal 317B to generate a stereo audio signal 149A. Therefore, the speech from the audio source 184A is more perceptible in the stereo audio signal 149A (e.g., the left channel signal) than in the stereo audio signal 149B (e.g., the right channel signal). In some implementations, the panning factor 315 changes dynamically over time as the audio source 184A moves from left to right.

[0153] In some implementations, instead of panning the one or more (delayed) audio signals 303, the audio mixer 148A pans the one or more enhanced mono audio signals 143 based on the visual context 147, the source direction selection 347, or both, as described in reference to FIG. Figure 5AIn an example in which one or more stereo audio signals 149 are based on one or more directional audio signals 165 (e.g., one or more delayed audio signals 303) rather than on a background audio signal 167 (e.g., delayed audio signal 203), the stereo audio signals 149 include speech and directional noise, but do not include diffuse noise.

[0154] refer to Figure 3B , shows another schematic diagram of illustrative aspects of audio mixer 148B. In certain aspects, audio mixer 148B corresponds to Figure 1A Implementation of the audio mixer 148 .

[0155] The audio mixer 148B includes a neural network 358 configured to process Figure 3A The one or more inputs of the audio mixer 148A of the training neural network 358 to generate one or more stereo audio signals 149. In certain aspects, the training neural network 358 is used to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 generated by the training neural network 358. Figure 3A The audio mixer 148A of the embodiment of the present invention approximates the one or more stereo audio signals 149 generated by the audio mixer 148A of the embodiment of the present invention. For example, the audio mixer 148B generates one or more input feature values ​​based on the one or more enhanced audio signals 151 (e.g., one or more enhanced mono audio signals 143), one or more directional audio signals 165, the visual context 147, the source direction selection 347, or a combination thereof, and provides the one or more input feature values ​​to the neural network 358. The neural network 358 processes the one or more input feature values ​​to generate one or more output feature values ​​of the one or more stereo audio signals 149.

[0156] In certain aspects, the neural network 358 includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. In some implementations, the one or more hidden layers include a fully connected layer, one or more convolutional layers, or both. In some implementations, the one or more hidden layers include at least one recurrent layer, such as an LSTM layer, a GRU layer, or another recursive neural network structure.

[0157] In a particular implementation, the input layer of the neural network 358 includes at least one input node for each signal input to the neural network 358. For example, the input layer of the neural network 358 may include at least one input node for feature values ​​derived from the enhanced mono audio signal 143A, at least one input node for feature values ​​derived from the enhanced mono audio signal 143B, at least one input node for feature values ​​derived from each of the directional audio signals 165, and optionally at least one input node for feature values ​​derived from the visual context 147 and / or the source direction selection 347.

[0158] In a specific implementation, the output layer of the neural network 358 includes two nodes, which correspond to a right channel stereo output node and a left channel stereo output node. For example, the left channel stereo output node can output a stereo audio signal 149A, and the right channel stereo output node can output a stereo audio signal 149B.

[0159] During the training of the neural network 358, the audio mixer 148B (or another training component of the device) is based on the Figure 3A The loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the present invention to one or more stereo audio signals 149 generated by the neural network 358, and iteratively updating the neural network 358 (e.g., its weights and biases) to reduce the loss metric. In some aspects, the neural network 358 is considered trained when the loss metric satisfies a loss threshold.

[0160] refer to Figure 3C , showing a system configured and trained to perform Figure 3B Another schematic diagram of an illustrative neural network 358 for the operation of the audio mixer 148B. The neural network 358 is configured to process Figure 3A The neural network 358 is trained to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 by Figure 3A The one or more stereo audio signals 149 generated by the audio mixer 148A are approximated. For example, the neural network 358 may be included in Figure 3B within audio mixer 148B.

[0161] exist Figure 3C In the example shown in , the neural network 358 includes an input layer 270, an output layer 276, and one or more hidden layers 274 coupled between the input layer 270 and the output layer 276. For example, in Figure 3C , hidden layer 274 includes hidden layer 274A coupled to input layer 270 and hidden layer 274B, and includes hidden layer 274B coupled to hidden layer 274A and output layer 276. Figure 3C Two hidden layers 274 are shown in , but the neural network 358 may include less than two hidden layers 274 (e.g., a single hidden layer 274) or more than two hidden layers 274.

[0162] exist Figure 3C , the input layer 270 includes at least one input node for each signal to be input to the neural network 358. Specifically, Figure 3CThe input layer 270 includes at least two input nodes to receive the enhanced audio signals 151A and 151B, respectively. Figure 3C In the example shown in , the enhanced audio signal 151A corresponds to the enhanced mono audio signal 143A, and the enhanced audio signal 151B corresponds to the enhanced mono audio signal 143B. In addition, Figure 3C The input layer 270 of includes at least one input node for feature values ​​derived from each of the directional audio signals 165 and at least one input node for feature values ​​derived from the visual context 147 and / or the source direction selection 347 .

[0163] exist Figure 3C , the ellipsis between the nodes of the input layer 270 indicates that although four nodes are shown, the input layer 270 may include more than four nodes. For example, one or more of the audio signals 143A, 143B, or 165 may be encoded into a multi-bit feature vector for input to the neural network 358, as described above with reference to Figure 2C as described.

[0164] exist Figure 3C In the example of , hidden layer 274A is fully connected to input layer 270 and hidden layer 274B, and each node of hidden layer 274A is optionally associated with a corresponding one of multiple biases 272A. Similarly, hidden layer 274B is fully connected to hidden layer 274A and output layer 276, and each node of hidden layer 274B is optionally associated with a corresponding one of biases 272B. In other implementations, hidden layer 274 includes other types of interconnection schemes, such as convolutional layer interconnection schemes. Optionally, one or more of hidden layers 274 is a recurrent layer including feedback connection 278.

[0165] The output layer 276 of the neural network 358 includes at least two nodes corresponding to the output nodes for the features of the stereo audio signal 149A and the stereo audio signal 149B. Optionally, each of the nodes of the output layer 276 may be associated with a corresponding one of the biases 272C. Figure 3C , the ellipsis between the nodes of the output layer 276 indicates that although two nodes are shown (one node for each signal output by the neural network 358), the output layer 276 may include more than two nodes. For example, each of the stereo audio signals 149 may be represented as a multi-bit feature vector in the output of the neural network 358. In this example, the output layer 276 may include at least one node for each bit of the multi-bit feature vector for each stereo audio signal 149.

[0166] During the training of the neural network 358, the audio mixer 148B (or another training component of the device) is based on the Figure 3A The loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the neural network 358 with one or more stereo audio signals 149 generated by the neural network 358, and iteratively updating the link weights between the various layers 270, 274, 276 and / or the bias 272 of the neural network 358 to reduce the loss metric. In some aspects, the neural network 358 is considered trained when the loss metric satisfies a loss threshold.

[0167] refer to Figure 4A , shows another schematic diagram of illustrative aspects of audio mixer 148A. In certain aspects, audio mixer 148A corresponds to Figure 1A Implementation of the audio mixer 148 .

[0168] The audio mixer 148A performs one or more binauralization operations 416 on the one or more enhanced mono audio signals 143 based on the visual context 147, the source direction selection 347, or both to generate one or more binaural audio signals 417. The one or more binaural audio signals 417 correspond to the one or more enhanced audio signals 151. For example, the audio mixer 148A performs a binauralization operation 416A on the enhanced mono audio signal 143A and performs a binauralization operation 416B on the enhanced mono audio signal 143B to generate a binaural audio signal 417A and a binaural audio signal 417B, respectively. The binaural audio signal 417A and the binaural audio signal 417B correspond to the enhanced audio signal 151A and the enhanced audio signal 151B, respectively.

[0169] As an example, visual context 147, source direction selection 347, or both indicate that audio source 184A is in Figure 1A The image data 127 may be used to represent a specific location (e.g., a relative location, an absolute location, or both) in a visual scene represented by the image data 127. Performing the binauralization operation 416A includes applying a head-related transfer function (HRTF) to the enhanced mono audio signal 143A based on the specific location of the audio source 184A to generate a binaural audio signal 417A (e.g., an enhanced binaural signal). The audio mixer 148A mixes the binaural audio signal 417A with one or more delayed audio signals 303 to generate a stereo audio signal 149A, and mixes the binaural audio signal 417B with one or more delayed audio signals 303 to generate a stereo audio signal 149B. As described in reference Figure 3A As described, one or more delayed audio signals 303 may be generated based on the one or more directional audio signals 165 (eg, by applying delay 302 ).

[0170] In this example, the one or more stereo audio signals 149 are based on the one or more directional audio signals 165 (eg, the one or more delayed audio signals 303), rather than on the background audio signals 167 (eg, Figure 2A As a result, the stereo audio signal 149 includes speech and directional noise, but does not include diffuse noise.

[0171] refer to Figure 4B , shows another schematic diagram of illustrative aspects of audio mixer 148B. In certain aspects, audio mixer 148B corresponds to Figure 1A Implementation of the audio mixer 148 .

[0172] The audio mixer 148B includes a neural network 458 configured to process Figure 4A The one or more inputs of the audio mixer 148A of the training neural network 458 to generate one or more stereo audio signals 149. In certain aspects, the training neural network 458 is used to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 generated by the training neural network 458. Figure 4A The audio mixer 148A of the embodiment of the present invention approximates the one or more stereo audio signals 149 generated by the audio mixer 148A of the embodiment of the present invention. For example, the audio mixer 148B generates one or more input feature values ​​based on the one or more enhanced audio signals 151 (e.g., one or more enhanced mono audio signals 143), one or more directional audio signals 165, the visual context 147, the source direction selection 347, or a combination thereof, and provides the one or more input feature values ​​to the neural network 458. The neural network 458 processes the one or more input feature values ​​to generate one or more output feature values ​​of the one or more stereo audio signals 149.

[0173] In certain aspects, the neural network 458 includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. In some implementations, the one or more hidden layers include a fully connected layer, one or more convolutional layers, or both. In some implementations, the one or more hidden layers include at least one recurrent layer, such as an LSTM layer, a GRU layer, or another recursive neural network structure.

[0174] In a particular implementation, the input layer of the neural network 458 includes at least one input node for each signal input to the neural network 458. For example, the input layer of the neural network 458 may include at least one input node for feature values ​​derived from the enhanced mono audio signal 143A, at least one input node for feature values ​​derived from the enhanced mono audio signal 143B, at least one optional input node for feature values ​​derived from the visual context 147 and / or the source direction selection 347, and at least one input node for feature values ​​derived from each of the directional audio signals 165.

[0175] In a specific implementation, the output layer of the neural network 458 includes two nodes, which correspond to a right channel stereo output node and a left channel stereo output node. For example, the left channel stereo output node can output a stereo audio signal 149A, and the right channel stereo output node can output a stereo audio signal 149B.

[0176] During the training of the neural network 458, the audio mixer 148B (or another training component of the device) is based on the Figure 4A The loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the neural network 458 to one or more stereo audio signals 149 generated by the neural network 458, and iteratively updating the neural network 458 (e.g., its weights and biases) to reduce the loss metric. In some aspects, the neural network 458 is considered trained when the loss metric satisfies a loss threshold.

[0177] refer to Figure 4C , showing a system configured and trained to perform Figure 4B Another schematic diagram of an illustrative neural network 458 for the operation of the audio mixer 148B. The neural network 458 is configured to process Figure 4A The one or more inputs of the audio mixer 148A of the embodiment of the present invention are used to generate one or more stereo audio signals 149. The neural network 458 is trained to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 generated by the audio mixer 148A of the embodiment of the present invention. Figure 4A The one or more stereo audio signals 149 generated by the audio mixer 148A are approximated. For example, the neural network 458 may be included in Figure 4B within audio mixer 148B.

[0178] exist Figure 4C In the example shown in , the neural network 458 includes an input layer 270, an output layer 276, and one or more hidden layers 274 coupled between the input layer 270 and the output layer 276. For example, in Figure 4C, hidden layer 274 includes hidden layer 274A coupled to input layer 270 and hidden layer 274B, and includes hidden layer 274B coupled to hidden layer 274A and output layer 276. Figure 4C Two hidden layers 274 are shown in , but the neural network 458 may include less than two hidden layers 274 (e.g., a single hidden layer 274) or more than two hidden layers 274.

[0179] exist Figure 4C , the input layer 270 includes at least one input node for each signal to be input to the neural network 458. Specifically, Figure 4C The input layer 270 includes at least two input nodes to receive the enhanced audio signals 151A and 151B, respectively. Figure 4C In the example shown in , the enhanced audio signal 151A corresponds to the enhanced mono audio signal 143A, and the enhanced audio signal 151B corresponds to the enhanced mono audio signal 143B. In addition, Figure 4C The input layer 270 of comprises at least one input node for feature values ​​derived from each of the directional audio signals 165 and optionally at least one input node for feature values ​​derived from the visual context 147 and / or the source direction selection 347 .

[0180] exist Figure 4C , the ellipsis between the nodes of the input layer 270 indicates that although four nodes are shown, the input layer 270 may include more than four nodes. For example, one or more of the audio signals 143A, 143B, or 165 may be encoded into a multi-bit feature vector for input to the neural network 458, as described above with reference to Figure 2C as described.

[0181] exist Figure 4C In the example of , hidden layer 274A is fully connected to input layer 270 and hidden layer 274B, and each node of hidden layer 274A is optionally associated with a corresponding one of multiple biases 272A. Similarly, hidden layer 274B is fully connected to hidden layer 274A and output layer 276, and each node of hidden layer 274B is optionally associated with a corresponding one of biases 272B. In other implementations, hidden layer 274 includes other types of interconnection schemes, such as convolutional layer interconnection schemes. Optionally, one or more of hidden layers 274 is a recurrent layer including feedback connection 278.

[0182] The output layer 276 of the neural network 458 includes at least two nodes corresponding to the output nodes for the features of the stereo audio signal 149A and the stereo audio signal 149B. Optionally, each of the nodes of the output layer 276 may be associated with a corresponding one of the biases 272C. Figure 4C , the ellipsis between the nodes of the output layer 276 indicates that although two nodes are shown (one node for each signal output by the neural network 458), the output layer 276 may include more than two nodes. For example, each of the stereo audio signals 149 may be represented as a multi-bit feature vector in the output of the neural network 458. In this example, the output layer 276 may include at least one node for each bit of the multi-bit feature vector for each stereo audio signal 149.

[0183] During the training of the neural network 458, the audio mixer 148B (or another training component of the device) is based on the Figure 4A The loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the neural network 458 with one or more stereo audio signals 149 generated by the neural network 458, and iteratively updating the link weights between the various layers 270, 274, 276 and / or the bias 272 of the neural network 458 to reduce the loss metric. In some aspects, the neural network 458 is considered trained when the loss metric satisfies a loss threshold.

[0184] refer to Figure 5A , shows another schematic diagram of illustrative aspects of audio mixer 148A. In certain aspects, audio mixer 148A corresponds to Figure 1A Implementation of the audio mixer 148 .

[0185] The audio mixer 148A performs one or more panning operations 516 on the one or more enhanced mono audio signals 143 based on the visual context 147, the source direction selection 347, or both to generate one or more panned audio signals 517. The one or more panned audio signals 517 correspond to the one or more enhanced audio signals 151. For example, the audio mixer 148A performs a panning operation 516A on the enhanced mono audio signal 143A and a panning operation 516B on the enhanced mono audio signal 143B to generate a panned audio signal 517A and a panned audio signal 517B, respectively. The panned audio signal 517A and the panned audio signal 517B correspond to the enhanced audio signal 151A and the enhanced audio signal 151B, respectively.

[0186] For example, visual context 147, source direction selection 347, or both indicate that audio source 184A is in the Figure 1AThe image data 127 of the audio source 184A is used to determine a specific location (e.g., a relative location, an absolute location, or both) in the visual scene represented by the image data 127 of the audio source 184A. Performing the panning operation 516A includes applying a first panning factor (e.g., a gain, a delay, or both) to the enhanced mono audio signal 143A based on the specific location of the audio source 184A to generate a panned audio signal 517A (e.g., an enhanced panned signal). Similarly, performing the panning operation 516B includes applying a second panning factor (e.g., a gain, a delay, or both) to the enhanced mono audio signal 143B based on the specific location of the audio source 184A to generate a panned audio signal 517B (e.g., an enhanced panned signal).

[0187] The audio mixer 148A includes a reverberation generator 544 that processes one or more directional audio signals 165, background audio signals 167, or both using a reverberation model 554 to generate a reverberation signal 545. The audio mixer 148A mixes each of the one or more panned audio signals 517 with the reverberation signal 545 to generate one or more stereo audio signals 149. For example, the audio mixer 148A mixes each of the panned audio signal 517A and the panned audio signal 517B with the reverberation signal 545 to generate a stereo audio signal 149A and a stereo audio signal 149B, respectively.

[0188] In this example, the one or more stereo audio signals 149 include reverberation based on the one or more directional audio signals 165, the background audio signals 167, or both, but do not include the one or more directional sound signals 165 or the background audio signals 167. As a result, the stereo audio signals 149 include reverberation but do not include background noise (e.g., car noise or wind noise).

[0189] refer to Figure 5B , shows another schematic diagram of illustrative aspects of audio mixer 148B. In certain aspects, audio mixer 148B corresponds to Figure 1A Implementation of the audio mixer 148 .

[0190] The audio mixer 148B includes a neural network 558 configured to process Figure 5A The one or more inputs of the audio mixer 148A of the embodiment of the present invention are used to generate one or more stereo audio signals 149. The neural network 558 is trained to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 generated by the neural network 558. Figure 5AThe audio mixer 148A of the embodiment of the present invention approximates the one or more stereo audio signals 149 generated by the audio mixer 148A of the embodiment of the present invention. For example, the audio mixer 148B generates one or more input feature values ​​based on the one or more enhanced audio signals 151 (e.g., one or more enhanced mono audio signals 143), one or more directional audio signals 165, background audio signals 167, visual context 147, source direction selection 347, or a combination thereof, and provides the one or more input feature values ​​to the neural network 558. The neural network 558 processes the one or more input feature values ​​to generate one or more output feature values ​​of the one or more stereo audio signals 149.

[0191] In certain aspects, the neural network 558 includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. In some implementations, the one or more hidden layers include a fully connected layer, one or more convolutional layers, or both. In some implementations, the one or more hidden layers include at least one recurrent layer, such as an LSTM layer, a GRU layer, or another recurrent neural network structure.

[0192] In a particular implementation, the input layer of the neural network 558 includes at least one input node for each signal input to the neural network 558. For example, the input layer of the neural network 558 may include at least one input node for feature values ​​derived from the enhanced mono audio signal 143A, at least one input node for feature values ​​derived from the enhanced mono audio signal 143B, at least one input node for feature values ​​derived from the visual context 147 and / or the source direction selection 347, and optionally at least one input node for feature values ​​derived from each of the directional audio signals 165, and / or at least one input node for feature values ​​derived from the background audio signal 167.

[0193] In a specific implementation, the output layer of the neural network 558 includes two nodes, which correspond to a right channel stereo output node and a left channel stereo output node. For example, the left channel stereo output node can output a stereo audio signal 149A, and the right channel stereo output node can output a stereo audio signal 149B.

[0194] During the training of the neural network 558, the audio mixer 148B (or another training component of the device) is based on the Figure 5A The loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the present invention to one or more stereo audio signals 149 generated by the neural network 558, and iteratively updating the neural network 558 (e.g., its weights and biases) to reduce the loss metric. In some aspects, the neural network 558 is considered trained when the loss metric satisfies a loss threshold.

[0195] refer to Figure 5C , showing a system configured and trained to perform Figure 5B Another schematic diagram of an illustrative neural network 558 for the operation of the audio mixer 148B. The neural network 558 is configured to process Figure 5A The neural network 558 is trained to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 by the audio mixer 148A. Figure 5A The one or more stereo audio signals 149 generated by the audio mixer 148A are approximated. For example, the neural network 558 may be included in Figure 5B within audio mixer 148B.

[0196] exist Figure 5C In the example shown in , the neural network 558 includes an input layer 270, an output layer 276, and one or more hidden layers 274 coupled between the input layer 270 and the output layer 276. For example, in Figure 5C , hidden layer 274 includes hidden layer 274A coupled to input layer 270 and hidden layer 274B, and includes hidden layer 274B coupled to hidden layer 274A and output layer 276. Figure 5C Two hidden layers 274 are shown in , but the neural network 558 may include less than two hidden layers 274 (e.g., a single hidden layer 274) or more than two hidden layers 274.

[0197] exist Figure 5C , the input layer 270 includes at least one input node for each signal to be input to the neural network 558. Specifically, Figure 5C The input layer 270 includes at least two input nodes to receive the enhanced audio signals 151A and 151B, respectively. Figure 5C In the example shown in , the enhanced audio signal 151A corresponds to the enhanced mono audio signal 143A, and the enhanced audio signal 151B corresponds to the enhanced mono audio signal 143B. In addition, Figure 5C The input layer 270 of FIG. 5 may include at least one input node for feature values ​​derived from the background audio signal 167 and / or at least one input node for feature values ​​derived from each of the directional audio signals 165. The input layer 270 of FIG. 5 further includes at least one input node for feature values ​​derived from the visual context 147 and / or the source direction selection 347.

[0198] exist Figure 5C, the ellipsis between the nodes of the input layer 270 indicates that although five nodes are shown, the input layer 270 may include more than five nodes. For example, one or more of the audio signals 143A, 143B, 167, or 165 may be encoded into a multi-bit feature vector for input to the neural network 558, as described above with reference to Figure 2C as described.

[0199] exist Figure 5C In the example of , hidden layer 274A is fully connected to input layer 270 and hidden layer 274B, and each node of hidden layer 274A is optionally associated with a corresponding one of bias 272A. Similarly, hidden layer 274B is fully connected to hidden layer 274A and output layer 276, and each node of hidden layer 274B is optionally associated with a corresponding one of bias 272B. In other implementations, hidden layer 274 includes other types of interconnection schemes, such as convolutional layer interconnection schemes. Optionally, one or more of hidden layers 274 is a recurrent layer including feedback connection 278.

[0200] The output layer 276 of the neural network 558 includes at least two nodes corresponding to the output nodes for the features of the stereo audio signal 149A and the stereo audio signal 149B. Optionally, each of the nodes of the output layer 276 may be associated with a corresponding one of the biases 272C. Figure 5C , the ellipsis between the nodes of the output layer 276 indicates that although two nodes are shown (one node for each signal output by the neural network 558), the output layer 276 may include more than two nodes. For example, each of the stereo audio signals 149 may be represented as a multi-bit feature vector in the output of the neural network 558. In this example, the output layer 276 may include at least one node for each bit of the multi-bit feature vector for each stereo audio signal 149.

[0201] During the training of the neural network 558, the audio mixer 148B (or another training component of the device) is based on the Figure 5A The loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the neural network 558 with one or more stereo audio signals 149 generated by the neural network 558, and iteratively updating the link weights between the various layers 270, 274, 276 and / or the bias 272 of the neural network 558 to reduce the loss metric. In some aspects, the neural network 558 is considered trained when the loss metric satisfies a loss threshold.

[0202] refer to Fig. 6A , shows another schematic diagram of illustrative aspects of audio mixer 148A. In certain aspects, audio mixer 148A corresponds to Figure 1A Implementation of the audio mixer 148 .

[0203] The audio mixer 148A includes a reverberation generator 644 that uses a reverberation model 654 to generate a synthesized reverberation signal 645 corresponding to the position context 137, the visual context 147, or both (e.g., not as Figure 5A In some implementations, the visual context 147 is based on the surface of the environment (e.g., a wall, a ceiling, a floor, an object, etc.), the room geometry, or both. For example, in response to the visual context 147, the location context 137, or both indicating an environment corresponding to a conference hall, the reverberation model 654 generates a synthetic reverberation signal 645 corresponding to a large room. As another example, the reverberation model 654 generates a synthetic reverberation signal 645 corresponding to a longer reverberation time in response to the visual context 147 indicating a highly reflective surface (e.g., a metal surface) in a larger room. Alternatively, the reverberation model 654 generates a synthetic reverberation signal 645 corresponding to a shorter reverberation time in response to the visual context 147 indicating a less reflective surface (e.g., a fabric surface) or a smaller room.

[0204] The audio mixer 148A combines one or more panned audio signals 517 (e.g., as shown in FIG. Figure 5A 149A and 149B) are mixed with the synthesized reverberation signal 645 to generate one or more stereo audio signals 149. For example, the audio mixer 148A mixes each of the panned audio signal 517A and the panned audio signal 517B with the synthesized reverberation signal 645 to generate the stereo audio signal 149A and the stereo audio signal 149B, respectively.

[0205] In this example, one or more stereo audio signals 149 include reverberation based on position context 137, visual context 147, or both, rather than based on Figure 1A The stereo audio signal 149 includes reverberation of one or more directional audio signals 165 or background audio signals 167. As a result, the stereo audio signal 149 includes reverberation but does not include background noise (eg, car noise or wind noise).

[0206] refer to Figure 6B , shows another schematic diagram of illustrative aspects of audio mixer 148B. In certain aspects, audio mixer 148B corresponds to Figure 1A Implementation of the audio mixer 148 .

[0207] The audio mixer 148B includes a neural network 658 configured to process Fig. 6AThe neural network 658 is trained to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 by the audio mixer 148A. Fig. 6A The audio mixer 148A of the embodiment of the present invention approximates the one or more stereo audio signals 149 generated by the audio mixer 148A of the embodiment of the present invention. For example, the audio mixer 148B generates one or more input feature values ​​based on the one or more enhanced audio signals 151 (e.g., one or more enhanced mono audio signals 143), the position context 137, the visual context 147, the source direction selection 347, or a combination thereof, and provides the one or more input feature values ​​to the neural network 658. The neural network 658 processes the one or more input feature values ​​to generate one or more output feature values ​​of the one or more stereo audio signals 149.

[0208] In certain aspects, the neural network 658 includes an input layer, an output layer, and one or more hidden layers between the input layer and the output layer. In some implementations, the one or more hidden layers include a fully connected layer, one or more convolutional layers, or both. In some implementations, the one or more hidden layers include at least one recurrent layer, such as an LSTM layer, a GRU layer, or another recursive neural network structure.

[0209] In a particular implementation, the input layer of the neural network 658 includes at least one input node for each signal input to the neural network 658. For example, the input layer of the neural network 658 may include at least one input node for feature values ​​derived from the enhanced mono audio signal 143A, at least one input node for feature values ​​derived from the enhanced mono audio signal 143B, and at least one input node for feature values ​​derived from the visual context 147, the source direction selection 347, and / or the position context 137.

[0210] In a specific implementation, the output layer of the neural network 658 includes two nodes, which correspond to a right channel stereo output node and a left channel stereo output node. For example, the left channel stereo output node can output a stereo audio signal 149A, and the right channel stereo output node can output a stereo audio signal 149B.

[0211] During the training of the neural network 658, the audio mixer 148B (or another training component of the device) is based on the Fig. 6AThe loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the present invention to one or more stereo audio signals 149 generated by the neural network 658, and iteratively updating the neural network 658 (e.g., its weights and biases) to reduce the loss metric. In some aspects, the neural network 658 is considered trained when the loss metric satisfies a loss threshold.

[0212] refer to Figure 6C , showing a system configured and trained to perform Figure 6B Another schematic diagram of an illustrative neural network 658 for the operation of the audio mixer 148B. The neural network 658 is configured to process Fig. 6A The neural network 658 is trained to generate one or more stereo audio signals 149, and the one or more stereo audio signals 149 are combined with the one or more stereo audio signals 149 by the audio mixer 148A. Fig. 6A The one or more stereo audio signals 149 generated by the audio mixer 148A are approximated. For example, the neural network 658 may be included in Figure 6B within audio mixer 148B.

[0213] exist Figure 6C In the example shown in , the neural network 658 includes an input layer 270, an output layer 276, and one or more hidden layers 274 coupled between the input layer 270 and the output layer 276. For example, in Figure 6C , hidden layer 274 includes hidden layer 274A coupled to input layer 270 and hidden layer 274B, and includes hidden layer 274B coupled to hidden layer 274A and output layer 276. Figure 6C Two hidden layers 274 are shown in , but the neural network 658 may include less than two hidden layers 274 (e.g., a single hidden layer 274) or more than two hidden layers 274.

[0214] exist Figure 6C , the input layer 270 includes at least one input node for each signal to be input to the neural network 658. Specifically, Figure 6C The input layer 270 includes at least two input nodes to receive the enhanced audio signals 151A and 151B, respectively. Figure 6C In the example shown in , the enhanced audio signal 151A corresponds to the enhanced mono audio signal 143A, and the enhanced audio signal 151B corresponds to the enhanced mono audio signal 143B. In addition, Figure 6C The input layer 270 of includes at least one input node for feature values ​​derived from the visual context 147 , the source direction selection 347 , and / or the position context 137 .

[0215] exist Figure 6C , the ellipsis between the nodes of the input layer 270 indicates that although three nodes are shown, the input layer 270 may include more than three nodes. For example, one or more of the audio signals 143A or 143B may be encoded into a multi-bit feature vector for input to the neural network 658, as described above with reference to Figure 2C as described.

[0216] exist Figure 6C In the example of , hidden layer 274A is fully connected to input layer 270 and hidden layer 274B, and each node of hidden layer 274A is optionally associated with a corresponding one of bias 272A. Similarly, hidden layer 274B is fully connected to hidden layer 274A and output layer 276, and each node of hidden layer 274B is optionally associated with one of bias 272B. In other implementations, hidden layer 274 includes other types of interconnection schemes, such as convolutional layer interconnection schemes. Optionally, one or more of hidden layers 274 is a recurrent layer including feedback connection 278.

[0217] The output layer 276 of the neural network 658 includes at least two nodes corresponding to the output nodes for the features of the stereo audio signal 149A and the stereo audio signal 149B. Optionally, each of the nodes of the output layer 276 may be associated with a corresponding one of the biases 272C. Figure 6C , the ellipsis between the nodes of the output layer 276 indicates that although two nodes are shown (one node for each signal output by the neural network 658), the output layer 276 may include more than two nodes. For example, each of the stereo audio signals 149 may be represented as a multi-bit feature vector in the output of the neural network 658. In this example, the output layer 276 may include at least one node for each bit of the multi-bit feature vector for each stereo audio signal 149.

[0218] During the training of the neural network 658, the audio mixer 148B (or another training component of the device) is based on the Fig. 6A The loss metric is generated by comparing one or more stereo audio signals 149 generated by the audio mixer 148A of the neural network 658 with one or more stereo audio signals 149 generated by the neural network 658, and iteratively updating the link weights between the various layers 270, 274, 276 and / or the bias 272 of the neural network 658 to reduce the loss metric. In some aspects, the neural network 658 is considered trained when the loss metric satisfies a loss threshold.

[0219] It should be understood that Figure 2A-6CNon-limiting illustrative aspects of implementations of the audio mixer 148 are provided. Other implementations of the audio mixer 148 may include various other aspects. By way of example, in some examples, the audio mixer 148 may generate a stereo audio signal 149A by mixing one of the one or more audio signals 155 with multiple of the one or more enhanced audio signals 151.

[0220] refer to Figure 7 , shows a schematic diagram 700 of a device 102 operable to perform audio signal enhancement on one or more input audio signals 125 based on encoded data 749 received from a device 704. The device 102 includes an audio analyzer 140 coupled to a receiver 740.

[0221] During operation, the receiver 740 receives encoded data 749 from the device 704. The encoded data 749 represents one or more input audio signals 125, image data 127, position data 163, or a combination thereof. For example, the encoded data 749 represents sounds 185 of one or more audio sources 184, images of one or more audio sources 184, positions of audio scenes associated with the sounds 185, or a combination thereof. In some implementations, a decoder of the device 102 decodes the encoded data 749 to generate one or more input audio signals 125, image data 127, position data 163, or a combination thereof.

[0222] As reference Figure 1A As described, the audio analyzer 140 processes the one or more input audio signals 125 based on the image data 127, the position data 163, or both to generate one or more stereo audio signals 149. The audio analyzer 140 provides the one or more stereo audio signals 149 to one or more speakers 722 to output sound 785. Figure 1A As depicted, the one or more stereo audio signals 149 (eg, sound 785 ) include less noise than the one or more input audio signals 125 and more audio context than the one or more enhanced mono audio signals 143 .

[0223] Figure 8 An example of a device operable to transmit encoded data based on an enhanced audio signal according to some examples of the present disclosure is shown.

[0224] refer to Figure 8 , a schematic diagram 800 of a device 102 operable to send encoded data 849 to a device 804 is shown. Figure 1AAs depicted, the encoded data 849 corresponds to one or more stereo audio signals 149, and the one or more stereo audio signals 149 are based on one or more enhanced mono audio signals 143. The device 102 includes an audio analyzer 140 coupled to the transmitter 840.

[0225] During operation, the audio analyzer 140 obtains one or more input audio signals 125, image data 127, or both. In some aspects, the one or more input audio signals 125 correspond to microphone outputs of the one or more microphones 120, and the image data 127 corresponds to camera outputs of the one or more cameras 130. The one or more input audio signals 125 represent sounds 185 from an audio source 184. The audio analyzer 140 processes the one or more input audio signals 125, the image data 127, or both to generate one or more stereo audio signals 149. For example, as described in reference Figure 1A As depicted, the audio analyzer 140 performs signal enhancement to generate one or more enhanced mono audio signals 143 , and processes the one or more enhanced mono audio signals 143 to generate one or more stereo audio signals 149 .

[0226] Transmitter 840 sends encoded data 849 to device 804. Encoded data 849 is based on one or more stereo audio signals 149. In some examples, an encoder of device 102 encodes one or more stereo audio signals 149 to generate encoded data 849.

[0227] The decoder of the device 804 decodes the encoded data 849 to generate a decoded audio signal. The device 804 provides the decoded audio signal to one or more speakers 822 to output sound 885.

[0228] In some implementations, device 804 is the same as device 704. For example, device 102 receives encoded data 749 from a second device and outputs sound 785 via one or more speakers 722, while simultaneously capturing sound 185 via one or more microphones 120 and sending encoded data 849 to the second device.

[0229] Fig. 9An implementation 900 of the device 102 is depicted as an integrated circuit 902 that includes one or more processors 190. The integrated circuit 902 also includes an audio input 904 (such as one or more bus interfaces) to enable one or more input audio signals 125 to be received for processing. In some aspects, the integrated circuit 902 includes an image input 903 (such as one or more bus interfaces) to enable image data 127 to be received for processing. The integrated circuit 902 also includes an audio output 906 (such as a bus interface) to enable audio signals, such as one or more stereo audio signals 149 to be transmitted. The integrated circuit 902 enables audio signal enhancement as a component in a system, such as: Fig.10 A mobile phone or tablet device, such as Fig.11 The headphones depicted in Fig.12 Wearable electronic devices such as those depicted in Fig.13 The voice-controlled speaker system depicted in Fig.14 The camera depicted in Fig.15 A virtual reality, mixed reality, or augmented reality headset as depicted in Fig.16 or Fig.17 The vehicle depicted in .

[0230] Fig.10 An implementation 1000 is depicted in which the device 102 includes a mobile device 1002 (such as a phone or tablet device) as an illustrative, non-limiting example. The mobile device 1002 includes one or more microphones 120, one or more cameras 130, one or more speakers 722, and a display screen 1004. Components of one or more processors 190 including an audio analyzer 140 are integrated into the mobile device 1002 and are shown using dashed lines to indicate internal components that are generally not visible to a user of the mobile device 1002. In a specific example, user voice activity is detected in one or more stereo audio signals 149 generated by the audio analyzer 140, and the user voice activity is processed to perform one or more operations at the mobile device 1002, such as launching a graphical user interface or otherwise displaying other information associated with the user's voice at the display screen 1004 (e.g., via an integrated "smart assistant" application).

[0231] Fig.11An implementation 1100 is depicted in which the device 102 includes a headphone device 1102. The headphone device 1102 includes one or more microphones 120, one or more cameras 130, one or more speakers 722, or a combination thereof. Components of the one or more processors 190 including the audio analyzer 140 are integrated into the headphone device 1102. In a specific example, user voice activity is detected in one or more stereo audio signals 149 generated by the audio analyzer 140, which can cause the headphone device 1102 to perform one or more operations at the headphone device 1102 to send audio data corresponding to the user voice activity to a second device (not shown) (such as Figure 7 Device 704 or Figure 8 device 804), for further processing, or a combination thereof.

[0232] Fig.12 An implementation 1200 is shown in which the device 102 includes a wearable electronic device 1202, which is shown as a "smart watch". An audio analyzer 140, one or more microphones 120, one or more cameras 130, one or more speakers 722, or a combination thereof are integrated into the wearable electronic device 1202. In a specific example, user voice activity is detected in one or more enhanced mono audio signals 143 generated by the audio analyzer 140, and the user voice activity is processed to perform one or more operations at the wearable electronic device 1202, such as launching a graphical user interface or otherwise displaying other information associated with the user's voice at a display screen 1204 of the wearable electronic device 1202. By way of example, the wearable electronic device 1202 may include a display screen 1204 configured to display a notification based on the user's voice detected by the wearable electronic device 1202. In a specific example, the wearable electronic device 1202 includes a haptic device that provides a tactile notification (e.g., a vibration) in response to the detection of the user voice activity. For example, the tactile notification can cause the user to look at the wearable electronic device 1202 to see the displayed notification indicating that a keyword spoken by the user has been detected. Therefore, the wearable electronic device 1202 can remind the hearing-impaired user or the user wearing a headset that the user's voice activity has been detected.

[0233] Fig.131300 is an implementation in which the device 102 includes a wireless speaker and voice activated device 1302. The wireless speaker and voice activated device 1302 may have a wireless network connection and is configured to perform auxiliary operations. One or more processors 190 including an audio analyzer 140, one or more microphones 120, one or more cameras 130, or a combination thereof are included in the wireless speaker and voice activated device 1302. The wireless speaker and voice activated device 1302 also includes one or more speakers 722. During operation, in response to receiving a verbal command recognized as a user's voice in one or more enhanced mono audio signals 143 generated by the audio analyzer 140, the wireless speaker and voice activated device 1302 may perform an auxiliary operation, such as via executing a voice activated system (e.g., an integrated auxiliary application). The auxiliary operation may include adjusting the temperature, playing music, turning on the lights, etc. For example, the auxiliary operation is performed in response to receiving a command following a keyword or key phrase (e.g., "Hello, assistant").

[0234] Fig.14 An implementation 1400 is depicted in which device 102 includes a portable electronic device corresponding to camera device 1402. In some aspects, camera device 1402 includes Figure 1A The one or more cameras 130 of the audio analyzer 140. The audio analyzer 140, the one or more microphones 120, the one or more cameras 130, the one or more speakers 722, or a combination thereof are included in the camera device 1402. During operation, in response to receiving a verbal command recognized as a user voice in the one or more enhanced mono audio signals 143 generated by the audio analyzer 140, the camera device 1402 can perform operations responsive to the verbal user command, such as adjusting image or video capture settings, image or video playback settings, or image or video capture instructions, as illustrative examples.

[0235] Fig.15 An implementation 1500 is depicted in which the device 102 includes a portable electronic device corresponding to a virtual reality, mixed reality, or augmented reality headset 1502. The audio analyzer 140, the one or more microphones 120, the one or more cameras 130, the one or more speakers 722, or a combination thereof are integrated into the headset 1502. In some examples, the headset 1502 receives the encoded data 749, processes the encoded data 742 to generate one or more stereo audio signals 149, and provides the one or more stereo audio signals 149 to the one or more speakers 722, as described with reference to FIG. Figure 7In some aspects, user voice activity detection can be performed on one or more enhanced mono audio signals 143 generated by the audio analyzer 140. The visual interface device is positioned in front of the user's eyes to enable the display of augmented reality, mixed reality, or virtual reality images or scenes to the user when the headset 1502 is worn. In a specific example, the visual interface device is configured to display a notification indicating the user's voice detected in the audio signal.

[0236] Fig.16 An implementation 1600 is shown in which the device 102 corresponds to or is integrated within a vehicle 1602, which is shown as a manned or unmanned aerial device (e.g., a package delivery drone). An audio analyzer 140, one or more microphones 120, one or more cameras 130, one or more speakers 722, or a combination thereof are integrated into the vehicle 1602. User voice activity detection may be performed on one or more enhanced mono audio signals 143 generated by the audio analyzer 140, such as for delivery instructions from an authorized user of the vehicle 1602.

[0237] Fig.17Another implementation 1700 is depicted, wherein the device 102 corresponds to a vehicle 1702 or is integrated in the vehicle 1702, and the vehicle 1702 is shown as a car. The vehicle 1702 includes one or more processors 190, which include an audio analyzer 140. The vehicle 1702 also includes one or more microphones 120, one or more cameras 130, one or more speakers 722, or a combination thereof. User voice activity detection can be performed on one or more enhanced monophonic audio signals 143 generated by the audio analyzer 140. In some implementations, one or more input audio signals 125 are received from an internal microphone (e.g., one or more microphones 120), such as for voice commands from authorized passengers. For example, user voice activity detection can be used to detect voice commands from the operator of the vehicle 1702 (e.g., voice commands from parents to set the volume to 5 or set a destination for the autonomous vehicle), and ignore the voice of another passenger (e.g., voice commands from children to set the volume to 10 or discuss other passengers at another location). In some implementations, one or more input audio signals 125 are received from an external microphone (e.g., one or more microphones 120), such as from an authorized user of the vehicle 1702. In a particular implementation, in response to recognizing a spoken command in one or more enhanced mono audio signals 143 generated by the audio analyzer 140, the voice activation system initiates one or more operations of the vehicle 1702 based on one or more keywords (e.g., "unlock," "start engine," "play music," "show weather forecast," or another voice command) detected in the one or more enhanced mono audio signals 143, such as by providing feedback or information via the display 1720 or one or more speakers (e.g., one or more speakers 722).

[0238] refer to Fig.18 , a specific implementation of the method 1800 for audio signal enhancement is shown. In certain aspects, one or more operations of the method 1800 are performed by Figure 1A The method may be performed by at least one of the audio analyzer 140, the processor 190, the device 102, the system 100, or a combination thereof.

[0239] At 1802, method 1800 includes performing signal enhancement on an input audio signal to generate an enhanced mono audio signal. Figure 1A As described, Figure 1A The signal enhancer 142 generates one or more enhanced mono audio signals 143.

[0240] At 1804, the method 1800 further includes mixing the first audio signal and the second audio signal to generate a stereo audio signal, the first audio signal being based on the enhanced mono audio signal. Figure 1A As depicted, the audio mixer 148 mixes the one or more enhanced audio signals 151 with the one or more audio signals 155 to generate one or more stereo audio signals 149. The one or more enhanced audio signals 151 are based on the one or more enhanced mono audio signals 143.

[0241] The method 1800 balances the signal enhancement of the enhanced monophonic audio signal with the directional context associated with the second audio signal when generating the stereo audio signal. For example, one or more stereo audio signals 149 may include directional noise or reverberation, which provides an audio context to the listener while removing at least some (e.g., diffuse noise or all background noise) of the background noise.

[0242] Fig.18 The method 1800 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a controller, another hardware device, a firmware device, or any combination thereof. As an example, Fig.18 The method 1800 may be performed by a processor executing instructions, such as reference Fig.19 as described.

[0243] refer to Fig.19 , depicts a block diagram of a particular illustrative implementation of a device and generally designated 1900. In various implementations, device 1900 may have the same Fig.19 In an illustrative implementation, device 1900 may correspond to device 102. In an illustrative implementation, device 1900 may execute the reference Figure 1A-18 Describes one or more operations.

[0244] In certain implementations, the device 1900 includes a processor 1906 (e.g., a CPU). The device 1900 may include one or more additional processors 1910 (e.g., one or more DSPs, one or more GPUs, or a combination thereof). In certain aspects, Figure 1A The one or more processors 190 may correspond to processor 1906, processor 1910, or a combination thereof. Processor 1910 may include a speech and music encoder-decoder (CODEC) 1908 including a speech encoder ("vocoder") encoder 1936, a vocoder decoder 1938, an audio analyzer 140, or a combination thereof.

[0245] The device 1900 may include a memory 1986 and a CODEC 1934. The memory 1986 may include instructions 1956 executable by one or more additional processors 1910 (or processor 1906) to implement the functionality described with reference to the audio analyzer 140. The device 1900 may include a modem 1970 coupled to an antenna 1952 via a transceiver 1950.

[0246] The device 1900 may include a display 1928 coupled to a display controller 1926. One or more speakers 722, one or more microphones 120, or a combination thereof may be coupled to a CODEC 1934. The CODEC 1934 may include a digital-to-analog converter (DAC) 1902, an analog-to-digital converter (ADC) 1904, or both. In a particular implementation, the CODEC 1934 may receive an analog signal from the one or more microphones 120, convert the analog signal to a digital signal using the analog-to-digital converter 1904, and provide the digital signal to a speech and music codec 1908. The speech and music codec 1908 may process the digital signal, and the digital signal may be further processed by the audio analyzer 140. In a particular implementation, the speech and music codec 1908 may provide the digital signal to the CODEC 1934. The CODEC 1934 may convert the digital signal to an analog signal using the digital-to-analog converter 1902, and may provide the analog signal to the one or more speakers 722.

[0247] In a particular implementation, device 1900 may be included in a system-in-package or system-on-chip device 1922. In a particular implementation, memory 1986, processor 1906, processor 1910, display controller 1926, CODEC 1934, and modem 1970 are included in the system-in-package or system-on-chip device 1922. In a particular implementation, input device 1930 and power supply 1944 are coupled to the system-in-package or system-on-chip device 1922. In addition, in a particular implementation, as shown in FIG. Fig.19 As shown, the display 1928, input device 1930, one or more speakers 722, one or more microphones 120, antenna 1952, and power supply 1944 are external to the system-in-package or system-on-chip device 1922. In a particular implementation, each of the display 1928, input device 1930, one or more speakers 722, one or more microphones 120, antenna 1952, and power supply 1944 may be coupled to a component of the system-in-package or system-on-chip device 1922, such as an interface or controller.

[0248] Device 1900 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet device, a personal digital assistant, a display device, a television, a game console, a music player, a radio unit, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, headphones, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aircraft, a home automation system, a voice-activated device, a wireless speaker and voice-activated device, a portable electronic device, an automobile, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0249] In conjunction with the described implementation, an apparatus includes a unit for performing signal enhancement on an input audio signal to generate an enhanced mono audio signal. For example, the unit for performing signal enhancement may correspond to Figure 1A The neural network 152, the signal enhancer 142, the audio mixer 148, the audio analyzer 140, one or more processors 190, the device 102, the system 100, one or more other circuits or components configured to perform signal enhancement, or any combination thereof.

[0250] The apparatus further comprises a unit for mixing a first audio signal and a second audio signal to generate a stereo audio signal, the first audio signal being based on the enhanced mono audio signal. For example, the unit for mixing may correspond to Figure 1A The audio mixer 148, the audio analyzer 140, the one or more processors 190, the device 102, the system 100, Figure 2A , 3A , audio mixer 148A for 4A, 5A and 6A, Figure 2B , 3B , audio mixer 148B for 4B, 5B and 6B, Figure 2C , 3C , the neural network of any of Figures 4C, 5C or 6C, one or more other circuits or components configured to mix the first audio signal and the second audio signal, or any combination thereof.

[0251] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 1986) includes instructions (e.g., instructions 1956) that, when executed by one or more processors (e.g., one or more processors 1910 or processor 1906), cause the one or more processors to perform signal enhancement on an input audio signal (e.g., one or more input audio signals 125) to generate an enhanced mono audio signal (e.g., one or more enhanced mono audio signals 143). The instructions, when executed by the one or more processors, also cause the one or more processors to mix a first audio signal (e.g., one or more enhanced audio signals 151) and a second audio signal (e.g., one or more audio signals 155) to generate a stereo audio signal (e.g., one or more stereo audio signals 149). The first audio signal is based on the enhanced mono audio signal.

[0252] Certain aspects of the present disclosure are described below in a set of interrelated examples:

[0253] According to Example 1, a device includes: a processor configured to: perform signal enhancement on an input audio signal to generate an enhanced mono audio signal; and mix a first audio signal and a second audio signal to generate a stereo audio signal, wherein the first audio signal is based on the enhanced mono audio signal.

[0254] Example 2 includes the apparatus of Example 1, wherein the second audio signal is associated with a context of the input audio signal.

[0255] Example 3 includes a device according to Example 1 or Example 2, wherein the processor is configured to perform the signal enhancement using a neural network.

[0256] Example 4 includes the apparatus of any one of Examples 1 to 3, wherein the input audio signal is based on microphone output of one or more microphones.

[0257] Example 5 includes a device according to any one of Examples 1 to 4, wherein the processor is configured to decode encoded audio data to generate the input audio signal.

[0258] Example 6 includes a device according to any one of Examples 1 to 5, wherein the signal enhancement includes at least one of noise suppression, audio scaling, beamforming, dereverberation, source separation, bass adjustment, or equalization.

[0259] Example 7 includes the apparatus of any one of Examples 1 to 6, wherein the signal enhancement is based at least in part on a configuration setting, a user input, or both.

[0260] Example 8 includes a device according to any one of Examples 1 to 7, wherein the processor is configured to use a neural network to mix the first audio signal and the second audio signal to generate the stereo audio signal.

[0261] Example 9 includes a device according to any one of Examples 1 to 8, wherein the processor is configured to: use a first neural network to perform signal enhancement on an input audio signal to generate the enhanced mono audio signal; and use a second neural network to mix the first audio signal and the second audio signal.

[0262] Example 10 includes a device according to any one of Examples 1 to 9, wherein the processor is configured to: perform signal enhancement on a second input audio signal to generate a second enhanced mono audio signal; and generate the stereo audio signal based on mixing the first audio signal, the second audio signal and a third audio signal, the third audio signal being based on the second enhanced mono audio signal.

[0263] Example 11 includes a device according to any one of Examples 1 to 10, wherein the processor is configured to generate at least one directional audio signal based on the input audio signal, and wherein the second audio signal is based on the at least one directional audio signal.

[0264] Example 12 includes a device according to any one of Examples 1 to 11, wherein the processor is configured to: generate a directional audio signal based on the input audio signal; and apply a delay to the directional audio signal to generate a delayed audio signal, wherein the second audio signal is based on the delayed audio signal.

[0265] Example 13 includes the device of Example 12, wherein the processor is configured to pan the delayed audio signal based on a visual context to generate the second audio signal.

[0266] Example 14 includes a device according to any one of Examples 1 to 13, wherein the processor is configured to pan the enhanced mono audio signal to generate the first audio signal.

[0267] Example 15 includes the device of Example 14, wherein the processor is configured to receive a user selection of an audio source direction, and wherein the enhanced mono audio signal is panned based on the audio source direction.

[0268] Example 16 includes a device according to Example 15, wherein the processor is configured to determine the user selection based on gesture detection, head tracking, eye gaze detection, user interface input, or a combination thereof.

[0269] Example 17 includes a device according to Example 15 or Example 16, wherein the processor is configured to apply a head-related transfer function (HRTF) to the enhanced monophonic audio signal based on the audio source direction to generate the first audio signal.

[0270] Example 18 includes a device according to any one of Examples 1 to 17, wherein the processor is configured to generate a background audio signal from an input audio signal, wherein the second audio signal is at least partially based on the background audio signal.

[0271] Example 19 includes the apparatus of Example 18, wherein the processor is configured to: apply a delay to the background audio signal to generate a delayed background audio signal; and attenuate the delayed background audio signal to generate the second audio signal.

[0272] Example 20 includes the device of Example 19, wherein the processor is configured to attenuate the delayed background audio signal based on a visual context to generate the second audio signal.

[0273] Example 21 includes a device according to any one of Examples 18 to Example 20, wherein the processor is configured to: generate at least one directional audio signal from the input audio signal; and use a reverberation model to process the background audio signal, the at least one directional audio signal, or a combination thereof to generate a reverberation signal, wherein the second audio signal includes the reverberation signal.

[0274] Example 22 includes a device according to any one of Examples 1 to Example 20, wherein the processor is configured to: determine a visual context of the input audio signal based on image data, the image data representing a visual scene associated with an audio source of the input audio signal; and use a reverberation model to generate a synthetic reverberation signal corresponding to the visual context, wherein the second audio signal includes the synthetic reverberation signal.

[0275] Example 23 includes a device according to Example 22, wherein the visual context is based on a surface of the environment, a room geometry, or both.

[0276] Example 24 includes an apparatus according to Example 22 or Example 23, wherein the image data is based on at least one of a camera output, a graphics visual stream, decoded image data, or stored image data.

[0277] Example 25 includes a device according to any one of Examples 22 to 24, wherein the processor is configured to determine the visual context based at least in part on performing facial detection on the image data.

[0278] Example 26 includes a device according to any one of Examples 1 to 20, wherein the processor is configured to: determine a location context based on the location data; and use a reverberation model to generate a synthetic reverberation signal corresponding to the location context, wherein the second audio signal includes the synthetic reverberation signal.

[0279] According to Example 27, a method includes: performing signal enhancement on an input audio signal at a device to generate an enhanced mono audio signal; and mixing a first audio signal and a second audio signal at the device to generate a stereo audio signal, the first audio signal being based on the enhanced mono audio signal.

[0280] Example 28 includes the method of Example 27, wherein the second audio signal is associated with a context of the input audio signal.

[0281] Example 29 includes the method according to Example 27 or Example 28, further comprising: using a neural network to perform the signal enhancement.

[0282] Example 30 includes a method according to any one of Examples 27 to 29, wherein the input audio signal is based on microphone output of one or more microphones.

[0283] Example 31 includes a method according to any one of Examples 27 to 30, further comprising: decoding the encoded audio data to generate the input audio signal.

[0284] Example 32 includes a method according to any one of Examples 27 to Example 31, wherein the signal enhancement includes at least one of noise suppression, audio scaling, beamforming, de-reverberation, source separation, bass adjustment, or equalization.

[0285] Example 33 includes the method of any one of Examples 27 to 32, wherein the signal enhancement is based at least in part on a configuration setting, a user input, or both.

[0286] Example 34 includes the method of any one of Examples 27 to 33, further comprising: using a neural network to mix the first audio signal and the second audio signal to generate the stereo audio signal.

[0287] Example 35 includes a method according to any one of Examples 27 to Example 34, further comprising: using a first neural network to perform signal enhancement on an input audio signal to generate the enhanced mono audio signal; and using a second neural network to mix the first audio signal and the second audio signal.

[0288] Example 36 includes a method according to any one of Examples 27 to Example 35, further comprising: performing signal enhancement on a second input audio signal to generate a second enhanced mono audio signal; and generating the stereo audio signal based on mixing the first audio signal, the second audio signal and a third audio signal, the third audio signal being based on the second enhanced mono audio signal.

[0289] Example 37 includes a method according to any one of Examples 27 to Example 36, further comprising: generating at least one directional audio signal based on the input audio signal, and wherein the second audio signal is based on the at least one directional audio signal.

[0290] Example 38 includes a method according to any one of Examples 27 to Example 37, further comprising: generating a directional audio signal based on the input audio signal; and applying a delay to the directional audio signal to generate a delayed audio signal, wherein the second audio signal is based on the delayed audio signal.

[0291] Example 39 includes the method of Example 38, further comprising: panning the delayed audio signal based on a visual context to generate the second audio signal.

[0292] Example 40 includes the method according to any one of Examples 27 to 39, further comprising: panning the enhanced mono audio signal to generate the first audio signal.

[0293] Example 41 includes the method of Example 40, further comprising receiving a user selection of an audio source direction, wherein the enhanced mono audio signal is panned based on the audio source direction.

[0294] Example 42 includes the method according to Example 41, further comprising: determining the user selection based on gesture detection, head tracking, eye gaze detection, user interface input, or a combination thereof.

[0295] Example 43 includes the method according to Example 41 or Example 42, further comprising: applying a head-related transfer function (HRTF) to the enhanced monophonic audio signal based on the audio source direction to generate the first audio signal.

[0296] Example 44 includes a method according to any one of Examples 27 to Example 43, further comprising: generating a background audio signal from the input audio signal, wherein the second audio signal is at least partially based on the background audio signal.

[0297] Example 45 includes the method of Example 44, further comprising: applying a delay to the background audio signal to generate a delayed background audio signal; and attenuating the delayed background audio signal to generate the second audio signal.

[0298] Example 46 includes the method of Example 45, further comprising attenuating the delayed background audio signal based on a visual context to generate the second audio signal.

[0299] Example 47 includes a method according to any one of Examples 44 to 46, further comprising: generating at least one directional audio signal from the input audio signal; and using a reverberation model to process the background audio signal, the at least one directional audio signal, or a combination thereof to generate a reverberation signal, wherein the second audio signal includes the reverberation signal.

[0300] Example 48 includes a method according to any one of Examples 27 to Example 46, further comprising: determining a visual context of the input audio signal based on image data, the image data representing a visual scene associated with an audio source of the input audio signal; and using a reverberation model to generate a synthetic reverberation signal corresponding to the visual context, wherein the second audio signal includes the synthetic reverberation signal.

[0301] Example 49 includes a method according to Example 48, wherein the visual context is based on a surface of the environment, a room geometry, or both.

[0302] Example 50 includes a method according to Example 48 or Example 49, wherein the image data is based on at least one of a camera output, a graphics visual stream, decoded image data, or stored image data.

[0303] Example 51 includes a method according to any one of Examples 48 to Example 50, further comprising: determining the visual context based at least in part on performing facial detection on the image data.

[0304] Example 52 includes a method according to any one of Examples 27 to Example 46, further comprising: determining a location context based on the location data; and using a reverberation model to generate a synthetic reverberation signal corresponding to the location context, wherein the second audio signal includes the synthetic reverberation signal.

[0305] According to Example 53, a device includes: a memory configured to store instructions; and a processor, the processor configured to execute the instructions to perform a method according to any one of Examples 27 to 52.

[0306] According to Example 54, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform a method according to any one of Examples 27 to 52.

[0307] According to Example 55, an apparatus includes a unit for performing a method according to any one of Examples 27 to 52.

[0308] According to Example 56, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations: perform signal enhancement on an input audio signal to generate an enhanced mono audio signal; and mix a first audio signal and a second audio signal to generate a stereo audio signal, wherein the first audio signal is based on the enhanced mono audio signal.

[0309] Example 57 includes the non-transitory computer-readable medium of Example 56, wherein the second audio signal is associated with a context of the input audio signal.

[0310] According to Example 58, an apparatus comprises: a unit for performing signal enhancement on an input audio signal to generate an enhanced mono audio signal; and a unit for mixing a first audio signal and a second audio signal to generate a stereo audio signal, wherein the first audio signal is based on the enhanced mono audio signal.

[0311] Example 59 includes an apparatus according to Example 58, wherein the unit for performing the signal enhancement and the unit for mixing the first audio signal and the second audio signal are integrated into at least one of the following: a smart speaker, a bar speaker, a computer, a tablet device, a display device, a television, a game console, a music player, a radio unit, a digital video player, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aircraft, a home automation system, a voice-activated device, a wireless speaker and a voice-activated device, a portable electronic device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, or a mobile device.

[0312] It will also be appreciated by the skilled person that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the implementation disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above around their functions. Whether such functions are implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art can implement the described functions in a varying manner for each specific application, and such implementation decisions will not be interpreted as causing a departure from the scope of the present disclosure.

[0313] The steps of the method or algorithm described in conjunction with the implementation disclosed herein can be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module can reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, a storage medium can be integrated into a processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). The ASIC can reside in a computing device or a user terminal. Alternatively, the processor and the storage medium can be located in a computing device or a user terminal as discrete components.

[0314] The previous description of the disclosed aspects is provided to enable those skilled in the art to implement or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but is to be given the widest possible scope consistent with the principles and novel features defined by the claims that follow.

Claims

1. A device comprising: A processor configured to: performing signal enhancement on the input audio signal to generate an enhanced mono audio signal; as well as A first audio signal and a second audio signal are mixed to generate a stereo audio signal, the first audio signal being based on the enhanced mono audio signal.

2. The device according to claim 1, wherein The second audio signal is associated with a context of the input audio signal.

3. The device according to claim 1, wherein: The processor is configured to perform the signal enhancement using a neural network.

4. The device according to claim 1, wherein: The input audio signal is based on microphone output of one or more microphones.

5. The device according to claim 1, wherein: The processor is configured to decode encoded audio data to generate the input audio signal.

6. The device according to claim 1, wherein: The signal enhancement includes at least one of noise suppression, audio scaling, beamforming, dereverberation, source separation, bass adjustment, or equalization.

7. The device according to claim 1, wherein: The processor is configured to mix the first audio signal and the second audio signal using a neural network to generate the stereo audio signal.

8. The device according to claim 1, wherein: The processor is configured to: performing signal enhancement on the input audio signal using a first neural network to generate the enhanced monophonic audio signal; and The first audio signal and the second audio signal are mixed using a second neural network.

9. The device according to claim 1, wherein: The processor is configured to: performing signal enhancement on the second input audio signal to generate a second enhanced mono audio signal; and The stereo audio signal is generated based on mixing the first audio signal, the second audio signal and a third audio signal, the third audio signal being based on the second enhanced mono audio signal.

10. The device according to claim 1, wherein: The processor is configured to generate at least one directional audio signal based on the input audio signal, and wherein the second audio signal is based on the at least one directional audio signal.

11. The device according to claim 1, wherein: The processor is configured to: generating a directional audio signal based on the input audio signal; and A delay is applied to the directional audio signal to generate a delayed audio signal, wherein the second audio signal is based on the delayed audio signal.

12. The device according to claim 11, wherein The processor is configured to pan the delayed audio signal based on a visual context to generate the second audio signal.

13. The apparatus according to claim 1, wherein: The processor is configured to pan the enhanced mono audio signal to generate the first audio signal.

14. The device according to claim 13, wherein: The processor is configured to receive a user selection of an audio source direction, and wherein the enhanced mono audio signal is panned based on the audio source direction.

15. The device according to claim 14, wherein: The processor is configured to determine the user selection based on gesture detection, head tracking, eye gaze detection, user interface input, or a combination thereof.

16. The apparatus according to claim 14, wherein: The processor is configured to apply a head-related transfer function (HRTF) to the enhanced monophonic audio signal based on the audio source direction to generate the first audio signal.

17. The apparatus according to claim 1, wherein: The processor is configured to generate an background audio signal from an input audio signal, wherein the second audio signal is based at least in part on the background audio signal.

18. The apparatus according to claim 17, wherein: The processor is configured to: applying a delay to the background audio signal to generate a delayed background audio signal; and The delayed background audio signal is attenuated to generate the second audio signal.

19. The apparatus according to claim 18, wherein: The processor is configured to attenuate the delayed background audio signal based on a visual context to generate the second audio signal.

20. The apparatus of claim 17, wherein: The processor is configured to: generating at least one directional audio signal from the input audio signal; and The background audio signal, the at least one directional audio signal, or a combination thereof are processed using a reverberation model to generate a reverberation signal, wherein the second audio signal includes the reverberation signal.

21. The apparatus of claim 1, wherein: The processor is configured to: determining a visual context of the input audio signal based on image data, the image data representing a visual scene associated with an audio source of the input audio signal; as well as A synthetic reverberation signal corresponding to the visual context is generated using a reverberation model, wherein the second audio signal includes the synthetic reverberation signal.

22. The apparatus according to claim 21, wherein The visual context is based on the surfaces of the acoustic environment, the room geometry, or both.

23. The apparatus of claim 21, wherein: The image data is based on at least one of a camera output, a graphics visual stream, decoded image data, or stored image data.

24. The apparatus of claim 21, wherein: The processor is configured to determine the visual context based at least in part on performing facial detection on the image data.

25. A method comprising: performing, at the device, signal enhancement of the input audio signal to generate an enhanced monophonic audio signal; as well as A first audio signal and a second audio signal are mixed at the device to generate a stereo audio signal, the first audio signal being based on the enhanced monophonic audio signal.

26. The method according to claim 25, further comprising: determining a location context based on the location data; as well as A synthetic reverberation signal corresponding to the location context is generated using a reverberation model, wherein the second audio signal includes the synthetic reverberation signal.

27. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: performing signal enhancement on the input audio signal to generate an enhanced mono audio signal; and A first audio signal and a second audio signal are mixed to generate a stereo audio signal, the first audio signal being based on the enhanced mono audio signal.

28. The non-transitory computer readable medium of claim 27, wherein: The signal enhancement is based at least in part on configuration settings, user input, or both.

29. An apparatus comprising: a unit for performing signal enhancement on an input audio signal to generate an enhanced monophonic audio signal; as well as Unit for mixing a first audio signal and a second audio signal to generate a stereo audio signal, the first audio signal being based on the enhanced mono audio signal.

30. The device according to claim 29, wherein: The unit for performing the signal enhancement and the unit for mixing the first audio signal and the second audio signal are integrated into at least one of the following: a smart speaker, a bar speaker, a computer, a tablet device, a display device, a television, a game console, a music player, a radio unit, a digital video player, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aircraft, a home automation system, a voice-activated device, a wireless speaker and a voice-activated device, a portable electronic device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, or a mobile device.

Images