Audio processing using sound source representations

By using sound source combination representation processing technology in small computing devices and using neural networks to generate masks, the problems of waste of resources and poor user experience in multi-sound source processing are solved, and more efficient and accurate audio signal processing is achieved.

CN118891674AActive Publication Date: 2024-11-01QUALCOMM INC
View PDF 9 Cites 0 Cited by

Patent Information

Application Number
CN202380026687.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-18
Filing Date
2023-03-16
Publication Date
2024-11-01
Estimated Expiration
2043-03-16

AI Technical Summary

Technical Problem

When existing small computing devices process audio signals from multiple sound sources, it is difficult to effectively distinguish between interesting and uninterested sound sources, resulting in waste of resources and poor user experience.

Method used

Using a combined representation processing technology based on sound sources, the sound source is selectively retained or removed through a combined representation of multiple sound sources, and a mask is generated using a neural network to enhance the perceptibility of the sound of interest in the output audio signal. .

Benefits of technology

Improves processing efficiency and accuracy, enhances the perceptibility of the sound of interest, and reduces the consumption of computing resources and bandwidth usage.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118891674A_ABST
    Figure CN118891674A_ABST
Patent Text Reader

Abstract

An apparatus includes one or more processors configured to receive an input audio signal. The one or more processors are further configured to process the input audio signal based on the combined representation of the plurality of sound sources to generate an output audio signal. The combined representation is used to selectively retain or remove sound of the plurality of sound sources from the input audio signal. The one or more processors are further configured to provide the output audio signal to a second device.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] I. Cross-Reference to Related Applications

[0002] This application claims the benefit of priority from commonly owned U.S. non-provisional patent application No. 17 / 655,511 filed on March 18, 2022, the entire contents of which are expressly incorporated herein by reference.

[0003] II. Technical field

[0004] The present disclosure generally relates to processing audio based on representations of sound sources.

[0005] III. Related Fields

[0006] Technological advances have led to smaller and more powerful computing devices. For example, there are a variety of portable personal computing devices, including small, lightweight and user-friendly wireless phones (such as mobile and smart phones, tablet computers and laptop computers). These devices can communicate voice and data packets over wireless networks. In addition, many such devices incorporate additional functionality, such as digital still cameras, digital video cameras, digital recorders, and audio file players. In addition, such devices can process executable instructions, including software applications, such as web browser applications, which can be used to access the Internet. As such, these devices can include significant computing power.

[0007] Such computing devices often incorporate the functionality of receiving audio signals from one or more microphones. For example, audio signals may include sounds from multiple sound sources. In some cases, only sounds from some of these sound sources are of interest. In such cases, audio from uninteresting sound sources may be processed or sent together with audio from an interested sound source, which may result in a less than satisfactory user experience, inefficient use of resources (e.g., processor time or transmission bandwidth), or both.

[0008] IV. Summary of the Invention

[0009] According to a specific implementation of the present disclosure, a device includes one or more processors configured to receive an input audio signal. The one or more processors are also configured to process the input audio signal based on a combination representation of multiple sound sources to generate an output audio signal. The combination representation is used to selectively retain or remove the sounds of multiple sound sources from the input audio signal. The one or more processors are further configured to provide the output audio signal to a second device.

[0010] According to another specific implementation of the present disclosure, a method includes receiving an input audio signal at a first device. The method also includes processing the input audio signal based on a combined representation of multiple sound sources to generate an output audio signal. The combined representation is used to selectively retain or remove sounds from multiple sound sources from the input audio signal. The method also includes providing the output audio signal to a second device.

[0011] According to another specific implementation of the present disclosure, a non-transitory computer-readable medium includes instructions, which, when executed by one or more processors, cause the one or more processors to receive an input audio signal at a first device. When the instructions are executed by one or more processors, the one or more processors are further caused to process the input audio signal based on a combined representation of multiple sound sources to generate an output audio signal. The combined representation is used to selectively retain or remove sounds from multiple sound sources from the input audio signal. When the instructions are executed by one or more processors, the one or more processors are further caused to provide an output audio signal to a second device.

[0012] According to another specific implementation of the present disclosure, an apparatus includes a component for receiving an input audio signal at a first device. The apparatus also includes a component for processing the input audio signal based on a combined representation of multiple sound sources to generate an output audio signal. The combined representation is used to selectively retain or remove sounds from multiple sound sources from the input audio signal. The apparatus also includes a component for providing an output audio signal to a second device.

[0013] Other aspects, advantages, and features of the present disclosure will become apparent upon review of the entire application, including the Brief Description of the Drawings, Detailed Description, and Claims.

[0014] V. Description of the Figures

[0015] Figure 1 is a block diagram of certain illustrative aspects of a system operable to process audio based on sound source representations according to some examples of the present disclosure.

[0016] Figure 2A is a diagram of illustrative aspects of a sound source encoder operable to generate a sound source representation according to some examples of the present disclosure.

[0017] Figure 2B is a diagram of another illustrative aspect of a sound source encoder operable to generate a sound source representation according to some examples of the present disclosure.

[0018] Figure 2C is a diagram of another illustrative aspect of a sound source encoder operable to generate a sound source representation according to some examples of the present disclosure.

[0019] Figure 3 According to some examples of the present disclosure Figure 1 Illustration of illustrative aspects of a neural network of a system.

[0020] Figure 4 According to some examples of the present disclosure Figure 1 Illustration of illustrative aspects of operations associated with joint training of a neural network and sound source encoder of a system.

[0021] Figure 5 According to some examples of the present disclosure Figure 1 00261 00262 002634 002645 002654 FIGURE 2 is a diagram of another illustrative aspect of operations associated with joint training of a neural network and a sound source encoder of a system.

[0022] Figure 6 An example of a device operable to process audio received from another device using a sound source representation in accordance with some examples of the present disclosure is illustrated.

[0023] Figure 7 Examples of devices operable to transmit audio processed based on a sound source representation to another device according to some examples of the present disclosure are illustrated.

[0024] Figure 8 According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of the operation of components of a system.

[0025] Fig. 9 Examples of an integrated circuit operable to process audio based on a sound source representation in accordance with some examples of the present disclosure are illustrated.

[0026] Fig.10 is a diagram of a mobile device operable to process audio based on a sound source representation according to some examples of the present disclosure.

[0027] Fig.11 is a diagram of a headset operable to process audio based on a sound source representation according to some examples of the present disclosure.

[0028] Fig.12 is a diagram of a wearable electronic device operable to process audio based on a sound source representation according to some examples of the present disclosure.

[0029] Fig.13 is a diagram of a voice-controlled speaker system operable to process audio based on a sound source representation according to some examples of the present disclosure.

[0030] Fig.14 is a diagram of a camera operable to process audio based on a sound source representation according to some examples of the present disclosure.

[0031] Fig.15 is a diagram of a headset (such as a virtual reality, mixed reality, or augmented reality headset) operable to process audio based on a sound source representation in accordance with some examples of the present disclosure.

[0032] Fig.16 is a diagram of a first example of a vehicle operable to process audio based on a sound source representation in accordance with some examples of the present disclosure.

[0033] Fig.17 is a diagram of a second example of a vehicle operable to process audio based on a sound source representation in accordance with some examples of the present disclosure.

[0034] Fig.18 According to some examples of the present disclosure, Figure 1 A diagram of a specific implementation of a method for processing audio based on sound source representation performed by a device.

[0035] Fig.19 is a block diagram of a specific illustrative example of a device operable to process audio based on a sound source representation according to some examples of the present disclosure.

[0036] VI. Specific Implementation Methods

[0037] The input audio signal may include sounds from multiple sound sources. Sounds from only some of the multiple sound sources may be interesting. For example, during a conference call, the voices of the conference call participants may be interesting, while the voices of people speaking in the background, non-speech noise, etc. may be distracting. In another example, the audio signal includes the voices of one or more known users and the voices of unknown people, and the voices of the unknown people are interesting. Retaining or removing the sounds of the selected sound sources may enhance the perceptibility of the sounds of interest.

[0038] Disclosed are systems and methods for processing audio based on sound source representations. For example, an input audio signal is processed based on sound representations of one or more sound sources to generate an output audio signal. Sound source representations are used to retain or remove sounds of one or more sound sources from the input audio signal. For example, a sound source representation of the voice of a participant in a conference call can be used to retain the voice of the participant in the input audio signal without retaining other sounds. As another example, a sound source representation of a known user can be used to remove the voice of a known user in the input audio signal without removing other sounds, including the voice of an unknown user. Therefore, the perceptibility of sounds of interest (such as conference call participants or unknown people) is enhanced in the output audio signal.

[0039] Certain aspects of the present disclosure are described below with reference to the accompanying drawings. In this specification, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some implementations and plural in other implementations. For example, Figure 1 Depicts a system including one or more processors ( Figure 1 190), indicating that in some implementations, the device 102 includes a single processor 190, while in other implementations, the device 102 includes multiple processors 190. For ease of reference herein, such features are generally introduced as "one or more" features and are subsequently referred to in the singular unless aspects are described that relate to multiple ones of the features.

[0040] As used herein, the term "including" can be used interchangeably with "comprising". In addition, the term "wherein" can be used interchangeably with "wherein". As used herein, "exemplary" indicates examples, specific implementations and / or aspects, and should not be interpreted as restrictive or indicating a preference or preferred specific implementation. As used herein, ordinal terms (e.g., "first", "second", "third", etc.) used to modify elements (such as, structures, components, operations, etc.) do not themselves indicate any priority or order of the element relative to another element, but only distinguish the element from another element with the same name (but using ordinal terms). As used herein, the term "set" refers to one or more specific elements in a specific element, and the term "plurality" refers to multiple (e.g., two or more) specific elements.

[0041] As used herein, "coupling" may include "communicatively coupled", "electrically coupled" or "physically coupled", and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device or in different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling. In some specific implementations, two devices (or components) that are communicatively coupled (such as electrically connected) may directly or indirectly transmit and receive signals (e.g., digital signals or analog signals) via one or more wires, buses, networks, etc. As used herein, "direct coupling" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled or physically coupled) without intermediate components.

[0042] In the present disclosure, terms such as "determine", "calculate", "estimate", "shift", "adjust", etc. may be used to describe how to perform one or more operations. It should be noted that such terms should not be interpreted as limiting, and other technologies may be used to perform similar operations. In addition, as mentioned herein, "generate", "calculate", "estimate", "use", "select", "access" and "determine" may be used interchangeably. For example, "generating", "calculating", "estimating" or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating or determining the parameter (or signal), or may refer to using, selecting or accessing a parameter (or signal) that has already been generated (such as by another component or device).

[0043] refer to Figure 1 , discloses certain illustrative aspects of a system configured to process audio based on a sound source representation and generally designated as 100. System 100 includes a device 102 coupled to one or more microphones 120, one or more speakers 160, or a combination thereof. Device 102 includes one or more processors 190 coupled to a memory 132.

[0044] The memory 132 stores one or more sound source representations (SSRs) 154, such as sound source representation (SSR) 154A, sound source representation 154B, sound source representation 154C, one or more additional sound source representations, or a combination thereof. For example, sound source representation 154A represents the sound (e.g., speech, non-speech sound, or both) of sound source 184A. In certain aspects, sound source representation 154A is based on the sound of sound source 184A or on the sound of a particular sound source of the same sound source type as sound source 184A, such as reference 184A. Figure 2Aas further described.

[0045] In some aspects, the memory 132 stores one or more combined sound source representations 147, such as combined sound source representation 147A, combined sound source representation 147B, combined sound source representation 147C, one or more additional combined sound source representations, or combinations thereof. For example, combined sound source representation 147A represents sounds (e.g., speech, non-speech sounds, or both) of multiple sound sources, such as sound source 184A, sound source 184B, and sound source 184C. In a particular aspect, combined sound source representation 147A is based on sound source representations 154A, sound source representation 154B, and sound source representation 154C that represent sounds from sound source 184A, sounds from sound source 184B, and sounds from sound source 184C, respectively. In another aspect, combined sound source representation 147A is based on sounds from sound source 184A, sound source 184B, and sound source 184C or sounds from sound sources of the same type as sound source 184A, sound source 184B, and sound source 184C, as described in reference to FIG. Figure 2B to Figure 2C as further described.

[0046] In some aspects, a sound source representation (e.g., sound source representation 154A or combined sound source representation 147B) can represent the sounds of a sound source in a particular environment, such as reference Figure 2C As further described. For example, combined sound source representation 147B may represent sounds (e.g., wind, road noise, etc.) captured in a vehicle of a particular type (e.g., brand, model, sedan, sport utility vehicle (SUV), van, electric vehicle, gas-powered vehicle, hybrid vehicle, etc.) while traveling on a highway. In some aspects, one or more of the sound source representations 154, one or more of the combined sound source representations 147, or both are associated with metadata indicating corresponding sound source type information (such as demographic information, object type information, vehicle type information, person identifier, sound source identifier, environmental information, etc.).

[0047] The one or more processors 190 include an audio analyzer 140 configured to process audio based on a sound source representation. The audio analyzer 140 includes a configurator 144 coupled to an audio adjuster 148. The configurator 144 is configured to determine an adjuster configuration setting 143 based on a context 149.

[0048] The adjuster configuration setting 143 indicates the value of the retain flag 145 and indicates the selected sound source 162. For example, a first value (e.g., 1) of the retain flag 145 indicates that the sound of the selected sound source 162 will be retained. A second value (e.g., 0) of the retain flag 145 indicates that the sound of the selected sound source 162 will be removed.

[0049] The context 149 indicates that when the detected condition 159 matches the activation condition 139, the audio adjuster 148 will be activated using the adjuster configuration settings 143 determined based on the sound source selection criteria 157 and the retention mark criteria 137. In some aspects, the activation condition 139, the sound source selection criteria 157, the retention mark criteria 137, or a combination thereof are based on a default configuration, a user input 103, a configuration input from an application, a configuration request from another device, or a combination thereof. The configurator 144 is configured to determine the selected sound source 162 based on the sound source selection criteria 157 and determine the value of the retention mark 145 based on the retention mark criteria 137.

[0050] Additionally, in some aspects, the configurator 144 is configured to provide the combined SSR 147A representing the sound of the selected sound source 162 to the audio adjuster 148. In some examples, the configurator 144 is configured to provide the one or more selected sound source representations 156 of the selected sound source 162 to the SSR generator 146. The SSR generator 146 is configured to generate the combined SSR 147A based on the one or more selected sound source representations 156 and store the combined sound source representation 147A in the memory 132. In these examples, the SSR generator 146 is configured to provide the combined SSR 147A to the audio adjuster 148.

[0051] The audio conditioner 148 is configured to use the neural network 150 to process the input audio signal 126 based on the combined sound source representation 147A to generate the output audio signal 135. For example, the audio conditioner 148 is configured to use the neural network 150 to generate the mask 151, as shown in FIG. Figure 3 As further described. The audio adjuster 148 generates an output audio signal 135 by applying a mask 151 to the input audio signal 126 to retain or remove the sound of the selected sound source 162 based on the value of the retention flag 145. By selectively retaining or removing the sound corresponding to the selected sound source 162, the perceptibility of the sound of interest is enhanced in the output audio signal 135. Compared to processing the input audio signal 126 using a separate sound source representation, using a combined sound source representation 147A that can represent multiple sound sources to generate the mask 151 can be faster and use less calculations. In some aspects, the neural network 150 includes a convolutional neural network (CNN), an autoregressive (AR) generation network, an audio generation network (AGN), an attention network (AN), a long short-term memory (LSTM) network, or a combination thereof.

[0052] In some implementations, the device 102 corresponds to or is included in one of various types of devices. In the illustrative example, the one or more processors 190 are integrated into a headphone device that includes one or more microphones 120, one or more speakers 160, or a combination thereof, such as the one described in the reference example. Fig.11 In other examples, the one or more processors 190 are integrated into at least one of the following: Fig.10 The mobile phone or tablet computer device, as referenced Fig.12 The wearable electronic device, as shown in reference Fig.13 The voice controlled speaker system, as shown in reference Fig.14 The camera device or as referenced Fig.15 In another illustrative example, the one or more processors 190 are integrated into a vehicle that also includes one or more microphones 120, one or more speakers 160, or a combination thereof, such as the one described in the reference Fig.16 and Fig.17 as further described.

[0053] During operation, the configurator 144 determines a context 149 indicating that the audio adjuster 148 will be activated when the detected condition 159 matches the activation condition 139 to retain or remove the sound of the selected sound source 162 from the input audio signal 126. The context 149 indicates that the sound source that meets the sound source selection criteria 157 will be the selected sound source 162, and the sound will be retained or removed based on the retention marking criteria 137. The context 149 (e.g., the activation condition 139, the sound source selection criteria 157, and the retention marking criteria 137) is based on a default configuration, a user input 103, a configuration input from an application, a configuration request from another device, or a combination thereof.

[0054] In some examples, user 101 provides user input 103 to audio analyzer 140. Audio analyzer 140 determines context 149 based at least in part on user input 103. In a first illustrative example, user input 103 corresponds to the creation or acceptance of a meeting invitation for a scheduled meeting. In a particular aspect, user input 103 indicates that the voice of one or more participants of the scheduled meeting will be retained. On the other hand, the default configuration indicates that the voice of any participant of the scheduled meeting will be retained. Configurator 144 generates context 149 in response to receiving user input 103 to indicate activation condition 139, which indicates that audio adjuster 148 will be activated to initiate processing of input audio signal 126 in response to the start of the scheduled meeting. Configurator 144 also generates context 149 to indicate sound source selection criteria 157 and retention mark criteria 137, which sound source selection criteria indicate that the selected sound source 162 will correspond to the participant of the scheduled meeting, and the retention mark criteria indicate that retention mark 145 will have a first value (e.g., 1).

[0055] In the second illustrative example, the user input 103 corresponds to a user selection in the audio processing application to activate the audio adjuster 148 to remove the sound of the known user (e.g., sound source 184A, sound source 184B, and sound source 184C) from the input audio signal 126. The configurator 144 generates a context 149 in response to receiving the user input 103 to indicate that the detected condition 159 (e.g., receiving the user input 103) matches the activation condition 139 for activating the audio adjuster 148. The configurator 144 also updates the context 149 to indicate that the selected sound source 162 will include the sound source selection criteria 157 of the known user (e.g., sound source 184A, sound source 184B, and sound source 184C) and the retention mark criteria 137 that the retention mark 145 will have a second value (e.g., 0). For ease of description, the operation of the audio adjuster 148 is described herein with reference to the first illustrative example and the second illustrative example. It should be understood that various other examples of the operation of the audio adjuster 148 to retain or remove sounds are possible. To illustrate, in one example, user input 103 corresponds to activation of a recording application. Configurator 144 generates context 149 in response to receiving user input 103 to indicate that detected condition 159 (e.g., receiving user input 103) matches activation condition 139 for activating audio adjuster 148. Configurator 144 also updates context 149 to indicate that selected sound source 162 will correspond to sound source selection criteria 157 of an estimated target for recording and retention flag 145 will have retention flag criteria 137 of a first value (e.g., 1).

[0056] The audio adjuster 148, when activated, initiates processing of the input audio signal 126. For example, the audio adjuster 148 uses the sound source representation of the selected sound source 162 to retain or remove sounds from the input audio signal 126, as described further below. In some implementations, prior to activation of the audio adjuster 148, the configurator 144 retrieves a sound source representation of at least one sound source that may be included in the selected sound sources 162 (e.g., sound sources that are expected to satisfy the sound source selection criteria 157). For example, the configurator 144 retrieves a sound source representation 154A (e.g., representing speech) of the sound source 184A from the server in response to determining that the selected sound sources 162 may include the sound source 184A and stores the sound source representation 154A locally at the device 102 (e.g., in the device's memory 132). In a first illustrative example, configurator 144 determines that sound source 184A may satisfy sound source selection criteria 157 (e.g., may be included in selected sound sources 162) in response to determining that a meeting invitation has been transmitted to sound source 184A for a scheduled meeting. In a second illustrative example, configurator 144 determines that sound source 184A may satisfy sound source selection criteria 157 (e.g., may be included in selected sound sources 162) in response to determining that a known user includes sound source 184A.

[0057] In some implementations, prior to activation of the audio adjuster 148, the configurator 144 generates (or updates) an SSR for at least one sound source that may be included in the selected sound sources 162. For example, the configurator 144 generates the sound source representation 154B based on the input audio signal representing the sound of the sound source 184B in response to determining that the selected sound sources 162 may include the sound source 184B, as shown in FIG. Figure 2A As further described, by way of example, an input audio signal is received during a conference call between user 101 and sound source 184B.

[0058] In some implementations, the configurator 144 generates (or updates) the SSR of one or more sound sources 184 (e.g., at least one of the selected sound sources 162) when the audio adjuster 148 is activated. For example, the configurator 144 generates (or updates) the sound source representation 154B based on the portion of the input audio signal 126 in response to determining that the portion of the input audio signal 126 corresponds to a single talker and the single talker is the sound source 184B, as shown in FIG. Figure 2A as further described.

[0059] The input audio signal 126 corresponds to a sound 186 from a sound source 184, such as sound source 184A, sound source 184B, sound source 184C, sound source 184D, sound source 184E, one or more additional sound sources, or a combination thereof. The sound source 184 may include one or more of a vehicle, an emergency vehicle, traffic, wind, reverberation, channel distortion, a bird, an animal, a siren, another non-voice sound source, a person, an authorized user, another voice source, or an audio player.

[0060] The audio analyzer 140 receives the input audio signal 126 from another device (e.g., a server or a storage device), one or more microphones 120, the memory 132, or a combination thereof. In a first illustrative example, the audio analyzer 140 receives the input audio signal 126 from the server during a scheduled meeting. For example, the input audio signal 126 is based on audio data received from another device (e.g., a server), such as reference Figure 6 In the second illustrative example, the audio analyzer 140 receives the input audio signal 126 from the one or more microphones 120 . By way of example, the one or more microphones 120 capture the sound 186 , and the input audio signal 126 is based on microphone outputs of the one or more microphones 120 .

[0061] In some examples, the input audio signal 126 is based on audio data retrieved from the memory 132. By way of example, the input audio signal 126 is based on audio data generated by an application (such as a music application, a gaming application, a graphics application, an augmented reality application, a communication application, an entertainment application, or a combination thereof) of the one or more processors 190. In some examples, the audio adjuster 148 processes the input audio signal 126 in real time as the audio analyzer 140 receives the input audio signal 126. In other examples, the input audio signal 126 corresponds to a previously generated audio signal.

[0062] The configurator 144 determines that the context 149 indicates that when the detected condition 159 matches the activation condition 139, the audio adjuster 148 will be activated with the adjuster configuration settings 143 based on the sound source selection criteria 157 and the retention mark criteria 137. In a first illustrative example, the configurator 144 activates the audio adjuster 148 in response to determining that the detected condition 159 (e.g., the start of a scheduled meeting) matches the activation condition 139 (e.g., the start of a scheduled meeting). The configurator 144 determines the adjuster configuration settings 143 based on the context 149. For example, the configurator 144 designates a participant (e.g., an expected participant, a detected participant, or a combination thereof) as the selected sound source 162 and sets the retention mark 145 to have a first value (e.g., 1) indicating that the corresponding sound is to be retained in response to determining that the sound source selection criteria 157 indicates that the selected sound source 162 will correspond to a participant of the scheduled meeting and the retention mark criteria 137 indicates that the retention mark 145 will have a first value (e.g., 1).

[0063] The selected sound sources 162 may be determined statically (e.g., expected participants), dynamically (e.g., detected participants), or both using the sound source selection criteria 157. In a first illustrative example, the static determination of the selected sound sources 162 may include the expected participants of the scheduled meeting, and detected additional participants (e.g., participants who join the meeting despite not being initially invited) may be dynamically added to the selected sound sources 162.

[0064] In a second illustrative example, the configurator 144 activates the audio adjuster 148 in response to determining that the context 149 indicates that the detected condition 159 (e.g., receiving the user input 103 indicating a user selection in the audio processing application) matches the activation condition 139. The configurator 144 determines the adjuster configuration setting 143 based on the context 149. For example, the configurator 144 activates the audio adjuster 148 in response to determining that the sound source selection criteria 157 indicate that the selected sound source 162 will correspond to a known user (e.g., sound source 184A, sound source 184B, and sound source 184C) and the retention flag criteria 137 indicate that the retention flag 145 will have a second value (e.g., 0), designates the known user as the selected sound source 162, and sets the retention flag 145 to have a second value (e.g., 0) indicating that the corresponding sound is to be removed.

[0065] The selected sound sources 162 may be determined statically (e.g., all of the known users), dynamically (e.g., detected of the known users), or both using the sound source selection criteria 157. In a second illustrative example, the static determination of the selected sound sources 162 may include all of the known users, and one or more of the known users (e.g., whose voices have not been detected for a threshold duration) may be dynamically removed from the selected sound sources 162.

[0066] The value of the retention mark 145 can be determined statically or dynamically using the retention mark criteria 137. In a first illustrative example, the retention mark criteria 137 corresponds to a static determination of the value of the retention mark 145 (e.g., a first value that retains the sound). In a second illustrative example, the retention mark criteria 137 corresponds to a static determination of the value of the retention mark 145 (e.g., a second value that removes the sound). In various aspects, the retention mark criteria 137 can be used to dynamically determine the value of the retention mark 145. In one example, the user input 103 corresponds to the activation of the recording application. The configurator 144 generates a context 149 in response to receiving the user input 103 to indicate that the detected condition 159 (e.g., receiving the user input 103) matches the activation condition 139 that activates the audio adjuster 148. The configurator 144 also updates the context 149 to indicate the sound source selection criteria 157, that is, in the case where the sound source representation of the estimated target is available, the selected sound source 162 will correspond to one or more estimated targets of the recording. The sound source selection criteria 157 indicates that, in the case where the sound source of the estimated target indicates that it is unavailable, and in the case where the sound source of the interfering sound source indicates that it is available, the selected sound source 162 will correspond to one or more interfering sound sources. The retention mark criteria 137 indicates that the retention mark 145 will have a first value (e.g., 1) of retaining sound when the selected sound source 162 corresponds to the estimated target. The retention mark criteria 137 also indicates that the retention mark 145 will have a second value (e.g., 0) of removing sound when the selected sound source 162 corresponds to the interfering sound source. In this example, the sound source selection criteria 157 is used to dynamically determine the selected sound source 162, and the retention mark criteria 137 is used to dynamically determine the value of the retention mark 145.

[0067] The configurator 144, upon activating the audio adjuster 148, concurrently provides the audio adjuster 148 with the value of the reserved flag 145 and the combined sound source representation 147A of the selected sound source 162. In some examples, the configurator 144 has access to the combined sound source representation 147A and provides the combined sound source representation 147A to the audio adjuster 148 (e.g., bypassing the SSR generator 146). In other examples, the configurator 144 provides the SSR generator 146 with one or more selected sound source representations 156 of the selected sound source 162, and the SSR generator 146 provides the combined sound source representation 147A to the audio adjuster 148.

[0068] In some implementations, the adjuster configuration setting 143 includes a combo. setting 164. A first value (e.g., 0) of the combo. setting 164 indicates that the configurator 144 will provide the SSR generator 146 with individual sound source representations of the selected sound sources 162, regardless of whether a combined sound source representation of two or more of the selected sound sources 162 is available. A second value (e.g., 1) of the combo. setting 164 indicates that the configurator 144 will bypass the SSR generator 146 and provide the audio adjuster 148 with a combined sound source representation of all of the selected sound sources 162 when the combined sound source representation is available.

[0069] In some implementations, the combined setting 164 is based on the context 149. For example, the context 149 includes a combination criterion 141 that determines the combined setting 164. The combination criterion 141 is based on a default configuration, a user input 103, a configuration input from an application, a configuration request from another device, or a combination thereof. The combination criterion 141 can indicate that the combined setting 164 will have a specific value (e.g., 0 or 1). In certain aspects, the combination criterion 141 can be used to determine the combined setting 164 statically, dynamically, or both. For example, the static determination of the combined setting 164 can have a first value (e.g., 0) that the configurator 144 can dynamically update to a second value (e.g., 1) in response to a detected combined condition (e.g., the remaining battery life is less than a threshold).

[0070] As an illustrative example, the selected sound sources 162 include the sound source 184A, the sound source 184B, and the sound source 184C. The configurator 144 determines whether a combined sound source representation representing all selected sound sources in the selected sound sources 162 is available (e.g., in the memory 132 or another device) in response to determining that the combined setting 164 has a second value (e.g., 1). The configurator 144 determines that the combined sound source representation 147A represents all selected sound sources in the selected sound sources 162 in response to determining that the sound source type information of the combined sound source representation 147A matches the sound source type information of each selected sound source in the selected sound sources 162, and bypasses the SSR generator 146 to provide the combined sound source representation 147A to the audio adjuster 148.

[0071] Alternatively, configurator 144 determines whether SSRs for multiple selected sound sources in selected sound sources 162 are available (e.g., in memory 132 or another device) in response to determining that combined sound source representations of all selected sound sources in selected sound sources 162 are not available. Configurator 144 adds combined sound source representation 147B to one or more selected sound source representations 156 in response to determining that combined setting 164 has a second value (e.g., 1) and combined sound source representation 147B of sounds of sound source representation 154B and sound source representation 154C is available.

[0072] The configurator 144 determines whether a separate SSR for the sound source 184A is available in response to determining that the combination setting 164 has a first value (e.g., 0) or that the sound source 184A of the selected sound source 162 is not represented by any combined SSR included in the one or more selected sound source representations 156. For example, the configurator 144 selects the sound source representation 154A to represent the sound source 184A in response to determining that the first sound source type information of the sound source representation 154A matches the second sound source type information of the sound source 184A, and adds the sound source representation 154A to the one or more selected sound source representations 156. The configurator 144 provides the one or more selected sound source representations 156 to the SSR generator 146.

[0073] The SSR generator 146 generates a combined sound source representation 147A based on one or more selected sound source representations 156. In some specific implementations, the SSR includes feature values ​​of audio features (e.g., short-term spectral features, speech source features, spectral time domain features, prosodic features, high-level features, or a combination thereof) of the sound representing the sound source. For example, the sound source representation 154A includes a first feature value of a first audio feature, the sound source representation 154B includes a second feature value of the first audio feature, and the sound source representation 154C includes a third feature value of the first audio feature. In some specific implementations, the SSR indicates feature values ​​of 512 audio features. In the illustrative example, the SSR is represented by a multidimensional (e.g., 512-dimensional) vector.

[0074] Combined sound source representation 147B (representing the sound of sound source 184B and sound source 184C) includes a first specific feature value of the first audio feature, the first specific feature value being based on the second feature value and the third feature value. In a specific implementation, combined sound source representation 147B corresponds to sound source representation 154B concatenated with sound source representation 154C, an average of sound source representation 154B and sound source representation 154C, or both. In this example, the first specific feature value corresponds to a concatenation of the second feature value and the third feature value (or a list including the second feature value and the third feature value), an average of the second feature value and the third feature value, or both. Combined sound source representation 147A includes a second specific feature value of the first audio feature, the second specific feature value being based on the first feature value (e.g., the first feature value of sound source representation 154A) and the first specific feature value (e.g., the first specific feature value of combined sound source representation 147B) (e.g., a list, an average, or both).

[0075] One or more selected sound source representations 156 including a sound source representation 154A and a combined sound source representation 147B are provided as illustrative examples. In other examples, the one or more selected sound source representations 156 may include a separate SSR for each selected sound source in the selected sound sources 162 or a plurality of combined SSRs for various combinations of the selected sound sources 162. For example, the one or more selected sound source representations 156 may include a sound source representation 154A representing the sound of the sound source 184A, a sound source representation 154B representing the sound of the sound source 184B, and a sound source representation 154C representing the sound of the sound source 184C. For another example, the one or more selected sound source representations 156 may include a combined sound source representation 147B representing the sound of the sound source 184B and the sound source 184C, and a combined sound source representation 147C representing the sound of the sound source 184A and the sound source 184C.

[0076] In some implementations, the configurator 144 dynamically updates the adjuster configuration settings 143 and the combined sound source representation 147A while the audio adjuster 148 processes the input audio signal 126. For example, the configurator 144 dynamically updates the selected sound source 162 (and the combined sound source representation 147A provided to the audio adjuster 148) in response to determining that the sound source selection criteria 157, the sound source 184 that satisfies the sound source selection criteria 157, or both have changed, the context 149 indicates that the selected sound source 162 will correspond to a participant of the scheduled meeting, and an update of the participant is detected (e.g., due to a person leaving or joining a conference call). In a first illustrative example, the configurator 144 dynamically updates the selected sound source 162 to include the detected participant (and updates the combined sound source representation 147A provided to the audio adjuster 148) in response to determining that the sound source selection criteria 157 indicates that the selected sound source 162 will correspond to a detected participant of the scheduled meeting and an update of the participant is detected (e.g., due to a person leaving or joining a conference call). In a second illustrative example, the configurator 144 dynamically removes the known user from the selected sound sources 162 (and updates the combined sound source representation 147A provided to the audio adjuster 148) in response to determining that the context 149 indicates that the selected sound source 162 will correspond to a known user (e.g., sound source 184A, sound source 184B, and sound source 184C) and detecting that the voice of one of the known users (e.g., sound source 184A) is not detected in the input audio signal 126 within a threshold time. If the voice of the known user is later detected in the input audio signal 126, the configurator 144 may subsequently add the removed known user to the selected sound sources 162 (and update the combined sound source representation 147A provided to the audio adjuster 148).

[0077] When activated, the audio conditioner 148 processes the input audio signal 126 based on the combined sound source representation 147A and the value of the retention flag 145 to retain or remove the sound to generate the output audio signal 135. For example, the audio conditioner 148 uses the neural network 150 to process the input audio signal 126 based on one or more combined sound source representations 147 to generate a mask 151, as shown in FIG. Figure 3 as further described.

[0078] The audio adjuster 148 applies a filter corresponding to the mask 151 to the input audio signal 126 in response to the retain flag 145 having a first value (e.g., 1) to retain the sound of the selected sound source 162 and remove the remaining sound. In the first illustrative example, the audio adjuster 148 retains the sound of the participants (e.g., sound source 184A, sound source 184B, and sound source 184C) of the scheduled meeting in the input audio signal 126 to generate the output audio signal 135. The remaining sound, such as the sound source 184D (e.g., speech noise, such as non-participant speech in the background) and the sound source 184E (e.g., non-speech noise, such as passing vehicles), is not included (or reduced) in the output audio signal 135. Alternatively, the audio adjuster 148 applies a filter corresponding to the inverse of the mask 151 to the input audio signal 126 in response to the retain flag 145 having a second value (e.g., 0) to remove the sound of the selected sound source 162 and retain the remaining sound. In the second illustrative example, the audio adjuster 148 removes the sounds of the known users (e.g., sound source 184A, sound source 184B, and sound source 184C) in the input audio signal 126 to generate the output audio signal 135. The remaining sounds, such as sound source 184D (e.g., the voice of an unknown person) and sound source 184E (e.g., non-speech sounds, such as emergency vehicles), are included in the output audio signal 135 (or relatively enhanced).

[0079] The technical effect of filtering the sounds of the input audio signal 126 using the mask 151 based on the combined sound source representation 147A may improve efficiency (e.g., less computation) compared to the technical effect of filtering the sounds based on the independent sound source representation for each selected sound source in the selected sound sources 162. In some aspects, the improved efficiency may enable real-time processing of the input audio signal 126. Another technical effect of filtering the sounds of the input audio signal 126 using the mask 151 based on the combined sound source representation 147A may improve accuracy in filtering portions of the input audio signal 126 that include overlapping sounds from multiple selected sound sources in the selected sound sources 162 compared to filtering based on each independent sound source representation in sequence.

[0080] The audio analyzer 140 provides the output audio signal 135 to one or more speakers 160. The one or more speakers 160 output a sound 196 based on the output audio signal 135. The sound of interest is more perceptible in the sound 196 than the sound 186 of the sound source 184. In some examples, the audio analyzer 140 provides audio data based on the output audio signal 135 to another device, such as a reference Figure 7In some examples, audio analyzer 140 stores audio data based on output audio signal 135 in memory 132, another storage device, or both.

[0081] The system 100 thus enhances the perception of the sounds of interest in the output audio signal 135 by retaining the sounds of interest or removing the remaining sounds. Using the combined sound source representation 147A representing the sounds of the selected sound sources 162 can improve the efficiency and accuracy of processing the input audio signal 126 to generate the output audio signal 135.

[0082] Although one or more microphones 120, one or more speakers 160, or a combination thereof are shown as being coupled to the device 102, in other implementations, one or more microphones 120, one or more speakers 160, or a combination thereof may be integrated into the device 102. Although the device 102 is shown as including a configurator 144, an SSR generator 146, and an audio adjuster 148, in other implementations, the configurator 144, the SSR generator 146, and the audio adjuster 148 may be included in two or more separate devices. As an illustrative example, the configurator 144 and the SSR generator 146 may be included in a user device, and the audio adjuster 148 may be included in a headset.

[0083] refer to Figure 2A , a diagram 200 showing illustrative aspects of a sound source encoder 202 operable to generate a sound source representation 154. In some implementations, Figure 1 The device 102 includes a sound source encoder 202. In other specific implementations, the sound source encoder 202 is included in a second device that provides the sound source representation 154 to the device 102.

[0084] The input audio signal 226 represents the sound 286 of the sound source 284. In some aspects, the input audio signal 226 is based on microphone output of one or more microphones that capture the sound 286. In some aspects, the input audio signal 226 is based on audio data received from another device, audio data generated by an application, or a combination thereof.

[0085] The sound source encoder 202 processes the input audio signal 226 using various techniques to generate the sound source representation 154 of the sound 286. In a particular aspect, the sound source encoder 202 performs a fast Fourier transform (FFT) of a portion (e.g., a 20 millisecond time window) of the input audio signal 226 to determine time-frequency information of the input audio signal 226. In this aspect, the sound source representation 154 (e.g., a spectrogram) is based on a first FFT feature associated with a first time window, a second FFT feature associated with a second time window, and so on. In some implementations, the sound source representation 154 includes a time-domain envelope of the input audio signal 226, a frequency content of the input audio signal 226, a time-frequency representation of the input audio signal 226 (e.g., one or more spectrograms, one or more cochleograms, one or more correlograms, etc.), or a combination thereof.

[0086] The sound source encoder 202 also generates metadata for the sound source representation 154 , which includes sound source type information (eg, demographic information, object type information, vehicle type information, person identifiers, sound source identifiers, environmental information, etc.) for the sound source 284 . Figure 1 Configurator 144 determines that sound source representation 154 represents the sound of sound source 184 in response to determining that sound source 184 and sound source 284 are of the same sound source type. Configurator 144 determines that sound source 184 and sound source 284 are of the same sound source type in response to determining that sound source type information associated with sound source representation 154 matches sound source type information of sound source 184.

[0087] In some implementations, the sound source type information is sorted by matching priority. The configurator 144 selects the sound source representation with matching sound source type information and the highest matching priority as the sound representing the sound source 184 in response to selecting the corresponding sound source type information of the plurality of sound source representations with matching sound source type information of the sound source 184. For example, the sound source identifier has a higher matching priority than other types of sound source type information (such as demographic information). For example, the sound source representation 154 having the same source identifier as the sound source 184 has a higher priority (e.g., is a closer match) than the sound source representations having other types of matching type information because the matching sound source identifier indicates that the sound source 184 is the same as the sound source 284.

[0088] In some aspects, although sound source 284 is not identical to sound source 184, sound source 284 is of the same type as sound source 184. In a particular example, sound source 184 is a first person and sound source 284 is a second person having one or more second demographic characteristics (e.g., age, location, race, ethnicity, gender, or a combination thereof) that matches one or more first demographic characteristics of the first person. In some examples, sound source 284 is an object of the same type as sound source 184 (e.g., a vehicle, a bird, or broken glass). In another example, sound source 284 is a vehicle of the same type as sound source 184 (e.g., an ambulance, a fire truck, a police car, or an airplane). In one example, sound source 284 is an alarm of the same type (e.g., a fire alarm, a manufacturing alarm, a smoke detection alarm, etc.).

[0089] In some examples, configurator 144 updates sound source representation 154 based on the sound of sound source 184 in response to determining that sound source 184 is the same as sound source 284. In some examples, configurator 144 updates sound source representation 154 based on the sound of sound source 184 in response to determining that sound source 184 is the same sound source type as sound source 284 even though sound source 184 is different from sound source 284.

[0090] refer to Figure 2B , a diagram 230 showing illustrative aspects of a sound source encoder 202 operable to generate a combined sound source representation 147 is shown.

[0091] The input audio signal 236 represents sound 296 from multiple sound sources, such as sound source 284A and sound source 284B. In some aspects, the input audio signal 236 is based on microphone output of one or more microphones that capture the sound 296. In some aspects, the input audio signal 236 is based on audio data received from another device, audio data generated by an application, or a combination thereof.

[0092] The sound source encoder 202 uses various techniques to process the input audio signal 236 to generate a combined sound source representation 147 of the sound 296. For example, the combined sound source representation 147 includes a time domain envelope of the input audio signal 236, a frequency content of the input audio signal 236, a time-frequency representation of the input audio signal 236 (e.g., a spectrogram, a cochleogram, a correlogram, etc.), or a combination thereof.

[0093] The sound source encoder 202 also generates metadata for the combined sound source representation 147, which includes a first set of sound source type information (eg, first sound source type information for sound source 284A and second sound source type information for sound source 284B). Figure 1The configurator 144 determines that the combined sound source representation 147 represents the sound of the multiple sound sources (e.g., sound source 184A and sound source 184B) in response to determining that there is a one-to-one match between each first sound source type information set in the first sound source type information set and each second sound source type information set in the second sound source type information set of the multiple sound sources (e.g., sound source 184A and sound source 184B). For example, Figure 1 Configurator 144 determines that combined sound source representation 147 represents the sounds of sound source 184A and sound source 184B in response to determining that sound source 284A is the same sound source type as sound source 184A and sound source 284B is the same sound source type as sound source 184B.

[0094] In some examples, the configurator 144 updates the combined sound source representation 147 based on the sound of any of the plurality of sound sources (e.g., sound source 184A and sound source 184B). For example, the configurator 144 updates the combined sound source representation 147 based on the sound of sound source 184A and sound source 284B in response to determining that sound source 284A and sound source 184B are the same as sound source 184A and sound source 184B, respectively. In another example, the configurator 144 updates the combined sound source representation 147 based on the sound of sound source 184A and sound source 184B in response to determining that although sound source 184A and sound source 184B are different from sound source 284A and sound source 284B, respectively, sound source 184A and sound source 184B are the same sound source type as sound source 284A and sound source 284B, respectively.

[0095] refer to Figure 2C , a diagram 240 showing illustrative aspects of a sound source encoder 202 operable to generate a combined sound source representation 147 representing sounds of an environment 204 is shown.

[0096] The input audio signal 246 represents the sound 298 of the environment 204. The environment 204 corresponds to a plurality of sound sources 284 that may include various elements that affect the sound 298. For example, the environment 204 may include the interior of a particular vehicle during vehicle operation. In this example, the environment 204 may correspond to a sound source 284 that includes elements such as wind, traffic, tires, roads, operating conditions (e.g., speed, half-open windows, fully open windows, etc.), internal shapes, external shapes, other acoustic characteristics, or combinations thereof. For another example, the environment 204 may correspond to the interior of a particular manufacturing facility. In this example, the environment 204 may correspond to a sound source 284 that includes elements such as machine noise, specific alarms, windows, doors, operating conditions (e.g., machine speed, open windows, closed windows, open doors, closed doors), facility interior shapes, facility exterior shapes, other acoustic characteristics, or combinations thereof.

[0097] Vehicles and manufacturing facilities are provided as illustrative, non-limiting examples of environments 204. Environments 204 may include various other types of environments, such as an airplane environment, a stock exchange environment, other types of indoor environments, a beach, a concert, a mall, other types of outdoor environments, a virtual environment, an augmented environment, etc. In some examples, sound source 284 corresponds to background noise (e.g., sound 298) in environment 204.

[0098] In some aspects, the input audio signal 246 is based on microphone output of one or more microphones 206 that capture sound 298. In some aspects, the input audio signal 246 is based on audio data received from another device, audio data generated by an application, or a combination thereof.

[0099] The sound source encoder 202 processes the input audio signal 246 using various techniques to generate a combined sound source representation 147 of the sound 298. For example, the combined sound source representation 147 includes a time domain envelope of the input audio signal 246, a frequency content of the input audio signal 246, a time-frequency representation of the input audio signal 246 (e.g., a spectrogram, a cochleogram, a correlogram, etc.), or a combination thereof.

[0100] The sound source encoder 202 also generates metadata for the combined sound source representation 147, which includes sound source type information of the environment 204 (e.g., environment identifier, environment type, vehicle type, tire type, operating condition, interior shape type, exterior shape type, building type, location type, operating state, event type, or a combination thereof). Figure 1 Configurator 144 determines that combined sound source representation 147 represents sound source 184 (e.g., an environment with one or more matching elements of environment 204). Figure 1 The configurator 144 can determine that the combined sound source representation 147 represents the first environment (eg, a vehicle) with the first sound source type information matching the second sound source type information of the environment 204 (eg, the second vehicle).

[0101] In some examples, configurator 144 updates combined sound source representation 147 based on sounds of sound source 184 (e.g., first environment) in response to determining that sound source 184 is the same as sound source 284. In some examples, configurator 144 updates combined sound source representation 147 based on sounds of sound source 184 (e.g., first environment) in response to determining that, although sound source 184 is different from sound source 284 (e.g., environment 204), sound source 184 is of the same sound source type (e.g., the same type of vehicle) as sound source 284.

[0102] refer to Figure 3, an illustrative example of a neural network 150 is shown. Figure 3 In the example shown in , the neural network 150 includes a feature extractor 350 coupled to a CNN 352. The CNN 352 is coupled to an initial LSTM network layer, such as LSTM 354A. The output of LSTM 354A is coupled to the input of a fully connected (FC) layer 356.

[0103] In some implementations, the output of LSTM 354A is coupled to the input of fully connected layer 356 independently of (e.g., without the need for) any additional intermediate LSTM network layers. In some implementations, the output of LSTM 354A is coupled to the input of fully connected layer 356 via one or more LSTM combiner layers. The LSTM combiner layers include LSTM network layers coupled to combiners. For example, a first LSTM combiner layer includes LSTM 354B coupled to combiner 384A, and a second LSTM combiner layer includes LSTM 354C coupled to combiner 384B. The output of LSTM 354A is coupled to the input of the LSTM network layer of the initial LSTM combiner layer. For example, the output of LSTM 354A is coupled to the input of LSTM 354B.

[0104] The combiner of the LSTM combiner layer combines the input of the LSTM network layer of the LSTM combiner layer with the output of the LSTM network layer to generate the output of the combiner. For example, combiner 384A combines the input of LSTM 354B (e.g., the output of LSTM 354A) and the output of LSTM 354B to generate the output. Each subsequent LSTM combiner layer receives the output of the previous LSTM combiner layer. For example, LSTM 354C receives the output of combiner 384A. Combiner 384B combines the input of LSTM 354C (e.g., the output of combiner 384A) and the output of LSTM 354C to generate the output of combiner 384B.

[0105] The fully connected layer 356 processes the output of the last LSTM combiner layer to generate the output of the neural network 150. For example, the fully connected layer 356 processes the output of the combiner 384B to generate the output of the neural network 150.

[0106] During operation, the feature extractor 350 receives a first portion of the input audio signal 126 (e.g., one or more audio frames of the input audio signal). The feature extractor 350 extracts features 351 of the first portion of the input audio signal 126. In the illustrative example, the feature extractor 350 generates a spectrogram of the first portion of the input audio signal 126. In certain aspects, the feature extractor 350 performs an FFT of a sub-portion of the first portion (e.g., corresponding to a 20 millisecond time window) to determine time-frequency information of the first portion. In this aspect, the features 351 are based on a first FFT feature associated with the first time window, a second FFT feature associated with the second time window, and so on. In some aspects, the features 351 include features representing Figure 1 The short-term spectral features, speech source features, spectral time domain features, prosodic features, high-level features or a combination thereof of the sound 186.

[0107] CNN 352 processes features 351 to generate convolution features 353. CNN 352 considers time-domain dependencies on time-sequential portions of input audio signal 126. For example, CNN 352 includes one or more convolution layers that apply weights to feature sets received from feature extractor 350 to generate convolution features 353. In some implementations, higher weights are applied to the most recently received feature sets (e.g., features 351) to generate convolution features 353.

[0108] In certain aspects, CNN 352 includes a one-dimensional CNN (e.g., 1D CNN) or a two-dimensional CNN (e.g., 2D CNN). For example, a 1D CNN reduces the latency between receiving a portion of an input audio signal 126, generating a mask 151, and outputting a portion of an output audio signal 135 using the mask 151. In some real-time low-latency examples (e.g., voice calls), CNN 352 includes a 1D CNN. In some aspects, a 2D CNN corresponds to an improved accuracy of the output audio signal 135 in retaining or removing sounds corresponding to a selected sound source 162 from the input audio signal 126. In some high-latency examples (e.g., voice user interfaces), CNN 352 includes a 2D CNN.

[0109] LSTM 354A processes convolutional features 353 and combined sound source representation 147A. In some examples, the input of LSTM 354A corresponds to convolutional features 353 concatenated with combined sound source representation 147A. In some implementations, the output of LSTM 354A indicates one or more features in convolutional features 353 that have feature values ​​that match corresponding feature values ​​of one or more features of combined sound source representation 147A. The output of LSTM 354A is provided to fully connected layer 356 via any LSTM combiner layer to generate mask 151.

[0110] refer to Figure 4 , a diagram 400 showing illustrative aspects of operations associated with joint training of the neural network 150 and the sound source encoder 202 is shown.

[0111] The audio combiner 450 is coupled to the sound source encoder 202 via the audio conditioner 148. The sound source encoder 202 is coupled to the speaker detector 472. The network trainer 462 is coupled to the audio conditioner 148, and the classification network trainer 482 is coupled to the sound source encoder 202 and the speaker detector 472.

[0112] During operation, the audio combiner 450 receives a speech audio signal 424A from a sound source 484A, an interfering speech audio signal 424B from a sound source 484B, and a noise audio signal 424C from a sound source 484C. The audio combiner 450 generates an input audio signal 426 based on a combination of the speech audio signal 424A, the interfering speech audio signal 424B, the noise audio signal 424C, one or more additional audio signals, or a combination thereof. In some aspects, the audio combiner 450 performs channel distortion enhancement 452, reverberation enhancement 454, or both to update the input audio signal 426. In some aspects, the input audio signal 426 approximates various noise conditions (e.g., interfering speech, noise, channel distortion, reverberation, or a combination thereof) that may be present when the speech audio signal of interest is received.

[0113] The audio combiner 450 provides the input audio signal 426 to the audio conditioner 148. The audio conditioner 148 receives the sound source representation 154 representing the sound source 484A. The neural network 150 performs as described in reference Figure 3 Similar operations are described to process the first input portion of the input audio signal 426 and the sound source representation 154 to generate the mask 451, wherein the input audio signal 426, the sound source representation 154 and the mask 451 correspond to Figure 3 The audio conditioner 148 includes an input audio signal 126, a combined sound source representation 147A, and a mask 151. The audio conditioner 148 applies the mask 451 to the first input portion to generate a first output portion of the output audio signal 435. The audio conditioner 148 applies the mask 451 to retain the sound of the sound source 484A from the first input portion and remove (or reduce) the remaining sound of the sound source 484B, the sound source 484C, one or more additional audio signals, the channel distortion enhancement 452, the reverberation enhancement 454, or a combination thereof to generate the first output portion.

[0114] The audio conditioner 148 provides a first output portion of the output audio signal 435 to the sound source encoder 202 and to the network trainer 462. The sound source encoder 202 processes the first output portion (e.g., corresponding to the sound of the sound source 484A retained from the first input portion) to generate an updated version of the sound source representation 154 of the sound source 484A. The sound source encoder 202 provides the updated version of the sound source representation 154 to the audio conditioner 148 to process the second input portion of the input audio signal 426. The sound source encoder 202 also provides the updated version of the sound source representation 154 to the speaker detector 472.

[0115] The speaker detector 472 processes the updated version of the sound source representation 154 using the classification network 474 to generate an estimated speaker identifier 475. The speaker detector 472 provides the estimated speaker identifier 475 to the classification network trainer 482.

[0116] The network trainer 462 generates a noise reduction loss metric 464 based on a comparison of the first output portion of the output audio signal 435 and the corresponding first speech portion of the speech audio signal 424A. The first output portion corresponds to the sound of the sound source 484A retained from the input audio signal 426, and the first speech portion corresponds to the original sound of the sound source 484A. The network trainer 462 generates update data 463 based on the noise reduction loss metric 464. For example, the update data 463 indicates an update to the weights, bias values, or a combination thereof of the neural network 150 to reduce the noise reduction loss metric 464 in subsequent iterations. The network trainer 462 thus trains the neural network 150 over time to improve noise reduction and reduce the difference between the output audio signal 435 and the speech audio signal 424A.

[0117] The classification network trainer 482 determines a classification loss metric 486 based on the estimated speaker identifier 475 and the speaker identifier 481 of the sound source 484A. For example, the classification network trainer 482 retrieves a first sound source representation of a sound source (e.g., a person) associated with the estimated speaker identifier 475, retrieves a second sound source representation of the sound source 484A, and generates the classification loss metric 486 based on a comparison of the first sound source representation and the second sound source representation. In certain aspects, the second sound source representation was previously generated based on the sound of the sound source 484A.

[0118] The classification network trainer 482 generates update data 483 and update data 485 based on the classification loss metric 486. For example, the update data 483 indicates an update to the weights, bias values, or a combination thereof of the sound source encoder 202 to reduce the classification loss metric 486. The classification network trainer 482 thus trains the sound source encoder 202 over time to generate a sound source representation 154 that more closely matches the second sound source representation. Similarly, the update data 485 indicates an update to the weights, bias values, or a combination thereof of the classification network 474 to reduce the classification loss metric 486. The classification network trainer 482 thus trains the classification network 474 over time to generate an estimated speaker identifier 475 that corresponds to a speaker having a sound source representation that more closely matches the second sound source representation. The network trainer 462 and the classification network trainer 482 thus jointly train the neural network 150, the sound source encoder 202, and the classification network 474.

[0119] In some implementations, the neural network 150, the sound source encoder 202, the classification network 474, or a combination thereof is trained during a training phase and designated as ready for use when training is complete. For example, the network trainer 462 determines that training of the neural network 150 is complete in response to determining that the noise reduction loss metric 464 satisfies the training criteria (e.g., is less than a loss threshold). For another example, the classification network trainer 482 determines that training of the sound source encoder 202 and the classification network 474 is complete in response to determining that the classification loss metric 486 satisfies the training criteria (e.g., is less than a loss threshold).

[0120] In some aspects, the audio adjuster 148 is available for use when the network trainer 462 determines that training of the neural network 150 is complete. In some aspects, the sound source encoder 202 and the classification network 474 are available for use when the classification network trainer 482 determines that training of the sound source encoder 202 and the classification network 474 is complete.

[0121] In some implementations, the audio combiner 450, the network trainer 462, the speaker detector 472, the classification network trainer 482, or a combination thereof are integrated in Figure 1 In other implementations, the audio combiner 450, the network trainer 462, the classification network trainer 482, or a combination thereof are integrated into the second device. In these implementations, the second device provides the neural network 150, the audio adjuster 148, the sound source encoder 202, the speaker detector 472, the classification network 474, or a combination thereof to the device 102 in response to determining that the audio adjuster 148, the sound source encoder 202, the classification network 474, or a combination thereof is available for use.

[0122] In some implementations, the neural network 150, the sound source encoder 202, the classification network 474, or a combination thereof is trained (e.g., dynamically updated) during use. In these implementations, the input audio signal 426 corresponds to Figure 1 4A. In the illustrative example, the network trainer 462 receives the speech audio signal 424A from a first microphone that is closer to the sound source 484A (e.g., a known user with a speaker identifier 481), and receives the input audio signal 426 from one or more second microphones. The network trainer 462 trains the neural network 150 so that the output audio signal 435 matches the speech audio signal 424A (e.g., the noise reduction loss metric 464 is less than a loss threshold), and uses the output audio signal 435 to generate the sound source representation 154 of the sound source 484A. The classification network trainer 482 uses the sound source representation 154 to train the sound source encoder 202 and the classification network 474. The network trainer 462 can use the sound source representation 154 and the trained version of the neural network 150 to retain or remove the sound of the sound source 484A to generate the output audio signal 135.

[0123] refer to Figure 5 , a diagram 500 showing illustrative aspects of operations associated with joint training of the neural network 150 and the sound source encoder 202 is shown.

[0124] During operation, the audio combiner 450 receives the speech audio signal 524A from the sound source 484A and the sound source 484D, the interfering speech audio signal 424B from the sound source 484B, and the noise audio signal 424C from the sound source 484C. The audio combiner 450 generates an input audio signal 526 based on a combination of the speech audio signal 524A, the interfering speech audio signal 424B, the noise audio signal 424C, one or more additional audio signals, or a combination thereof. In some aspects, the audio combiner 450 performs channel distortion enhancement 452, reverberation enhancement 454, or both to update the input audio signal 526. In some aspects, the input audio signal 526 approximates various noise conditions (e.g., interfering speech, noise, channel distortion, reverberation, or a combination thereof) that may be present when the speech audio signal of interest is received.

[0125] The audio combiner 450 provides the input audio signal 526 to the audio conditioner 148. The audio conditioner 148 receives the combined sound source representation 147 representing the sound source 484A and the sound source 484D. The neural network 150 performs the same as described in reference Figure 3 Similar operations are described to process the first input portion of the input audio signal 526 and the combined sound source representation 147 to generate the mask 551, wherein the input audio signal 526, the combined sound source representation 147 and the mask 551 correspond to Figure 3 The audio signal 126 of FIG. 1 is an input audio signal 126, a combined sound source representation 147A, and a mask 151. The audio adjuster 148 applies the mask 551 to the first input portion to generate a first output portion of the output audio signal 535. The audio adjuster 148 applies the mask 551 to retain the sounds of the sound source 484A and the sound source 484D from the first input portion and remove (or reduce) the remaining sounds of the sound source 484B, the sound source 484C, one or more additional audio signals, the channel distortion enhancement 452, the reverberation enhancement 454, or a combination thereof to generate the first output portion.

[0126] The audio conditioner 148 provides a first output portion of the output audio signal 535 to the sound source encoder 202 and to the network trainer 462. The sound source encoder 202 processes the first output portion (e.g., the sounds corresponding to the sound sources 484A and 484D retained from the first input portion) to generate an updated version of the sound source representation 147 of the sound sources 484A and 484D. The sound source encoder 202 provides an updated version of the combined sound source representation 147 to the audio conditioner 148 to process the second input portion of the input audio signal 526. The sound source encoder 202 also provides an updated version of the combined sound source representation 147 to the speaker detector 472.

[0127] The speaker detector 472 processes the updated version of the combined sound source representation 147 using the classification network 574 to generate an estimated speaker identifier 575. The speaker detector 472 provides the estimated speaker identifier 575 to the classification network trainer 482.

[0128] The network trainer 462 generates a noise reduction loss metric 564 based on a comparison of the first output portion of the output audio signal 535 and the corresponding first speech portion of the speech audio signal 524A. The first output portion corresponds to the sound of the sound source 484A and the sound source 484D retained from the input audio signal 526, and the first speech portion corresponds to the original sound of the sound source 484A and the sound source 484D. The network trainer 462 generates update data 563 based on the noise reduction loss metric 564. For example, the update data 563 indicates an update to the weights, bias values, or a combination thereof of the neural network 150 to reduce the noise reduction loss metric 564 in subsequent iterations. The network trainer 462 thus trains the neural network 150 over time to improve noise reduction and reduce the difference between the output audio signal 535 and the speech audio signal 524A.

[0129] The classification network trainer 482 determines a classification loss metric 586 based on the estimated speaker identifier 575 and the speaker identifiers 581 of the sound source 484A and the sound source 484D. For example, the classification network trainer 482 retrieves a first combined sound source representation of a first pair of sound sources (e.g., two persons) associated with the estimated speaker identifier 575 and a second combined sound source representation of the sound source 484A and the sound source 484D. The classification network trainer 482 generates the classification loss metric 486 based on a comparison of the first combined sound source representation and the second combined sound source representation. In a particular aspect, the second combined sound source representation was previously generated based on the sounds of the sound source 484A and the sound source 484D.

[0130] The classification network trainer 482 generates update data 583 and update data 585 based on the classification loss metric 586. For example, the update data 583 indicates an update to the weights, bias values, or a combination thereof of the sound source encoder 202 to reduce the classification loss metric 586. The classification network trainer 482 thus trains the sound source encoder 202 over time to generate a combined sound source representation 147 that more closely matches the second combined sound source representation. Similarly, the update data 585 indicates an update to the weights, bias values, or a combination thereof of the classification network 574 to reduce the classification loss metric 586. The classification network trainer 482 thus trains the classification network 574 over time to generate an estimated speaker identifier 575 that corresponds to a pair of speakers having a combined sound source representation that more closely matches the second combined sound source representation. The network trainer 462 and the classification network trainer 482 thus jointly train the neural network 150, the sound source encoder 202, and the classification network 574.

[0131] In some implementations, the neural network 150, the sound source encoder 202, the classification network 474, or a combination thereof is trained during a training phase and designated as ready for use when training is complete. For example, the network trainer 462 determines that training of the neural network 150 is complete in response to determining that the noise reduction loss metric 564 satisfies the training criteria (e.g., is less than a loss threshold). For another example, the classification network trainer 482 determines that training of the sound source encoder 202 and the classification network 574 is complete in response to determining that the classification loss metric 586 satisfies the training criteria (e.g., is less than a loss threshold).

[0132] In some aspects, the audio adjuster 148 is available for use when the network trainer 462 determines that training of the neural network 150 is complete. In some aspects, the sound source encoder 202 and the classification network 574 are available for use when the classification network trainer 482 determines that training of the sound source encoder 202 and the classification network 574 is complete.

[0133] In some implementations, the audio combiner 450, the network trainer 462, the speaker detector 472, the classification network trainer 482, or a combination thereof are integrated in Figure 1 In other implementations, the audio combiner 450, the network trainer 462, the classification network trainer 482, or a combination thereof are integrated into the second device. In these implementations, the second device provides the neural network 150, the audio adjuster 148, the sound source encoder 202, the speaker detector 472, the classification network 574, or a combination thereof to the device 102 in response to determining that the audio adjuster 148, the sound source encoder 202, the classification network 574, or a combination thereof is available for use.

[0134] In some implementations, the neural network 150, the sound source encoder 202, the classification network 474, or a combination thereof is trained (e.g., dynamically updated) during use. In these implementations, the input audio signal 526 corresponds to Figure 1 581 . In the illustrative example, the network trainer 462 receives the speech audio signal 524A from a first microphone that is closer to the sound source 484A and the sound source 484D (e.g., a pair of known sound sources with speaker identifiers 581), and receives the input audio signal 526 from one or more second microphones. The network trainer 462 trains the neural network 150 so that the output audio signal 535 matches the speech audio signal 524A (e.g., the noise reduction loss metric 564 is less than a loss threshold), and uses the output audio signal 535 to generate a combined sound source representation 147 of the sound source 484A and the sound source 484D. The classification network trainer 482 uses the combined sound source representation 147 to train the sound source encoder 202 and the classification network 574. The network trainer 462 can use the sound source representation 154 and the trained version of the neural network 150 to retain or remove the sounds of the sound source 484A and the sound source 484D to generate the output audio signal 135.

[0135] refer to Figure 6 , a diagram 600 of a device 102 operable to receive audio data 626 from a device 650 and process the audio data 626 using a sound source representation. The device 102 includes an audio analyzer 140 coupled to a receiver 640.

[0136] During operation, the receiver 640 receives audio data 626 from the device 650. The audio data 626 represents the input audio signal 126. For example, the audio data 626 represents the sound 186 of the sound source 184. In some implementations, the audio data 626 corresponds to encoded audio data, and the decoder of the device 102 decodes the audio data 626 to generate the input audio signal 126.

[0137] As reference Figure 1 As depicted, the audio analyzer 140 processes the input audio signal 126 based on the combined sound source representation 147A to generate the output audio signal 135. The audio analyzer 140 provides the output audio signal 135 to one or more speakers 160 to output the sound 196.

[0138] refer to Figure 7 , shows a diagram 700 of a device 102 operable to process an input audio signal 126 using a sound source representation to generate an output audio signal 135, to generate audio data 726 based on the output audio signal 135, and to transmit the audio data 726 to a device 702. The device 102 includes an audio analyzer 140 coupled to a transmitter 740.

[0139] During operation, the audio analyzer 140 receives an input audio signal 126 corresponding to the microphone output of the one or more microphones 120. The input audio signal 126 represents the sound 186 of the sound source 184 captured by the one or more microphones 120. Figure 1 As depicted, audio analyzer 140 processes input audio signal 126 using combined sound source representation 147A to generate output audio signal 135 .

[0140] The transmitter 740 transmits the audio data 726 to the device 702. The audio data 726 is based on the output audio signal 135. In some examples, the audio data 726 corresponds to encoded audio data, and an encoder of the device 102 encodes the output audio signal 135 to generate the audio data 726.

[0141] The decoder of the device 702 decodes the audio data 726 to generate a decoded audio signal. The device 702 provides the decoded audio signal to one or more speakers 160 for outputting sound 196.

[0142] In some implementations, device 702 is the same as device 650. For example, device 102 receives audio data 626 from a second device and outputs sound 196 via one or more speakers 160, while concurrently capturing sound 186 via one or more microphones 120 and transmitting audio data 726 to the second device.

[0143] Figure 8 According to some examples of the present disclosure Figure 181 . The neural network 150 is configured to receive a sequence 810 of audio data samples (such as a sequence of consecutively captured frames of the input audio signal 126), which is shown as a first frame (F1) 812, a second frame (F2) 814, and one or more other frames including an Nth frame (FN) 816 (where N is an integer greater than two). The neural network 150 is also configured to receive a sequence 840 of sound source representations, such as a sequence of combined sound source representations (including the combined sound source representation 147A), which are shown as a first sound source representation (S1) 842, a second sound source representation (S2) 844, and one or more additional sound source representations (including an Rth sound source representation (SR) 846) (where R is an integer greater than two and less than or equal to N). The neural network 150 is configured to output a sequence 820 of masks including a first mask (M1) 822, a second mask (M2) 824, and one or more additional masks including an Nth mask (MN) 826.

[0144] The neural network 150 is configured to receive the sequence 810 of audio data samples and adaptively use the sequence 840 of sound source representations to generate a mask (e.g., a second mask 824) of the sequence 810 corresponding to a frame (e.g., a second frame (F2) 814) of the sequence 820 based at least in part on a previous frame (e.g., a first frame (F1) 812) of the audio data samples in the sequence 810. As an illustrative, non-limiting example, the neural network 150 may include a CNN.

[0145] The audio adjuster 148 is configured to apply a mask of the sequence 810 (e.g., a first mask (M1) 822) to a corresponding frame of the sequence 820 (e.g., a first frame (F1) 812) to generate a frame of a sequence 830 of audio data samples (e.g., a first frame (O1) 832), such as a sequence of consecutive frames of the output audio signal 135, which are shown as a first frame (O1) 832, a second frame (O2) 834, and one or more additional frames (including an Nth frame (ON) 836) (where N is an integer greater than two).

[0146] During operation, the neural network 150 processes the first frame (F1) 812 using the first sound source representation (S1) 842 to generate a first mask (M1) 822, and the audio adjuster 148 applies the first mask (M1) 822 to the first frame (F1) 812 to generate a first frame (O1) 832 of the sequence 830 of audio data samples. The neural network 150 processes the second frame (F2) 814 using the second sound source representation (S2) 844 to generate a second mask (M2) 824, and the audio adjuster 148 applies the second mask (M2) 824 to the second frame (F2) 814 to generate a second frame (O2) 834 of the sequence 830 of audio data samples. In some implementations, the second mask (M2) 824 is based on the second frame (F2) 814 and at least partially on the first frame (F1) 812 of audio data samples. This processing continues, including the neural network 150 processing the Nth frame (FN) 816 using the Nth sound source representation (SR) 846 to generate the Nth mask (MN) 826, and the audio modifier 148 applying the Nth mask (MN) 826 to the Nth frame (FN) 816 to generate the Nth frame (ON) 836 of the sequence 830 of audio data samples.

[0147] In some implementations, the configurator 144 provides the SSR to the neural network 150 to process each frame of the sequence 810 of audio data samples (e.g., the integer R is equal to the integer N). In some examples, if one or more selected sound sources 162 remain unchanged from the first frame (F1) 812 to the second frame (F2) 814, the second sound source representation (S2) 844 is the same as the first sound source representation (S1) 842 or is an updated (e.g., dynamically trained) version of the first sound source representation (S1) 842.

[0148] In some implementations, the configurator 144 provides the SSR to the neural network 150 at a rate different from the rate at which the neural network 150 processes the frames of the sequence 810 of audio data samples. For example, the configurator 144 provides an SSR to the neural network 150 for every four frames of the sequence 810 of audio data samples (e.g., the integer N is equal to 4 times the integer R). For example, the neural network 150 processes the first four frames of the sequence 810 of audio data samples using the first sound source representation (S1) 842, processes the next four frames of the sequence 810 of audio data samples using the second sound source representation (S2) 844, and so on. If the one or more selected sound sources 162 are the same for the first frame and the fifth frame, the second sound source representation (S2) 844 is the same as the first sound source representation (S1) 842 or is an updated (e.g., dynamically trained) version of the first sound source representation (S1) 842.

[0149] In some implementations, the configurator 144 provides the SSR to the neural network 150 in response to a change in the one or more selected sound sources 162. For example, if the one or more selected sound sources 162 remain unchanged when processing the sequence 810 of audio data samples, the configurator 144 provides only the first sound source representation (S1) 842 (e.g., the integer R is equal to 1) to the neural network 150, and the neural network 150 uses the first sound source representation (S1) 842 to process all frames of the sequence 810 of audio data samples.

[0150] In some implementations, the Nth mask (MN) 826 is based on the Nth frame (FN) 816 and at least in part on one or more previous frames of the audio data samples of the sequence 810. By dynamically generating a mask based on one or more previous frames of the audio data samples, the accuracy of the audio adjustment by the audio adjuster 148 for audio (e.g., speech, music, etc.) that may span multiple frames of audio data can be improved.

[0151] Fig. 9 The device 102 is depicted as a specific implementation 900 of an integrated circuit 902 that includes one or more processors 190. The integrated circuit 902 also includes an audio input 904 (such as one or more bus interfaces) to enable receiving input audio signals 126 for processing. The integrated circuit 902 also includes a signal output 906 (such as a bus interface) to enable transmission of output signals, such as the output audio signal 135. The integrated circuit 902 enables the processing of audio based on a sound source representation to be implemented as a component in a system that includes a microphone, such as in Fig.10 A mobile phone or tablet as shown in Fig.11 The headphones shown in Fig.12 Wearable electronic devices such as those shown in Fig.13 The voice controlled speaker system shown in Fig.14 The camera shown in Fig.15 A virtual reality, mixed reality, or augmented reality headset as shown in Fig.16 or Fig.17 The means of transport shown in .

[0152] As an illustrative, non-limiting example, Fig.10 An implementation 1000 is depicted in which the device 102 includes a mobile device 1002, such as a phone or tablet. The mobile device 1002 includes one or more microphones 120, one or more speakers 160, and a display screen 1004. Components of the one or more processors 190, including the audio analyzer 140, are integrated into the mobile device 1002 and are shown using dashed lines to indicate internal components that are not generally visible to a user of the mobile device 1002.

[0153] In a particular example, the audio analyzer 140 operates to process the input audio signal 126 using the sound source representation to generate the output audio signal 135. In a particular aspect, the input audio signal 126 is based on the microphone output of one or more microphones 120, and the audio analyzer 140 sends a signal to another device (not shown) such as a microphone 120 based on the output audio signal 135. Figure 7 Device 702 provides audio data.

[0154] In certain aspects, the input audio signal 126 may be based on a signal received from another device (not shown) such as Figure 6 The audio analyzer 140 may be configured to receive audio data received by the device 650. In some implementations, the audio analyzer 140 provides an output audio signal 135 to one or more speakers 160 for playback. In some implementations, the output audio signal 135 is processed to perform one or more operations at the mobile device 1002, such as launching a graphical user interface or otherwise displaying other information associated with the user's voice at the display screen 1004 (e.g., via an integrated "smart assistant" application).

[0155] Fig.11 A specific implementation 1100 is depicted in which the device 102 includes a headphone device 1102. The headphone device 1102 includes one or more microphones 120, one or more speakers 160, or a combination thereof. Components of the one or more processors 190, including the audio analyzer 140, are integrated into the headphone device 1102. In a specific example, the audio analyzer 140 operates to process the input audio signal 126 using the sound source representation to generate the output audio signal 135. In a specific aspect, the input audio signal 126 can be based on the microphone output of the one or more microphones 120, and the audio analyzer 140 sends a signal to another device (not shown), such as a sound source representation, based on the output audio signal 135. Figure 7 Device 702 provides audio data for further processing.

[0156] In certain aspects, the input audio signal 126 may be based on a signal received from another device (not shown) such as Figure 6 The audio analyzer 140 may be configured to receive audio data received by the device 650. In some implementations, the audio analyzer 140 may provide the output audio signal 135 to one or more speakers 160 for playback. In some implementations, the output audio signal 135 may be processed to perform one or more operations at the headphone device 1102.

[0157] Fig.12A specific implementation 1200 is depicted in which the device 102 includes a wearable electronic device 1202 (shown as a "smart watch"). An audio analyzer 140, one or more microphones 120, and one or more speakers 160 are integrated into the wearable electronic device 1202. In a specific example, the audio analyzer 140 operates to process an input audio signal 126 using a sound source representation to generate an output audio signal 135. In a specific aspect, the input audio signal 126 is based on the microphone output of the one or more microphones 120, and the audio analyzer 140 sends a signal to another device (not shown), such as a wireless speaker, based on the output audio signal 135. Figure 7 Device 702 provides audio data.

[0158] In certain aspects, the input audio signal 126 may be based on a signal received from another device (not shown) such as Figure 6 The device 650 receives audio data. In some specific implementations, the audio analyzer 140 provides an output audio signal 135 to one or more speakers 160 for playback. In some specific implementations, the output audio signal 135 is processed to perform one or more operations at the wearable electronic device 1202, such as starting a graphical user interface or otherwise displaying other information associated with the user's voice at the display screen 1204 of the wearable electronic device 1202. For example, the wearable electronic device 1202 may include a display screen configured to display a notification based on the user's voice detected by the wearable electronic device 1202 in the output audio signal 135. In a specific example, the wearable electronic device 1202 includes a tactile device that provides a tactile notification (e.g., vibration) in response to the detection of the user's voice activity. For example, the tactile notification can enable the user to view the wearable electronic device 1202 to see the displayed notification indicating that the keyword spoken by the user has been detected. Therefore, the wearable electronic device 1202 can remind a user with hearing impairment or a user wearing a headset that the user's voice activity is detected.

[0159] Fig.13 1300 is a specific implementation in which the device 102 includes a wireless speaker and a voice activated device 1302. The wireless speaker and voice activated device 1302 may have wireless network connectivity and be configured to perform auxiliary operations. One or more processors 190 including an audio analyzer 140, one or more microphones 120, one or more speakers 160, or a combination thereof are included in the wireless speaker and voice activated device 1302.

[0160] In a particular example, the audio analyzer 140 operates to process the input audio signal 126 using the sound source representation to generate the output audio signal 135. In a particular aspect, the input audio signal 126 may be based on the microphone output of one or more microphones 120, and the audio analyzer 140 transmits a signal to another device (not shown) such as a microphone 120 based on the output audio signal 135. Figure 7 Device 702 provides audio data for further processing.

[0161] In certain aspects, the input audio signal 126 may be based on a signal received from another device (not shown) such as Figure 6 The audio analyzer 140 may include audio data received by the device 650. In some implementations, the audio analyzer 140 provides an output audio signal 135 to one or more speakers 160 for playback. In some implementations, the output audio signal 135 is processed to perform one or more operations at the wireless speaker and voice activation device 1302. For example, in response to detecting a verbal command in the output audio signal 135, the wireless speaker and voice activation device 1302 may perform an auxiliary operation, such as via execution of a voice activation system (e.g., an integrated auxiliary application). The auxiliary operation may include adjusting the temperature, playing music, turning on the lights, etc. For example, the auxiliary operation is performed in response to receiving a command after a keyword or key phrase (e.g., "Hello, assistant").

[0162] Fig.14 An implementation 1400 is depicted in which the device 102 includes a portable electronic device corresponding to a camera device 1402. Included in the camera device 1402 is an audio analyzer 140, one or more microphones 120, one or more speakers 160, or a combination thereof.

[0163] In a particular example, the audio analyzer 140 operates to process the input audio signal 126 using the sound source representation to generate the output audio signal 135. In a particular aspect, the input audio signal 126 may be based on the microphone output of one or more microphones 120, and the audio analyzer 140 transmits a signal to another device (not shown) such as a microphone 120 based on the output audio signal 135. Figure 7 Device 702 provides audio data for further processing.

[0164] In certain aspects, the input audio signal 126 may be based on a signal received from another device (not shown) such as Figure 61402 receives audio data from the device 650. In some implementations, the audio analyzer 140 provides the output audio signal 135 to one or more speakers 160 for playback. In some implementations, the output audio signal 135 is processed to perform one or more operations at the camera device 1402. For example, as an illustrative example, in response to detecting a spoken command in the output audio signal 135, the camera device 1402 may perform an operation in response to the spoken user command, such as adjusting image or video capture settings, image or video playback settings, or image or video capture instructions.

[0165] Fig.15 An implementation 1500 is depicted in which the device 102 includes a portable electronic device corresponding to a virtual reality, mixed reality, or augmented reality headset 1502. An audio analyzer 140, one or more microphones 120, one or more speakers 160, or a combination thereof are integrated into the headset 1502. In a particular aspect, the headset 1502 includes a first microphone 120 positioned to primarily capture a user's voice and a second microphone 120 positioned to primarily capture ambient sounds.

[0166] In a particular example, the audio analyzer 140 operates to process the input audio signal 126 using the sound source representation to generate the output audio signal 135. In a particular aspect, the input audio signal 126 may be based on the microphone output of one or more microphones 120, and the audio analyzer 140 transmits a signal to another device (not shown) such as a microphone 120 based on the output audio signal 135. Figure 7 Device 702 provides audio data for further processing.

[0167] In certain aspects, the input audio signal 126 may be based on a signal received from another device (not shown) such as Figure 6 The audio analyzer 140 may be configured to receive audio data received by the device 650 of the present invention. In some implementations, the audio analyzer 140 provides the output audio signal 135 to one or more speakers 160 for playback. In some implementations, the output audio signal 135 is processed to perform one or more operations at the headset 1502. The visual interface device is positioned in front of the user's eyes so that augmented reality, mixed reality, or virtual reality images or scenes can be displayed to the user while wearing the headset 1502. In a specific example, the visual interface device is configured to display a notification indicating the user's voice detected in the output audio signal 135.

[0168] Fig.16An implementation 1600 is depicted in which the device 102 corresponds to or is integrated within a vehicle 1602 (shown as a manned or unmanned aerial device (e.g., a package delivery drone)). An audio analyzer 140, one or more microphones 120, one or more speakers 160, or a combination thereof are integrated into the vehicle 1602.

[0169] In a particular example, the audio analyzer 140 operates to process the input audio signal 126 using the sound source representation to generate the output audio signal 135. In a particular aspect, the input audio signal 126 may be based on the microphone output of one or more microphones 120, and the audio analyzer 140 transmits a signal to another device (not shown) such as a microphone 120 based on the output audio signal 135. Figure 7 Device 702 provides audio data for further processing.

[0170] In certain aspects, the input audio signal 126 may be based on a signal received from another device (not shown) such as Figure 6 The audio analyzer 140 may include an audio analyzer 140 that receives audio data from a device 650. In some implementations, the audio analyzer 140 provides an output audio signal 135 to one or more speakers 160 for playback. In some implementations, the output audio signal 135 is processed to perform one or more operations at the vehicle 1602. For example, user voice activity detection may be performed based on the output audio signal 135, such as a delivery instruction for an authorized user from the vehicle 1602.

[0171] Fig.17 Another implementation 1700 is depicted in which the device 102 corresponds to or is integrated within a vehicle 1702 (shown as a car). The vehicle 1702 includes one or more processors 190 including an audio analyzer 140. The vehicle 1702 also includes one or more microphones 120, one or more speakers 160, or a combination thereof.

[0172] In a particular example, the audio analyzer 140 operates to process the input audio signal 126 using the sound source representation to generate the output audio signal 135. In a particular aspect, the input audio signal 126 may be based on the microphone output of one or more microphones 120, and the audio analyzer 140 transmits a signal to another device (not shown) such as a microphone 120 based on the output audio signal 135. Figure 7 The device 702 provides the audio data for further processing. In the illustrative example, the combined sound source representation 147 represents the sound 298 of the sound source 284 of the first environment, as shown in FIG. Figure 2CThe audio analyzer 140 processes the input audio signal 126 using the combined sound source representation 147 to remove background noise to generate the output audio signal 135 in response to determining that the second environment of the vehicle 1702 matches the first environment associated with the combined sound source representation 147 .

[0173] In particular aspects, the audio analyzer 140 determines that the first environment matches the second environment in response to determining that the first environment is associated with a first vehicle that matches the vehicle 1702, associated with a first operating state of the first vehicle that matches a second operating state of the vehicle 1702, associated with one or more first external conditions that match one or more second external conditions of the vehicle 1702, or a combination thereof.

[0174] In certain aspects, the audio analyzer 140 determines that the first vehicle matches the vehicle 1702 in response to determining that the first vehicle is the same as the vehicle 1702. In another aspect, the audio analyzer 140 determines that the first vehicle matches the vehicle 1702 in response to determining that the first vehicle is the same vehicle model (e.g., The first vehicle matching vehicle 1702 is determined by the same vehicle model (e.g., a registered trademark of Polestar Holding), the same manufacturing year (e.g., 2022), the same vehicle type (e.g., electric SUV), or a combination thereof. In certain aspects, the operating state of the vehicle may include speed, reversing, turning, braking, etc. In certain aspects, the external conditions of the vehicle may include road type (e.g., highway, muddy road, suburban road, urban road), weather conditions (e.g., wind, rain, storm), traffic conditions (e.g., heavy traffic, medium traffic, no traffic), etc.

[0175] In certain aspects, the input audio signal 126 may be based on a signal received from another device (not shown) such as Figure 6650 of the device. In some embodiments, the audio analyzer 140 provides an output audio signal 135 to one or more speakers 160 for playback. In some embodiments, the output audio signal 135 is processed to perform one or more operations at the vehicle 1702. For example, user voice activity detection can be performed based on the output audio signal 135. In some embodiments, user voice activity detection can be performed based on the input audio signal 126 received from the internal microphone (e.g., one or more microphones 120), such as for voice commands from authorized passengers. For example, the input audio signal 126 includes a sound from a sound source 184A, such as a voice command from an authorized user (e.g., a parent) of the vehicle 1702 to set the volume to 5 or set the destination of the autonomous vehicle. The input audio signal 126 also includes a sound from a sound source 184B, such as a voice command from an unauthorized user (e.g., a child) of the vehicle 1702 to set the volume to 9. The input audio signal 126 may also include sounds from a sound source 184C, such as other passengers discussing another location.

[0176] The audio analyzer 140 can process the input audio signal 126 to retain the sound from the sound source 184A and remove the sound from the sound source 184B and the sound source 184C to generate the output audio signal 135, and can perform user voice activity detection on the output audio signal 135 to detect a voice command from an authorized user (e.g., a voice command from a parent to set the volume to 5 or set a destination for the autonomous vehicle).

[0177] In some implementations, user voice activity detection can be performed based on an input audio signal 126 received from an external microphone (e.g., one or more microphones 120), such as an authorized user of the vehicle. In a particular implementation, the voice activation system initiates one or more operations of the vehicle 1702 based on one or more keywords (e.g., "unlock", "start engine", "play music", "show weather forecast", or another voice command) detected in the output audio signal 135, such as by providing feedback or information via the display 1720 or one or more speakers 160.

[0178] refer to Fig.18 , shows a specific implementation of method 1800 for processing audio based on sound source representation. In certain aspects, one or more operations of method 1800 are performed by configurator 144, SSR generator 146, audio adjuster 148, neural network 150, audio analyzer 140, one or more processors 190, device 102, Figure 1system 100, feature extractor 350, CNN 352, LSTM 354A, LSTM 354B, LSTM 354C, Figure 3 The fully connected layer 356, Figure 4 Audio combiner 450, Figure 6 Receiver 640, Figure 7 It is executed by at least one of the transmitter 740 or a combination thereof.

[0179] At 1802, method 1800 includes receiving an input audio signal at a first device. For example, audio analyzer 140 receives input audio signal 126 at device 102. In some implementations, input audio signal 126 is based on microphone output of one or more microphones 120. In some implementations, input audio signal 126 is based on audio data 626 received from device 650, as described in reference to FIG. Figure 6 as described.

[0180] At 1804, the method 1800 further includes processing the input audio signal based on the combined representation of the plurality of sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the plurality of sound sources from the input audio signal. For example, the audio analyzer 140 processes the input audio signal 126 based on the combined sound source representation 147A of the sound source 184A and the sound source 184B to generate the output audio signal 135, as described in reference to FIG. Figure 1 The audio analyzer 140 selectively uses the combined sound source representation 147A to retain or remove the sounds of the sound source 184A and the sound source 184B from the input audio signal 126 based on the retain flag 145 of the adjuster configuration setting 143 .

[0181] At 1806, the method 1800 further includes providing the output audio signal to the second device. For example, the audio modulator 148 provides the output audio signal 135 to one or more speakers 160, as shown in FIG. Figure 1 As another example, the audio adjuster 148 provides the output audio signal 135 to the transmitter 740 to transmit the audio data 726 to the device 702. The audio data 726 is based on the output audio signal 135 (eg, is an encoded version of the output audio signal).

[0182] The method 1800 enhances the perception of the sounds of interest in the output audio signal 135 by retaining the sounds of interest or removing the remaining sounds. Using the combined sound source representation 147A representing the sounds of the selected sound sources 162 can improve the efficiency and accuracy of processing the input audio signal 126 to generate the output audio signal 135.

[0183] Fig.18The method 1800 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a digital signal processor (DSP), a controller, another hardware device, a firmware device, or any combination thereof. As an example, Fig.18 The method 1800 may be performed by a processor executing instructions, such as reference Fig.19 as described.

[0184] refer to Fig.19 , depicts a block diagram of a particular exemplary implementation of a device and generally designated 1900. In various implementations, device 1900 may have more Fig.19 In an exemplary implementation, device 1900 may correspond to device 102. In an exemplary implementation, device 1900 may perform the operations described in reference to Figures 1 to 18 One or more operations described.

[0185] In certain implementations, device 1900 includes a processor 1906 (e.g., a CPU). Device 1900 may include one or more additional processors 1910 (e.g., one or more DSPs). In certain aspects, Figure 1 The one or more processors 190 correspond to processor 1906, processor 1910, or a combination thereof. Processor 1910 may include a speech and music coder-decoder (codec) 1908, which includes a speech decoder ("vocoder") encoder 1936, a vocoder decoder 1938, an audio analyzer 140, or a combination thereof.

[0186] Device 1900 may include memory 1986 and codec 1934. Memory 1986 may include instructions 1956 that may be executed by one or more additional processors 1910 (or processor 1906) to implement the functionality described with reference to audio analyzer 140. Device 1900 may include a modem 1970 coupled to antenna 1952 via transceiver 1950. In certain aspects, memory 1986 includes Figure 1 Memory 132. In certain aspects, transceiver 1950 includes Figure 6 Receiver 640, Figure 7 transmitter 740 or both.

[0187] The device 1900 may include a display 1928 coupled to a display controller 1926. One or more microphones 120, one or more speakers 160, or a combination thereof may be coupled to a codec 1934. The codec 1934 may include a digital-to-analog converter (DAC) 1902, an analog-to-digital converter (ADC) 1904, or both. In a particular implementation, the codec 1934 may receive an analog signal from the one or more microphones 120, convert the analog signal to a digital signal using the analog-to-digital converter 1904, and provide the digital signal to a voice and music codec 1908. The voice and music codec 1908 may process the digital signal, and the digital signal may be further processed by the audio analyzer 140. In a particular implementation, the audio analyzer 140 of the voice and music codec 1908 may provide the digital signal to the codec 1934. The codec 1934 may convert the digital signal to an analog signal using the digital-to-analog converter 1902 and may provide the analog signal to one or more speakers 160.

[0188] In a particular implementation, device 1900 may be included in a system-in-package or system-on-chip device 1922. In a particular implementation, memory 1986, processor 1906, processor 1910, display controller 1926, codec 1934, and modem 1970 are included in a system-in-package or system-on-chip device 1922. In a particular implementation, input device 1930 and power supply 1944 are coupled to system-in-package or system-on-chip device 1922. In addition, in a particular implementation, as shown in FIG. Fig.19 As shown in , the display 1928, input device 1930, one or more microphones 120, one or more speakers 160, antenna 1952, and power supply 1944 are located external to the system-on-chip device 1922. In a particular implementation, each of the display 1928, input device 1930, one or more microphones 120, one or more speakers 160, antenna 1952, and power supply 1944 can be coupled to a component of the system-on-chip device 1922, such as an interface or a controller.

[0189] Device 1900 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet device, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a head-mounted device, an augmented reality head-mounted device, a mixed reality head-mounted device, a virtual reality head-mounted device, an aircraft, a home automation system, a voice-activated device, a wireless speaker and voice-activated device, a portable electronic device, an automobile, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0190] In conjunction with the described implementation, an apparatus includes means for receiving an input audio signal at a first device. For example, the means for receiving may correspond to an audio adjuster 148, a neural network 150, an audio analyzer 140, one or more processors 190, a device 102, Figure 1 System 100, Figure 6 Receiver 640, codec 1934, digital-to-analog converter 1904, voice and music codec 1908, antenna 1952, transceiver 1950, modem 1970, processor 1906, one or more processors 1910, Fig.19 device 1900, one or more other circuits or components configured to receive an input audio signal, or any combination thereof.

[0191] The apparatus also includes means for processing the input audio signal based on the combined representation of the plurality of sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the plurality of sound sources from the input audio signal. For example, the means for processing may correspond to the audio adjuster 148, the neural network 150, the audio analyzer 140, the one or more processors 190, the device 102, Figure 1 system 100, voice and music codec 1908, processor 1906, one or more processors 1910, Fig.19 device 1900, one or more other circuits or components configured to process an input audio signal based on a combined representation of a plurality of sound sources, or any combination thereof.

[0192] The apparatus also includes means for providing the output audio signal to the second device. For example, the means for providing may correspond to the audio adjuster 148, the neural network 150, the audio analyzer 140, the one or more processors 190, the device 102, Figure 1 System 100, Figure 7transmitter 740, codec 1934, digital-to-analog converter 1902, voice and music codec 1908, antenna 1952, transceiver 1950, modem 1970, processor 1906, one or more processors 1910, Fig.19 device 1900, one or more other circuits or components configured to provide an output audio signal to a second device, or any combination thereof.

[0193] In some specific implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 132 or memory 1986) stores instructions (e.g., instructions 1956), which, when executed by one or more processors (e.g., one or more processors 190, one or more processors 1910, or processor 1906), causes the one or more processors to receive an input audio signal (e.g., input audio signal 126) at a first device (e.g., device 102). The instructions, when executed by the one or more processors, also cause the one or more processors to process the input audio signal based on a combined representation (e.g., combined sound source representation 147A) of multiple sound sources (e.g., sound source 184A and sound source 184B) to generate an output audio signal (e.g., output audio signal 135). The combined representation is used to selectively retain or remove sounds of multiple sound sources from the input audio signal. The instructions, when executed by the one or more processors, further cause the one or more processors to provide an output audio signal to a second device (e.g., one or more speakers 160 or device 702).

[0194] Specific aspects of the present disclosure are described below in various sets of related embodiments:

[0195] According to embodiment 1, a device includes: one or more processors, wherein the one or more processors are configured to: receive an input audio signal; process the input audio signal based on a combined representation of multiple sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the multiple sound sources from the input audio signal; and provide the output audio signal to a second device.

[0196] Embodiment 2 includes a device according to embodiment 1, wherein the one or more processors are configured to use the combined representation to retain the sounds of the multiple sound sources and remove other sounds of one or more additional sound sources from the input audio signal based on a retention flag having a first value.

[0197] Embodiment 3 includes a device according to embodiment 2, wherein the one or more processors are configured to set the retention flag to have the first value indicating that the sounds of the multiple sound sources are to be retained in response to a detected condition indicating that processing of the input audio signal is to be initiated, wherein the first value of the retention flag is based on user input, a default configuration, a configuration input from an application, a configuration request from another device, or a combination thereof.

[0198] Embodiment 4 includes the apparatus of any one of Embodiments 1 to 3, wherein the plurality of sound sources includes one or more authorized users.

[0199] Embodiment 5 includes the apparatus of any one of Embodiments 1-4, wherein the plurality of sound sources includes emergency vehicles.

[0200] Embodiment 6 includes a device according to any one of Embodiments 1 to 5, wherein the one or more processors are configured to use the combined representation to remove the sounds of the multiple sound sources from the input audio signal and retain other sounds of one or more additional sound sources based on a retention flag having a second value.

[0201] Embodiment 7 includes a device according to embodiment 6, wherein the one or more processors are configured to set the retention flag to have a second value indicating that the sounds of the multiple sound sources are to be removed in response to a detected condition indicating that processing of the input audio signal is to be initiated, wherein the second value of the retention flag is based on user input, a default configuration, a configuration input from an application, a configuration request from another device, or a combination thereof.

[0202] Embodiment 8 includes the apparatus of any one of Embodiments 1 to 7, wherein the plurality of sound sources includes traffic, wind, reverberation, channel distortion, another non-speech sound source, a person, or a combination thereof.

[0203] Embodiment 9 includes the apparatus of any one of Embodiments 1 to 8, wherein the plurality of sound sources are associated with background noise in a particular environment.

[0204] Embodiment 10 includes the apparatus of Embodiment 9, wherein the specific environment corresponds to the interior of a specific type of vehicle.

[0205] Embodiment 11 includes the apparatus of any one of Embodiments 1 to 10, wherein the combined representation is based on a specific sound from a specific sound source, and wherein the specific sound source is the same sound source type as one of the plurality of sound sources.

[0206] Embodiment 12 includes the apparatus of any one of Embodiments 1 to 11, wherein the one or more processors are further configured to update the combined representation based on the sound of any sound source of the plurality of sound sources.

[0207] Embodiment 13 includes an apparatus according to any one of Embodiments 1 to 12, wherein the one or more processors are further configured to generate the combined representation based on individual representations of the plurality of sound sources based on a combination setting.

[0208] Embodiment 14 includes the device of embodiment 13, wherein the one or more processors are further configured to update the combined setting based on user input, the detected condition, or both.

[0209] Embodiment 15 includes an apparatus according to any one of Embodiments 1 to 14, wherein the multiple sound sources include at least a first sound source and a second sound source, wherein a first representation of the first sound source indicates a first value of a specific feature, wherein a second representation of the second sound source indicates a second value of the specific feature, and wherein the value of the specific feature indicated by the combined representation is based on the first value and the second value.

[0210] Embodiment 16 includes the apparatus of embodiment 15, wherein the first representation includes one or more spectrograms based on sounds from a particular sound source of the same type as the first sound source.

[0211] Embodiment 17 includes the apparatus of Embodiment 15 or Embodiment 16, wherein the combined representation corresponds to a concatenation of a first representation of the first sound source and a second representation of the second sound source.

[0212] Embodiment 18 includes the apparatus of any one of Embodiments 1 to 17, wherein the input audio signal is processed using a neural network to generate the output audio signal.

[0213] Embodiment 19 includes an apparatus according to embodiment 18, wherein the neural network includes a convolutional neural network (CNN), an autoregressive (AR) generation network, an audio generation network (AGN), an attention network (AN), a long short-term memory (LSTM) network, or a combination thereof.

[0214] Embodiment 20 includes a device according to embodiment 18 or embodiment 19, wherein the device also includes a sound source encoder, wherein the sound source encoder is configured to process sounds from one or more sound sources to generate representations of the one or more sound sources, wherein the sound source encoder and the neural network are jointly trained.

[0215] Embodiment 21 includes the apparatus of any one of Embodiments 1 to 20, further comprising a receiver configured to receive audio data representing the input audio signal.

[0216] Embodiment 22 includes the device of any one of Embodiments 1 to 21, further comprising a transmitter configured to transmit audio data to the second device, the audio data being based on the output audio signal.

[0217] According to embodiment 23, a method includes: receiving an input audio signal at a first device; processing the input audio signal based on a combined representation of multiple sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the multiple sound sources from the input audio signal; and providing the output audio signal to a second device.

[0218] Embodiment 24 includes the method of embodiment 23, wherein based on a retain flag having a first value, the combined representation is used to retain the sounds of the plurality of sound sources and remove other sounds of one or more additional sound sources from the input audio signal.

[0219] Embodiment 25 includes a method according to embodiment 24, the method further comprising setting the retention mark to have the first value indicating that the sounds of the multiple sound sources are to be retained in response to a detected condition indicating that processing of the input audio signal is to be initiated, wherein the first value of the retention mark is based on user input, a default configuration, a configuration input from an application, a configuration request from another device, or a combination thereof.

[0220] Embodiment 26 includes the method of any one of Embodiments 23 to 25, wherein the plurality of sound sources includes one or more authorized users.

[0221] Embodiment 27 includes the method of any one of Embodiments 23 to 26, wherein the plurality of sound sources includes emergency vehicles.

[0222] Embodiment 28 includes a method according to any one of Embodiments 23 to 27, wherein based on a retention flag having a second value, the combined representation is used to remove the sounds of the multiple sound sources from the input audio signal and retain other sounds of one or more additional sound sources.

[0223] Embodiment 29 includes a method according to embodiment 28, the method further comprising setting the retention mark to have a second value indicating that the sounds of the multiple sound sources are to be removed in response to a detected condition indicating that processing of the input audio signal is to be initiated, wherein the second value of the retention mark is based on user input, a default configuration, a configuration input from an application, a configuration request from another device, or a combination thereof.

[0224] Embodiment 30 includes the method of any one of Embodiments 23 to 29, wherein the plurality of sound sources include traffic, wind, reverberation, channel distortion, another non-speech sound source, a person, or a combination thereof.

[0225] Embodiment 31 includes the method of any one of Embodiments 23 to 30, wherein the plurality of sound sources are associated with background noise in a particular environment.

[0226] Embodiment 32 comprises the method of Embodiment 31, wherein the specific environment corresponds to the interior of a specific type of vehicle.

[0227] Embodiment 33 includes a method according to any one of Embodiments 23 to 32, wherein the combined representation is based on a specific sound from a specific sound source, and wherein the specific sound source is the same sound source type as one of the multiple sound sources.

[0228] Embodiment 34 includes the method of any one of Embodiments 23 to 33, further comprising updating the combined representation based on the sound of any sound source of the plurality of sound sources.

[0229] Embodiment 35 includes the method according to any one of Embodiments 23 to 34, further comprising generating the combined representation based on individual representations of the multiple sound sources based on a combination setting.

[0230] Embodiment 36 includes the method of embodiment 35, further comprising updating the combined setting based on user input, the detected condition, or both.

[0231] Embodiment 37 includes a method according to any one of Embodiments 23 to 36, wherein the multiple sound sources include at least a first sound source and a second sound source, wherein a first representation of the first sound source indicates a first value of a specific feature, wherein a second representation of the second sound source indicates a second value of the specific feature, and wherein the value of the specific feature indicated by the combined representation is based on the first value and the second value.

[0232] Embodiment 38 includes the method of Embodiment 37, wherein the first representation includes one or more spectrograms based on sounds from a particular sound source of the same type as the first sound source.

[0233] Embodiment 39 includes the method according to Embodiment 37 or Embodiment 38, wherein the combined representation corresponds to a concatenation of a first representation of the first sound source and a second representation of the second sound source.

[0234] Embodiment 40 includes the method of any one of Embodiments 23 to 39, wherein the input audio signal is processed using a neural network to generate the output audio signal.

[0235] Embodiment 41 comprises the method according to embodiment 40, wherein the neural network comprises a convolutional neural network (CNN), an autoregressive (AR) generation network, an audio generation network (AGN), an attention network (AN), a long short-term memory (LSTM) network, or a combination thereof.

[0236] Embodiment 42 includes a method according to embodiment 40 or embodiment 41, the method also comprising processing sounds from one or more sound sources using a sound source encoder to generate representations of the one or more sound sources, wherein the sound source encoder and the neural network are jointly trained.

[0237] According to embodiment 43, a device includes: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to any one of embodiments 23 to embodiment 42.

[0238] According to embodiment 44, a non-transitory computer-readable medium stores instructions, which, when executed by a processor, cause the processor to perform a method according to any one of embodiments 23 to 42.

[0239] According to embodiment 45, an apparatus includes means for performing the method according to any one of embodiments 23 to 42.

[0240] According to embodiment 46, a non-transitory computer-readable medium stores instructions, which, when executed by one or more processors, cause the one or more processors to: receive an input audio signal at a first device; process the input audio signal based on a combined representation of multiple sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the multiple sound sources from the input audio signal; and provide the output audio signal to a second device.

[0241] Embodiment 47 includes the non-transitory computer readable medium of Embodiment 46, wherein the input audio signal is processed using a neural network to generate the output audio signal.

[0242] According to embodiment 48, an apparatus includes: a component for receiving an input audio signal at a first device; a component for processing the input audio signal based on a combined representation of multiple sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the multiple sound sources from the input audio signal; and a component for providing the output audio signal to a second device.

[0243] Embodiment 49 comprises the apparatus of embodiment 48, wherein the means for receiving, the means for processing and the means for providing are integrated into at least one of the following items: a smart speaker, a speaker bar, a computer, a tablet computer, a display device, a television, a game console, a music player, a radio, a digital video player, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aviation vehicle, a home automation system, a voice-activated device, a wireless speaker and a voice-activated device, a portable electronic device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station or a mobile device.

[0244] It will also be appreciated by the skilled person that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the specific implementations disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the overall system. The skilled person may implement the described functionality in different ways for each specific application, and such specific implementation decisions will not be interpreted as causing a departure from the scope of the present disclosure.

[0245] The steps of the method or algorithm described in conjunction with the specific implementation disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral with the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In an alternative, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.

[0246] The previous description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but should be granted the broadest scope that may be consistent with the principles and novel features as defined by the following claims.

Claims

1. A device, the device include: One or more processors configured to: receiving an input audio signal; processing the input audio signal based on a combined representation of a plurality of sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the plurality of sound sources from the input audio signal; as well as The output audio signal is provided to a second device.

2. The apparatus of claim 1 , wherein the one or more processors are configured to use the combined representation to retain the sounds of the plurality of sound sources and remove other sounds of one or more additional sound sources from the input audio signal based on a retention flag having a first value.

3. A device according to claim 2, wherein the one or more processors are configured to set the retention flag to have the first value indicating that the sounds of the multiple sound sources are to be retained in response to a detected condition indicating that processing of the input audio signal is to be initiated, wherein the first value of the retention flag is based on user input, a default configuration, a configuration input from an application, a configuration request from another device, or a combination thereof.

4. The device of claim 1, wherein the plurality of sound sources comprises one or more authorized users.

5. The apparatus of claim 1, wherein the plurality of sound sources comprises emergency vehicles.

6. The apparatus of claim 1 , wherein the one or more processors are configured to use the combined representation to remove the sounds of the plurality of sound sources from the input audio signal and retain other sounds of one or more additional sound sources based on a retention flag having a second value.

7. A device according to claim 6, wherein the one or more processors are configured to set the retention flag to have the second value indicating that the sounds of the multiple sound sources are to be removed in response to the detected condition indicating that processing of the input audio signal is to be initiated, wherein the second value of the retention flag is based on user input, a default configuration, a configuration input from an application, a configuration request from another device, or a combination thereof.

8. The apparatus of claim 1, wherein the plurality of sound sources comprises traffic, wind, reverberation, channel distortion, another non-voice sound source, a person, or a combination thereof.

9. The apparatus of claim 1, wherein the plurality of sound sources are associated with background noise in a particular environment.

10. The apparatus of claim 9, wherein the specific environment corresponds to the interior of a specific type of vehicle.

11. The apparatus of claim 1, wherein the combined representation is based on a specific sound from a specific sound source, and wherein the specific sound source is of the same sound source type as one of the plurality of sound sources.

12. The device of claim 1, wherein the one or more processors are further configured to update the combined representation based on the sound of any sound source of the plurality of sound sources.

13. The device of claim 1, wherein the one or more processors are further configured to generate the combined representation based on individual representations of the plurality of sound sources based on a combination setting.

14. The device of claim 13, wherein the one or more processors are further configured to update the combined setting based on user input, the detected condition, or both.

15. An apparatus according to claim 1, wherein the plurality of sound sources comprises at least a first sound source and a second sound source, wherein a first representation of the first sound source indicates a first value of a specific feature, wherein a second representation of the second sound source indicates a second value of the specific feature, and wherein the value of the specific feature indicated by the combined representation is based on the first value and the second value.

16. The apparatus of claim 15, wherein the first representation comprises one or more spectrograms based on sounds from a particular sound source of the same type as the first sound source.

17. The apparatus of claim 15, wherein the combined representation corresponds to a concatenation of a first representation of the first sound source and a second representation of the second sound source.

18. The device of claim 1, wherein the one or more processors are configured to process the input audio signal using a neural network to generate the output audio signal.

19. The apparatus of claim 18, wherein the neural network comprises a convolutional neural network (CNN), an autoregressive (AR) generative network, an audio generation network (AGN), an attention network (AN), a long short-term memory (LSTM) network, or a combination thereof.

20. The apparatus of claim 18, further comprising a sound source encoder configured to process sounds from one or more sound sources to generate representations of the one or more sound sources, wherein the sound source encoder and the neural network are jointly trained.

21. The device of claim 1, further comprising a receiver configured to receive audio data representing the input audio signal.

22. The device of claim 1, further comprising a transmitter configured to transmit audio data to the second device, the audio data being based on the output audio signal.

23. A method, the method comprising: include: receiving an input audio signal at a first device; processing the input audio signal based on a combined representation of a plurality of sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the plurality of sound sources from the input audio signal; as well as The output audio signal is provided to a second device.

24. The method of claim 23, wherein based on a retain flag having a first value, the combined representation is used to retain the sounds of the plurality of sound sources and remove other sounds of one or more additional sound sources from the input audio signal.

25. The method of claim 23, wherein the plurality of sound sources are associated with background noise in a particular environment.

26. The method of claim 25, wherein the specific environment corresponds to the interior of a specific type of vehicle.

27. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: receiving an input audio signal at a first device; processing the input audio signal based on a combined representation of a plurality of sound sources to generate an output audio signal, wherein the combined representation is used to selectively retain or remove sounds of the plurality of sound sources from the input audio signal; and The output audio signal is provided to a second device.

28. The non-transitory computer readable medium of claim 27, wherein the input audio signal is processed using a neural network to generate the output audio signal.

29. A device, the device include: means for receiving an input audio signal at a first device; means for processing the input audio signal to generate an output audio signal based on a combined representation of a plurality of sound sources, wherein the combined representation is used to selectively retain or remove sounds of the plurality of sound sources from the input audio signal; as well as Means for providing the output audio signal to a second device.

30. An apparatus according to claim 29, wherein the means for receiving, the means for processing, and the means for providing are integrated into at least one of the following items: a smart speaker, a speaker bar, a computer, a tablet computer, a display device, a television, a game console, a music player, a radio, a digital video player, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aviation vehicle, a home automation system, a voice-activated device, a wireless speaker and a voice-activated device, a portable electronic device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, or a mobile device.

Citation Information

Patent Citations

  • Distributed audio capture and mixing

    CN108369811A

  • Linear filtering for noise-suppressed speech detection

    CN112424864A

  • Systems and methods for integrated conferencing platform

    CN112425146A

  • Method to remove talker interference to noise estimator

    CN114080637A

  • Ambient sound activated headphone

    US10361673B1