Sound event detection learning

Abstract

A device includes a processor configured to receive an audio data sample and provide the audio data sample to a first neural network to generate a first output corresponding to a first set of sound categories. The processor is also configured to provide the audio data sample to a second neural network to generate a second output corresponding to a second set of sound categories. The second category count of the second set of sound categories is greater than the first category count of the first set of sound categories. The processor is also configured to provide the first output to a neural adapter to generate a third output corresponding to the second set of sound categories. The processor is also configured to provide the second output and the third output to a merging adapter to generate sound event recognition data based on the audio data sample.

Description

[0001] Claiming priority

[0002] This application claims priority to commonly owned U.S. Provisional Patent Application No. 62 / 939,801, filed on November 25, 2019, and U.S. Non-Provisional Patent Application No. 17 / 102,797, filed on November 24, 2020, the contents of each of which are expressly incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to sound event detection and updating sound event detection models. Background Art

[0004] Advances in technology have led to smaller and more powerful computing devices. For example, there are currently a variety of portable personal computing devices, including wireless phones (such as mobile phones and smart phones), tablet computers and laptop computers, which are small, lightweight and easy for users to carry. These devices can transmit voice and data packets over wireless networks. In addition, many such devices contain additional functions, such as digital cameras, digital video cameras, digital recorders and audio file players. In addition, such devices can process executable instructions, including software applications that can be used to access the Internet, such as web browser applications. Therefore, these devices can include significant computing capabilities.

[0005] Sound event detection (SED) is a research field that has made progress recently. SED attempts to identify sound events (e.g., slamming of a door, car horns, etc.) in audio signals. SED systems are usually trained using supervised machine learning techniques to identify a set of specific sounds identified in labeled training data. Therefore, each SED system is often domain-specific (e.g., capable of classifying a set of predetermined sounds). After the SED system is trained, it is difficult to update the SED system to identify new sounds that are not identified in the labeled training data. For example, a set of labeled audio data samples can be used to train the SED system, and these samples include a series of urban noises, such as car horns, alarms, slamming of doors, and engine sounds. In this example, if there is a need to identify unlabeled sounds in a set of labeled audio data samples, such as doorbells, then updating the SED system to identify doorbells involves using both the labeled audio data samples for the doorbells and the original set of labeled audio data samples to completely retrain the SED system. Therefore, the computational resources (e.g., processor cycles, memory, etc.) required to train an SED system to recognize a new sound are roughly the same as those required to generate an entirely new SED system. Furthermore, over time, as more sounds are added to be recognized, the amount of audio data samples that must be maintained and used to train the SED system may become unmanageable. Summary of the invention

[0006] In one particular aspect, a device includes a processor configured to receive an audio data sample. The processor is further configured to provide the audio data sample to a first neural network that is trained to generate a first output corresponding to a first class count of a first set of sound classes. The processor is further configured to provide the audio data sample to a second neural network to generate a second output corresponding to a second class count of a second set of sound classes. The second class count is greater than the first class count. The processor is further configured to provide the first output to a neural adapter to generate a third output corresponding to the second class count of the second set of sound classes, and to provide the second output and the third output to a merging adapter to generate sound event recognition data based on the audio data sample.

[0007] In one particular aspect, a method includes receiving an audio data sample and providing the audio data sample to a first neural network that is trained to generate a first output corresponding to a first class count for a first set of sound classes. The method also includes providing the audio data sample to a second neural network to generate a second output corresponding to a second class count for a second set of sound classes. The second class count is greater than the first class count. The method also includes providing the first output to a neural adapter to generate a third output corresponding to the second class count for the second set of sound classes. The method also includes providing the second output and the third output to a merging adapter to generate sound event recognition data based on the audio data sample.

[0008] In one particular aspect, a non-transitory computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to provide an audio data sample to a first neural network that is trained to generate a first output corresponding to a first class count for a first set of sound classes. The instructions also cause the processor to provide the audio data sample to a second neural network to generate a second output corresponding to a second class count for a second set of sound classes. The second class count is greater than the first class count. The instructions also cause the processor to provide the first output to a neural adapter to generate a third output corresponding to the second class count for the second set of sound classes. The instructions also cause the processor to provide the second output and the third output to a merging adapter to generate sound event recognition data based on the audio data sample.

[0009] In one particular aspect, a device includes a component for generating a first output based on an audio data sample, the first output corresponding to a first category count of a first set of sound categories. The device also includes a component for generating a second output based on the audio data sample, the second output corresponding to a second category count of a second set of sound categories. The second category count is greater than the first category count. The device also includes a component for generating a third output based on the first output, the third output corresponding to the second category count of the second set of sound categories. The device also includes a component for generating sound event identification data based on the third output and the second output.

[0010] In one particular aspect, a device includes a memory and a processor coupled to the memory. The memory stores a sound event classifier that is trained to generate first sound recognition data in response to an input of audio data representing one or more of a first group of sound categories. The processor is configured to generate an updated sound event classifier that is trained to generate second sound recognition data in response to an input of audio data representing one or more of a second group of sound categories. The second group of sound categories includes the first group of sound categories and one or more additional sound categories. The updated sound event classifier includes a sound event classifier, a second sound event classifier, a neural adapter, and a merge adapter. The neural adapter includes one or more adapter layers that are configured to receive an input having a data element count corresponding to an output layer of the sound event classifier, and are configured to generate an output having a second data element count corresponding to an output layer of the second sound event classifier. The merge adapter includes one or more aggregation layers and an output layer. The one or more aggregation layers are configured to merge the output from the neural adapter and the output of the second neural network. The output layer is configured to generate second sound recognition data.

[0011] In one particular aspect, a device includes a component for storing a sound event classifier, the sound event classifier being trained to generate first sound recognition data in response to an input of audio data representing one or more of a first set of sound categories. The device also includes a component for generating an updated sound event classifier, the updated sound event classifier being trained to generate second sound recognition data in response to an input of audio data representing one or more of a second set of sound categories. The second set of sound categories includes a first set of sound categories and one or more additional sound categories. The updated sound event classifier includes a sound event classifier, a second sound event classifier, a neural adapter, and a merge adapter. The neural adapter includes one or more adapter layers, the one or more adapter layers being configured to receive an input having a data element count corresponding to an output layer of the sound event classifier, and being configured to generate an output having a second data element count corresponding to an output layer of the second sound event classifier. The merge adapter includes one or more aggregation layers and an output layer. The one or more aggregation layers are configured to merge the output from the neural adapter and the output of the second sound event classifier. The output layer is configured to generate the second sound recognition data.

[0012] In one particular aspect, a method includes generating a second neural network based on a first neural network of a first sound event classifier. The first neural network includes an input layer, a hidden layer, and a first output layer, and the second neural network includes a copy of the input layer of the first neural network, a copy of the hidden layer of the first neural network, and a second output layer coupled to the copy of the hidden layer. The first output layer includes a first output node count and the second output layer includes a second output node count, wherein the second output node count is greater than the first output node count. The method also includes linking the first neural network and the second neural network to one or more adapter networks, and providing labeled training data as input to the first neural network and the second neural network. The method also includes modifying the outputs of the first neural network and the second neural network via the one or more adapter networks. The method also includes training a second sound event classifier by updating the link weights of the second neural network and the one or more adapter networks based on the outputs of the adapter networks and the labels of the labeled training data.

[0013] In one particular aspect, a non-transitory computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to generate a second neural network based on a first neural network of a first sound event classifier. The first neural network includes an input layer, a hidden layer, and a first output layer, and the second neural network includes a copy of the input layer of the first neural network, a copy of the hidden layer of the first neural network, and a second output layer coupled to the copy of the hidden layer. The first output layer includes a first output node count and the second output layer includes a second output node count, wherein the second output node count is greater than the first output node count. The instructions also cause the processor to link the first neural network and the second neural network to one or more adapter networks. The instructions also cause the processor to provide labeled training data as input to the first neural network and the second neural network, and to modify outputs of the first neural network and the second neural network via the one or more adapter networks. The instructions also cause the processor to train a second sound event classifier by updating link weights of the second neural network and the one or more adapter networks based on outputs of the adapter networks and labels of the labeled training data.

[0014] Other aspects, advantages, and features of the present disclosure will become apparent upon careful examination of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and Claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a block diagram of an example of a device configured to generate sound recognition data in response to audio data samples and configured to generate an updated sound event detection model.

[0016] Figure 2 is a block diagram illustrating aspects of a sound event detection model according to certain examples.

[0017] Figure 3 is a diagram illustrating aspects of generating an updated sound event detection model according to a specific example.

[0018] Figure 4 is a diagram illustrating aspects of generating sound event detection output data using an updated sound event detection model according to a specific example.

[0019] Figure 5 It is combined Figure 1 , Figure 3 or Figure 4 Illustrative examples of various aspects of the updated sound event detection model for a vehicle.

[0020] Figure 6 Shows the combination Figure 1 , Figure 3 or Figure 4An updated sound event detection model for various aspects of a virtual reality or augmented reality headset.

[0021] Figure 7 Shows the combination Figure 1 , Figure 3 or Figure 4 An updated sound event detection model for various aspects of wearable electronic devices.

[0022] Figure 8 Shows the combination Figure 1 , Figure 3 or Figure 4 The updated sound event detection model has various aspects for voice-controlled speaker systems.

[0023] Fig. 9 Shows the combination Figure 1 , Figure 3 or Figure 4 An updated sound event detection model for all aspects of the camera.

[0024] Fig.10 Shows the combination Figure 1 , Figure 3 or Figure 4 An updated sound event detection model for various aspects of mobile devices.

[0025] Fig.11 Shows the combination Figure 1 , Figure 3 or Figure 4 An updated sound event detection model for various aspects of aviation equipment.

[0026] Fig.12 Shows the combination Figure 1 , Figure 3 or Figure 4 An updated sound event detection model for various aspects of the headset.

[0027] Fig.13 Shows the combination Figure 1 , Figure 3 or Figure 4 An updated sound event detection model for various aspects of home appliances.

[0028] Fig.14 It is shown that the use Figure 1 , Figure 3 or Figure 4 A flowchart of an example of a method for generating sound event detection data using an updated sound event detection model.

[0029] Fig.15 It shows that the generated Figure 1 , Figure 3 or Figure 4A flow chart of an example of a method of updating a sound event detection model. DETAILED DESCRIPTION

[0030] The sound event detection model can be trained using machine learning techniques. For example, a neural network can be trained as a sound event classifier using back propagation or other machine learning training techniques. The sound event detection model trained in this way can be small enough (in terms of the storage space occupied) and simple enough (in terms of the computing resources used during operation) for a portable computing device to store and use the sound event detection model. However, compared with using a trained sound event detection model to perform sound event detection, the training process uses much more processing resources. In addition, the training process uses a large set of labeled training data, including many audio data samples of each sound category that the sound event detection model is trained to detect. Therefore, in terms of memory utilization or other computing resources, training a sound event detection model from scratch on a portable computing device or another computing device with limited resources may be daunting. Therefore, users who wish to use a sound event detection model on a portable computing device may be limited to downloading a pre-trained sound event detection model to a portable computing device from a computing device with less limited resources or a library of pre-trained sound event detection models. Therefore, users have limited customization options.

[0031] The disclosed systems and methods use transfer learning techniques to generate updated sound event detection models in a manner that is much less resource intensive than training a sound event detection model from scratch. According to one particular aspect, transfer learning techniques can be used to generate an updated sound event detection model based on a previously trained sound event detection model (also referred to herein as a "base model"). The updated sound event detection model is configured to detect more types of sound events than the base model. For example, the base model is trained to detect any one of a first group of sound events, each of which corresponds to a sound category in a first group of sound categories, and the updated sound event detection model is trained to detect any one of the first group of sound events and any one of the second group of sound events, each of which corresponds to a sound category in a second group of sound categories. Therefore, the disclosed systems and methods reduce the computing resources (e.g., memory, processor cycles, etc.) used to generate updated sound event detection models. As an example of a use case for the disclosed systems and methods, a portable computing device can be used to generate a customized sound event detector.

[0032] According to a specific aspect, an updated sound event detection model is generated based on a previously trained sound event detection model, a subset of training data for training the previously trained sound event detection model, and one or more sets of training data corresponding to one or more additional sound categories that the updated sound event detection model can detect. In this regard, the previously trained sound event detection model (e.g., the first model) is retained and unchanged. In addition, a copy of the previously trained sound event detection model (e.g., the second model) is generated and modified to have a new output layer. The new output layer includes an output node for each sound category that the updated sound event detection model can detect. For example, if the first model is configured to detect ten different sound categories, the output layer of the first model may include ten output nodes. In this example, if the updated sound event detection model is to be trained to detect twelve different sound categories (e.g., the ten sound categories that the first model is configured to detect plus two additional sound categories), the output layer of the second model includes twelve output nodes.

[0033] One or more adapter networks are generated to link the output of the first model and the output of the second model. For example, the (multiple) adapter network converts the output of the first model to have a size corresponding to the output of the second model. To illustrate, in the example of the previous paragraph, the first model includes ten output nodes and generates an output with ten data elements, while the second model includes twelve output nodes and generates an output with twelve data elements. In this example, the (multiple) adapter network modifies the output of the first model to have twelve data elements. The (multiple) adapter network also combines the output of the second model with the modified output of the first model to generate a sound classification output of an updated sound event detection model.

[0034] The updated sound event detection model is trained using labeled training data, which includes audio data samples and labels for each sound category that the updated sound event detection model is trained to detect. However, since the first model has been trained to accurately detect the first set of sound categories, the labeled training data contains much fewer audio data samples for the first set of sound categories than the audio data samples originally used to train the first model. For illustration, hundreds or thousands of audio data samples for each sound category of the first set of sound categories can be used to train the first model. In contrast, the labeled training data used to train the updated sound event detection model can include tens or fewer audio data samples for each sound category of the first set of sound categories. The labeled training data also includes audio data samples for each sound category of the second set of sound categories. The audio data samples for the second set of sound categories can also include tens or fewer audio data samples for each sound category of the second set of sound categories.

[0035] Backpropagation or another machine learning technique is used to train the second model and one or more adapter networks. During this process, the first model is unchanged, which limits or eliminates the risk of the first model forgetting its previous training. For example, in its previous training, the first model was trained to accurately detect a first set of sound categories using a larger labeled training data set. Retraining the first model using the relatively smaller labeled training data set used during backpropagation runs the risk of causing the first model to lose accuracy (sometimes referred to as "forgetting" some of its previous training). Keeping the first model unchanged while training the updated sound event detector model can reduce the risk of forgetting the first set of sound categories.

[0036] Furthermore, before training, the second model is identical to the first model except for the output layer of the second model and its interconnections. Therefore, at the starting point of training, the second model is expected to be closer to convergence (e.g., closer to the training termination condition) than the random seed model. Therefore, the number of iterations required to train the second model should be less than the number of iterations required to train the first model.

[0037] After the updated sound event detection model has been trained, the updated sound event detection model can be used (rather than using the first model alone) to detect sound events. For example, when an audio data sample is received, the audio data sample is provided as input to the updated sound event detection model, and the updated sound event detection model generates a sound classification output. In the updated sound event detection model, the audio data sample is provided as input to the first model and the second model. The first model generates a first output, which is provided to (multiple) adapter networks, modified and combined with the second output from the second model. (Multiple) adapter networks generate a third output corresponding to the sound classification output of the updated sound event detection model.

[0038] Certain aspects of the present disclosure are described below with reference to the accompanying drawings. In the description, common features are represented by common reference numerals. As used herein, various terms are used only for the purpose of describing specific embodiments and are not intended to limit the embodiments. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some embodiments and plural in other embodiments. For illustration, Figure 1 Depicts a device including one or more microphones ( Figure 114”), indicating that in some embodiments, the device 100 includes a single microphone 114, while in other embodiments, the device 100 includes multiple microphones 114. For ease of reference, these features are generally introduced as "one or more" features and are subsequently referred to in the singular or optionally plural form (generally indicated by a term beginning with "(s)") unless aspects related to multiple features are being described.

[0039] It can be further understood that the term "including" can be used interchangeably with "comprising". In addition, it should be understood that the term "wherein" can be used interchangeably with "wherein". As used herein, "exemplary" can indicate examples, embodiments and / or aspects, and should not be interpreted as limiting or indicating preferences or preferred embodiments. As used herein, ordinal terms (e.g., "first", "second", "third", etc.) used to modify elements such as structures, components, operations, etc. do not themselves represent any priority or order of elements relative to another element, but only distinguish the element from another (if not using ordinal terms) element with the same name. As used herein, the term "group" refers to one or more specific elements, and the term "plurality" refers to multiple (e.g., two or more) specific elements.

[0040] As used herein, "coupling" may include "communicative coupling", "electrical coupling" or "physical coupling", and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicative coupling, electrical coupling or physical coupling) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device or in different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling. In some embodiments, two devices (or components) such as those communicatively coupled in an electrical communication manner may directly or indirectly send and receive electrical signals (digital signals or analog signals), such as via one or more wires, buses, networks, etc. As used herein, "direct coupling" may include two devices being coupled (e.g., communicatively coupled, electrically coupled or physically coupled) without an intermediate component.

[0041] In the present disclosure, terms such as "determine", "calculate", "estimate", "convert", "adjust", etc. may be used to describe how to perform one or more operations. It should be noted that these terms should not be interpreted as limiting, and other technologies may be used to perform similar operations. In addition, as mentioned herein, "generate", "calculate", "estimate", "use", "select", "access", and "determine" may be used interchangeably. For example, "generating", "calculating", "estimating" or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating or determining a parameter (or signal), or may refer to using, selecting or accessing an already generated parameter (or signal), such as generated by another component or device.

[0042] Figure 1 is a block diagram of an example of a device 100 configured to generate sound recognition data in response to input of audio data samples. Figure 1 In some embodiments, the remote computing device 150 generates an updated sound event detection model, and the device 100 uses the updated sound event detection model to generate sound recognition data in response to the audio data sample. In some embodiments, the remote computing device 150 and the device 100 collaborate to generate an updated sound event detection model, and the device 100 uses the updated sound event detection model to generate sound recognition data in response to the audio data sample. In various embodiments, the device 100 may have a Figure 1 More or fewer components as shown.

[0043] In certain embodiments, device 100 includes processor 120 (e.g., a central processing unit (CPU)). Device 100 may include one or more additional processors 132 (e.g., one or more DSPs). Processor 120, processor(s) 132, or both may be configured to generate sound recognition data, generate updated sound event detection models, or both. For example, in Figure 1 , processor(s) 132 include a sound event detection (SED) engine 108. SED engine 108 is configured to analyze audio data samples using a sound event classifier such as base model 104 or updated model 106. Base model 104 is a previously trained sound event detection model. In some embodiments, another computing device such as remote computing device 150 trains base model 104 and base model 104 is stored as a default model on device 100, or the device downloads the base model from another computing device. In some embodiments, device 100 trains base model 104. Training base model 104 requires using a relatively large set of labeled training data (e.g., Figure 1In some implementations, whether the remote computing device 150 or the device 100 trains the base model 104, the base training data 152 is stored at the remote computing device 150, which may have a larger storage capacity (e.g., more memory) than the device 100. Figure 2 An example of a specific implementation of the base model 104 is shown. The updated model 106 is an updated acoustic event detection model that is based on the base model 104 and is trained using a model updater 110 as described further below.

[0044] exist Figure 1 In the embodiment, the device 100 also includes a memory 130 and a codec 142. The memory 130 stores instructions 124 executable by the processor 120 or (multiple) processors 132 to implement the reference Figures 3 to 15 In one example, the instructions 124 include or correspond to the SED engine 108, the model updater 110, or both. The memory 130 may also store the base model 104, the updated model 106, or both. Figure 1 In the example shown, the memory 130 stores audio data samples 126 and audio data samples 128. The audio data samples 126 include audio data samples representing one or more of the first set of sound categories used to train the base model 104. That is, the audio data samples 126 include a relatively small subset of the base training data 152. In some embodiments, when the device 100 is ready to generate the updated model 106, the device 100 downloads the audio data samples 126 from the remote computing device 150. The audio data samples 128 include audio data samples representing one or more of the second set of sound categories used to train the updated model 106. In a particular embodiment, the device 100 captures one or more of the audio data samples 128 (e.g., using (multiple) microphones 114). In some embodiments, the device 100 obtains one or more of the audio data samples 128 from another device such as the remote computing device 150. Figure 3 An example of operation of the model updater 110 to generate the updated model 106 based on the base model 104 , the audio data samples 126 , and the audio data samples 128 is shown.

[0045] exist Figure 1 , speaker(s) 118 and microphone(s) 114 can be coupled to codec 142. In one particular aspect, microphone(s) 114 are configured to receive audio representative of an acoustic environment associated with device 100 and generate audio data samples that SED engine 108 provides to base model 104 or update model 106 to generate a sound classification output. Figure 4An example of operation of updating the model 106 to generate output data indicating detection of a sound event is shown. The microphone(s) 114 may also be configured to provide audio data samples 128 to the model updater 110 or to the memory 130 for use in generating the updated model 106 .

[0046] exist Figure 1 In the example shown, the codec 142 includes a digital-to-analog converter (DAC 138) and an analog-to-digital converter (ADC 140). In a particular embodiment, the codec 142 receives analog signals from the microphone(s) 114, converts the analog signals to digital signals using the ADC 140, and provides the digital signals to the processor(s) 132. In a particular embodiment, the processor(s) 132 (e.g., a voice and music codec) provide the digital signals to the codec 142, and the codec 142 converts the digital signals to analog signals using the DAC 138 and provides the analog signals to the speaker(s) 118.

[0047] exist Figure 1 In some embodiments, the device 100 also includes an input device 122. The device 100 may also include a display 102 coupled to the display controller 112. In one particular aspect, the input device 122 includes a sensor, a keyboard, a pointing device, etc. In some embodiments, the input device 122 and the display 102 are combined in a touch screen or similar touch or motion sensitive display. The input device 122 can be used to provide a label associated with one of the audio data samples 128 to generate labeled training data for training the update model 106. In some embodiments, the device 100 also includes a modem 136 coupled to the transceiver 134. In some embodiments, the device 100 also includes a modem 136 coupled to the transceiver 134. Figure 1 , transceiver 134 is coupled to antenna 146 to enable wireless communication with other devices, such as remote computing device 150. In other examples, transceiver 134 is additionally or alternatively coupled to a communication port (e.g., an Ethernet port) to enable wired communication with other devices, such as remote computing device 150.

[0048] In a particular embodiment, device 100 is included in a system-in-package or system-on-chip device 144. In a particular embodiment, memory 130, processor 120, processor(s) 132, display controller 112, codec 142, modem 136, and transceiver 134 are included in a system-in-package or system-on-chip device 144. In a particular embodiment, input device 122 and power supply 116 are coupled to system-in-package or system-on-chip device 144. In addition, in a particular embodiment, as shown in FIG. Figure 1As shown, the display 102, input device 122, speaker(s) 118, microphone(s) 114, antenna 146, and power supply 116 are external to the system-on-chip device 144. In a particular embodiment, each of the display 102, input device 122, speaker(s) 118, microphone(s) 114, antenna 146, and power supply 116 may be coupled to a component of the system-on-chip device 144, such as an interface or controller.

[0049] Device 100 may include, correspond to, or be included in a voice activated device, an audio device, a wireless speaker and voice activated device, a portable electronic device, an automobile, a vehicle, a computing device, a communication device, an Internet of Things (IoT) device, a virtual reality (VR) device, an augmented reality (AR) device, a smart speaker, a mobile computing device, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet, a personal digital assistant, a display device, a television, a game console, an appliance, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, or any combination thereof. In a particular aspect, processor 120, processor(s) 132, or a combination thereof are included in an integrated circuit.

[0050] Figure 2 2 is a block diagram illustrating aspects of a base model 104 according to a particular example. The base model 104 is a neural network having a topology (e.g., base topology 202) and trainable parameters (e.g., base parameters 236). The base topology 202 can be represented as a set of nodes and edges (or links); however, for ease of illustration and reference, the base topology 202 is represented in Figure 2 It should be understood that Figure 2 Each layer of a network consists of a set of nodes, and links interconnect nodes of different layers. The arrangement of links depends on the type of each layer.

[0051] During back-propagation training, the base topology 202 is static and the base parameters 236 are changed. Figure 2 , the base parameters 236 include the base link weights 238. The base parameters 236 may also include other parameters, such as bias values ​​associated with one or more nodes of the base model 104.

[0052] The basic topology structure 202 includes an input layer 204, one or more hidden layers ( Figure 2) and an output layer 234. The input node count of the input layer 204 depends on the arrangement of the audio data samples to be provided to the base model 104. For example, the audio data sample may include an array or matrix of data elements, where each data element corresponds to a feature of the input audio sample. As a specific example, the audio data sample may correspond to a Mel spectrum feature extracted from one second of audio data. In this example, the audio data sample may include a 128×128 matrix of eigenvalue elements. In other examples, other audio data sample configurations or sizes may be used. The node count of the output layer 234 depends on the number of sound categories that the base model 104 is configured to detect. As an example, the output layer 234 may include an output node for each sound category.

[0053] Depending on the specific implementation, the hidden layer(s) 206 may have various configurations and various numbers of layers. Figure 2 One specific example of hidden layer(s) 206 is shown. Figure 2 , the hidden layer(s) 206 include three convolutional neural networks (CNNs), including CNN 208, CNN 228, and CNN 230. In this example, the output layer 234 includes or corresponds to the activation layer 232. For example, the activation layer 232 receives the output of the CNN 230 and applies an activation function (such as a sigmoid function) to the output to generate a set of data elements as outputs, each data element including a value or a zero value.

[0054] Figure 2 Details of one specific implementation of CNN 208, CNN 228, and CNN 230 are also shown. Figure 2 In the specific example shown, CNN 208 includes a two-dimensional (2D) convolutional layer ( Figure 2 conv2d 210 in), the maximum pooling layer ( Figure 2 The max pooling 216) and batch normalization layers ( Figure 2 Batch Normalization 226 in ). Similarly, Figure 2 , CNN 228 includes conv2d 212, max pooling 222, and batch normalization 220, and CNN 230 includes conv2d 214, max pooling 224, and batch normalization 218. In other embodiments, the hidden layer(s) 206 include a different number of CNNs or other layers.

[0055] As described above, the updated model 106 includes the base model 104, a modified copy of the base model 104, and one or more adapter networks. The modified copy of the base model 104 uses the same Figure 2The same base topology 202 is shown, except that the output layer of the modified copy includes more output nodes than the output layer 234. Additionally, before training the update model 106, the modified copy is initialized to have the same base parameters 236 as the base model 104.

[0056] Figure 3 is a diagram illustrating various aspects of generating an update model 106 according to a specific example. Figure 3 The operations described may be performed by processor 120 or Figure 1 Alternatively, reference may be made to: Figure 3 The described operations may be performed by a remote computing device 150 using audio data samples 128 captured at device 100 and audio data samples 126 from base training data 152. Figure 3 The described operations are performed at the remote computing device 150 , and the device 100 may download the update model 106 or a portion thereof, such as the delta model 302 and one or more adapter networks 314 , from the remote computing device 150 .

[0057] To generate the updated model 106, the model updater 110 copies the base model 104 and replaces it with a different output layer (e.g., Figure 3 The output layer 234 of the copy of the base model 104 is replaced with the output layer 322 in the base model 104 to generate an incremental model 302 (also referred to herein as a second model). The incremental model 302 includes the base topology 202 of the base model 104, except that the output layer 234 is replaced with the output layer 322 and links are generated to link the output nodes of the output layer 322 to the hidden layers of the incremental model 302. The model parameters of the incremental model 302 (e.g., incremental model parameters 306) are initialized to be equal to the base parameters 236. The output layer 234 of the base model 104 includes a first node count (e.g., Figure 3 , where N is a positive integer), and the output layer 322 of the incremental model 302 includes a second node count (e.g., Figure 3 , where K is a positive integer). The first node count corresponds to the sound category count of the first set of sound categories that the base model 104 is trained to recognize (e.g., the first set of sound categories includes N different sound categories that the base model 104 can recognize). The second node count corresponds to the sound category count of the second set of sound categories that the update model 106 will be trained to recognize (e.g., the second set of sound categories includes N+K different sound categories that the update model 106 will be trained to recognize). Therefore, the second set of sound categories includes the first set of sound categories (e.g., N categories) plus one or more additional sound categories (e.g., K categories).

[0058] In addition to generating the delta model 302, the model updater 110 also generates one or more adapter networks 314. Figure 3 , adapter network(s) 314 include neural adapter 310 and merge adapter 308. Neural adapter 310 includes one or more adapter layers (e.g., Figure 3 ) adapter layer(s) 312 in the example. The adapter layer(s) 312 are configured to receive input from the base model 104 and generate outputs that can be merged with the outputs of the incremental model 302. For example, the base model 104 can generate as an output a first output 352 corresponding to a first category count for a first set of sound categories. For example, the first output includes one data element (e.g., N data elements) for each node of the output layer 234. In contrast, the incremental model 302 generates as an output a second output 354 corresponding to a second category count for a second set of sound categories. For example, the second output 354 includes one data element (e.g., N+K data elements) for each node of the output layer 322. In this example, the adapter layer(s) 312 receive inputs having a first data element count and generate a third output 356 having a second data element count (e.g., N+K). In one particular example, the adapter layer(s) 312 include two fully connected layers (e.g., an input layer including N nodes and an output layer including N+K nodes, where each node of the input layer is connected to each node of the output layer).

[0059] The merge adapter 308 is configured to generate output data 318 by merging the third output 356 from the neural adapter 310 and the second output 354 of the incremental model 302. Figure 3 , the merge adapter 308 includes an aggregation layer 316 and an output layer 320. The aggregation layer 316 is configured to combine the second output 354 and the third output 356 in an element-by-element manner. For example, the aggregation layer 316 can add each element of the third output 356 from the neural adapter 310 to the corresponding element of the second output 354 from the incremental model 302, and provide the resulting merged output to the output layer 320. The output layer 320 is an activation layer that applies an activation function (such as a sigmoid function) to the merged output to generate output data 318. The output data 318 includes or corresponds to a sound event identifier 360 that indicates the sound category to which the update model 106 assigns a particular audio sample (e.g., one of the audio data samples 126 or 128).

[0060] During training, the model updater 110 provides labeled training data 304 to the base model 104 and the incremental model 302. The labeled training data 304 includes one or more of the audio data samples 126 (which are used to train the base model 104) and one or more audio data samples 128 (which correspond to new sound categories that the base model 104 is not trained to recognize). In response to a particular audio sample of the labeled training data 304, the base model 104 generates a first output 352, which is provided as an input to the neural adapter 310. In addition, in response to the particular audio sample, the incremental model 302 generates a second output 354, which is provided to the merge adapter 308 together with the third output 356 of the neural adapter 310. The merge adapter 308 merges the second output 354 and the third output 356 to generate a merged output and generates output data 318 based on the merged output.

[0061] The output data 318, the sound event identifier 360, or both are provided to the model updater 110, which compares the sound event identifier 360 with the labels associated with the particular audio sample in the labeled training data 304 and calculates an updated link weight value ( Figure 3 The updated link weights 362 in the training data are used to modify the incremental model parameters 306, the link weights of the neural adapter 310, the link weights of the merge adapter 308, or a combination thereof. The training process continues iteratively until the model updater 110 determines that a training termination condition is satisfied. For example, the model updater 110 calculates an error value based on the labeled training data 304 and the output data 318. In this example, the error value indicates how accurately the update model 106 classifies each audio data sample 126 and 128 of the labeled training data 304 based on the label associated with each audio data sample 126 and 128. In this example, the training termination condition can be satisfied when the error value is less than a threshold or the change in the error value between training iterations is less than a threshold. In some embodiments, the termination condition is satisfied when the count of the training iterations performed is greater than or equal to a threshold count.

[0062] Figure 4 is a diagram illustrating various aspects of using the update model 106 to generate sound event detection output data according to a specific example. Figure 4 The operations described may be performed by processor 120 or Figure 1 Initiated, executed or controlled by (multiple) processors 132 executing instructions 124.

[0063] exist Figure 4 In the embodiment, one or more inputs 450 including audio data samples 406 are provided to the incremental models 302 of the base model 104 and the update model 106. In a specific example, the audio data samples 406 include, correspond to, or are based on the Figure 1The audio data samples 406 may correspond to features extracted from several seconds of audio data, and the input 450 may include an array or matrix of feature data extracted from the audio data.

[0064] In response to the input 450, the base model 104 generates a first output 452, which is provided as an input to the neural adapter 310. The base model 104 generates the first output 452 using the base parameters 236 including the base link weights 238. The first output 452 of the base model 104 corresponds to the first category counts of the first set of sound categories. In an illustrative example, the first output 452 includes an array or matrix of data elements and has a data element count (e.g., N data elements) corresponding to the number of output nodes (e.g., N nodes) of the output layer 234 of the base model 104, and the number of output nodes of the output layer 234 corresponds to the first category counts of the first set of sound categories.

[0065] In response to input 450, the incremental model 302 generates a second output 454, which is provided to the merge adapter 308. The second output 454 of the incremental model 302 corresponds to second category counts for the second set of sound categories. In one illustrative example, the second output 454 includes an array or matrix of data elements and has a data element count (e.g., N+K data elements) corresponding to the number of output nodes of the output layer 322 of the incremental model 302 (e.g., N+K nodes), and the number of output nodes of the output layer 322 corresponds to the second category counts for the second set of sound categories. The incremental model 302 generates the second output 454 using updated parameters 402 including updated link weights 404, which are generated by the model updater 110 during the training process. In one specific embodiment, the updated parameters 402 correspond to satisfying the reference Figure 3 The parameters of the incremental model 302 are described for the training termination condition.

[0066] The neural adapter 310 generates a third output 456 based on the first output 452 from the base model 104. In a specific example, the neural adapter 310 generates the third output 456 based on the link weights trained by the model updater 110 during the training process. The third output 456 of the neural adapter 310 corresponds to the second class counts of the second set of sound classes. In an illustrative example, the third output 456 includes an array or matrix of data elements and has a data element count (e.g., N+K data elements) corresponding to the second class counts of the second set of sound classes.

[0067] The third output 456 from the neural adapter 310 and the second output 454 from the incremental model 302 are provided to the merge adapter 308. The aggregation layer 316 of the merge adapter 308 combines the third output 456 and the second output 454 element by element to generate a merged output 458. The output layer 320 of the merge adapter 308 generates output data 408 based on the merged output 458. In a specific example, the merge adapter 308 generates the output data 408 based on the link weights trained by the model updater 110 during the training process. The output data 408 includes sound recognition data (e.g., sound event recognition data 460) indicating the sound category associated with the audio data sample 406.

[0068] Figure 5 It is combined Figure 1 , Figure 3 or Figure 4 1 and 1 . The present invention provides an illustrative example of a vehicle 500 that implements various aspects of the updated sound event detection model 106. According to one embodiment, the vehicle 500 is an autonomous vehicle. According to other embodiments, the vehicle 500 is a car, a truck, a motorcycle, an airplane, a water vehicle, etc. Figure 5 In the embodiment, the vehicle 500 includes a screen 502 (e.g., a display such as Figure 1 102), sensor(s) 504, device 100, or a combination thereof. Sensor(s) 504 and device 100 are shown using dashed lines to indicate that these components may not be visible to passengers of vehicle 500. Device 100 may be integrated into vehicle 500 or coupled to vehicle 500.

[0069] In one particular aspect, device 100 is coupled to screen 502 and provides output to screen 502 in response to detecting or recognizing various events described herein (e.g., sound events). For example, device 100 provides output to screen 502. Figure 4 Output data 408 indicates that a recognized sound event, such as a car horn, is detected in the audio data received from the sensor(s) 504. In some embodiments, the device 100 can perform an action in response to the recognized sound event, such as activating a camera or one of the sensors 504. In one specific example, the device 100 provides an output indicating whether an action is being performed in response to the recognized sound event. In one specific aspect, a user can select an option displayed on the screen 502 to enable or disable an action performed in response to the recognized sound event.

[0070] In one particular embodiment, the sensor(s) 504 include Figure 1One or more microphones 114, vehicle occupancy sensors, eye tracking sensors, or external environment sensors (e.g., lidar sensors or cameras). In one particular aspect, sensor inputs of sensor(s) 504 indicate the location of the user. For example, sensor(s) 504 are associated with various locations within vehicle 500.

[0071] Figure 5 The device 100 in the embodiment includes a base model 104, an update model 106, an SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the vehicle 500, the device 100 omits the model updater 110. For illustration, Figure 1 The remote computing device 150 may generate the update model 106. In such an embodiment, the update model 106 may be downloaded to the carrier 500 for use by the SED engine 108.

[0072] Therefore, about Figures 1 to 4 The described techniques enable a user of the vehicle 500 to update a sound event detection model (e.g., base model 104) stored in the memory of the vehicle 500 to generate a sound event detection model (e.g., updated model 106) that is capable of detecting a new set of sound categories. In addition, the sound event detection model can be updated without excessive use of computing resources onboard the vehicle 500. For example, the vehicle 500 does not have to store all of the base training data 152 used to train the base model 104 in a local memory to avoid forgetting the training associated with the base training data 152. Instead, the model updater 110 retains the base model 104 while generating the updated model 106. The model update process also converges faster (e.g., uses fewer processor cycles) than for generating a new sound event detection model from scratch.

[0073] Figure 6 An example of device 100 coupled to or integrated into a headset 602 is shown, such as a virtual reality headset, an augmented reality headset, a mixed reality headset, an extended reality headset, a head mounted display, or a combination thereof. A visual interface device, such as display 604, is positioned in front of the user's eyes to enable headset 602 to display augmented reality or virtual reality images or scenes to the user when worn. In a specific example, display 604 is configured to display output of device 100, such as an indication of an identified sound event (e.g., sound event identification data 460). Headset 602 may include one or more sensors 606, such as Figure 1(multiple) microphones 114, cameras, other sensors, or combinations thereof. Although shown in a single location, in other embodiments, one or more sensors 606 can be located at other locations of the headset 602, such as an array of one or more microphones and one or more cameras distributed around the headset 602 to detect multimodal input.

[0074] The sensor(s) 606 can detect audio data, which the device 100 uses to detect sound events or to update the base model 104 to generate an updated model 106. For example, the device 100 provides the display 604 with Figure 4 Output data 408 indicates that a recognized sound event, such as a car horn, is detected in the audio data received from the sensor(s) 606. In some implementations, the device 100 can perform an action in response to recognizing the sound event, such as activating a camera or one of the sensor(s) 606 or providing tactile feedback to the user.

[0075] exist Figure 6 In the example shown, the device 100 includes a base model 104, an update model 106, an SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the headset 602, the device 100 omits the model updater 110. For illustration, Figure 1 The remote computing device 150 may generate the updated model 106. In such an embodiment, the updated model 106 may be downloaded to the headset 602 for use by the SED engine 108.

[0076] Figure 7 An example of device 100 integrated into a wearable electronic device 702, shown as a "smart watch," is shown including a display 706 (e.g., Figure 1 The device 100 may include or correspond to a display 102 and a sensor(s) 704. The sensor(s) 704 may be capable of detecting user input based on modalities such as video, voice, and gesture, for example. The sensor(s) 704 may also be capable of detecting audio data, which the device 100 uses to detect sound events or to update the base model 104 to generate an updated model 106. For example, the sensor(s) 704 may include or correspond to Figure 1 (multiple) microphones 114.

[0077] Sensor(s) 704 can detect audio data, which device 100 uses to detect sound events or to update base model 104 to generate updated model 106. For example, device 100 Figure 4Output data 408 is provided to display 706 indicating that a recognized sound event was detected in the audio data received from sensor(s) 704. In some implementations, device 100 can perform an action in response to recognizing the sound event, such as activating a camera or one of sensor(s) 704 or providing tactile feedback to a user.

[0078] exist Figure 7 In the example shown, the device 100 includes a base model 104, an update model 106, an SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the wearable electronic device 702, the device 100 omits the model updater 110. For illustration, Figure 1 The remote computing device 150 may generate the update model 106. In such an embodiment, the update model 106 may be downloaded to the wearable electronic device 702 for use by the SED engine 108.

[0079] Figure 8 is an illustrative example of a voice controlled speaker system 800. The voice controlled speaker system 800 may have a wireless network connection and be configured to perform auxiliary operations. Figure 8 In the embodiment, the device 100 is included in a voice-controlled speaker system 800. The voice-controlled speaker system 800 also includes a speaker 802 and (multiple) sensors 804. The (multiple) sensors 804 may include Figure 1 One or more microphones 114 are provided to receive voice input or other audio input.

[0080] During operation, the voice-controlled speaker system 800 can perform auxiliary operations in response to receiving verbal commands. The auxiliary operations can include adjusting the temperature, playing music, turning on lights, etc. The sensor(s) 804 can detect audio data, which the device 100 uses to detect sound events or generate updated models 106. In addition, the voice-controlled speaker system 800 can perform some operations based on the sound events recognized by the device 100. For example, if the device 100 recognizes the sound of a door closing, the voice-controlled speaker system 800 can turn on one or more lights.

[0081] exist Figure 8 In the example shown, the device 100 includes a base model 104, an update model 106, an SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the voice-controlled speaker system 800, the device 100 omits the model updater 110. For illustration, Figure 1 The remote computing device 150 can generate the updated model 106. In such an embodiment, the updated model 106 can be downloaded to the voice-controlled speaker system 800 for use by the SED engine 108.

[0082] Fig. 9 Shows the combination Figure 1 , Figure 3 or Figure 4 The updated sound event detection model of various aspects of the camera 900. Fig. 9 , the device 100 is incorporated into or coupled to a camera 900. The camera 900 includes an image sensor 902 and one or more other sensors 904, such as Figure 1 (multiple) microphones 114. In addition, the camera 900 includes a device 100 configured to identify sound events based on audio data from the sensor (s) 904. For example, the camera 900 can cause the image sensor 902 to capture an image in response to the device 100 detecting a particular sound event in the audio data from the sensor (s) 904.

[0083] exist Fig. 9 In the example shown, the device 100 includes a base model 104, an update model 106, a SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the camera 900, the device 100 omits the model updater 110. For illustration, Figure 1 The remote computing device 150 may generate the update model 106. In such an embodiment, the update model 106 may be downloaded to the camera 900 for use by the SED engine 108.

[0084] Fig.10 Shows the combination Figure 1 , Figure 3 or Figure 4 Various aspects of the updated sound event detection model of the mobile device 1000. Fig.10 In the mobile device 1000, the mobile device 1000 includes or is coupled to Figure 1 The mobile device 1000 includes, by way of illustrative, non-limiting example, a phone or a tablet. The mobile device 1000 includes a display screen 1002 and one or more sensors 1004, such as Figure 1 (multiple) microphones 114.

[0085] During operation, the mobile device 1000 can perform a particular action in response to the device 100 detecting a particular sound event. For example, the action can include sending a command to another device, such as a thermostat, a home automation system, another mobile device, etc. The sensor(s) 1004 can detect audio data, which the device 100 uses to detect a sound event or generate an updated model 106.

[0086] exist Fig.10In the example shown, the device 100 includes a base model 104, an update model 106, an SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the mobile device 1000, the device 100 omits the model updater 110. For illustration, Figure 1 The remote computing device 150 can generate the update model 106. In such an embodiment, the update model 106 can be downloaded to the mobile device 1000 for use by the SED engine 108.

[0087] Fig.11 Shows the combination Figure 1 , Figure 3 or Figure 4 Aspects of the updated sound event detection model of the aerial device 1100. Fig.11 In the invention, the aviation device 1100 includes or is coupled to Figure 1 The aerial device 1100 is a manned, unmanned, or remotely piloted aerial device (e.g., a package delivery drone). The aerial device 1100 includes a control system 1102 and one or more sensors 1104, such as Figure 1 (multiple) microphones 114. The control system 1102 controls various operations of the aerial device 1100, such as cargo release, sensor activation, takeoff, navigation, landing, or a combination thereof. For example, the control system 1102 can control the flight of the aerial device 1100 between specified points and the deployment of cargo at specific locations. In a particular aspect, the control system 1102 performs one or more actions in response to the device 100 detecting a specific sound event. For illustration, the control system 1102 can initiate a safe landing protocol in response to the device 100 detecting an aircraft engine.

[0088] exist Fig.11 In the example shown, the device 100 includes a base model 104, an update model 106, an SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the aviation device 1100, the device 100 omits the model updater 110. For illustration, Figure 1 The remote computing device 150 may generate the updated model 106. In such an embodiment, the updated model 106 may be downloaded to the avionics 1100 for use by the SED engine 108.

[0089] Fig.12 Shows the combination Figure 1 , Figure 3 or Figure 4 Various aspects of the updated sound event detection model of the headset 1200. Fig.12 In the embodiment, the headset 1200 includes or is coupled to Figure 1The headset 1200 includes a microphone 1204 (e.g., Figure 1 The headset 1200 may include one of the microphone(s) 114 that is positioned to primarily capture the user's voice. The headset 1200 may also include one or more additional microphones that are positioned to primarily capture ambient sound (e.g., for noise cancellation operations). In one particular aspect, the headset 1200 performs one or more actions in response to the device 100 detecting a particular sound event. For illustration, the headset 1200 may activate a noise cancellation feature in response to the device 100 detecting a gunshot.

[0090] exist Fig.12 In the example shown, the device 100 includes a base model 104, an update model 106, an SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the headset 1200, the device 100 omits the model updater 110. For illustration, Figure 1 The remote computing device 150 may generate the updated model 106. In such an embodiment, the updated model 106 may be downloaded to the headset 1200 for use by the SED engine 108.

[0091] Fig.13 Shows the combination Figure 1 , Figure 3 or Figure 4 Various aspects of the updated sound event detection model of the home appliance 1300. Fig.13 In the embodiment, the home appliance 1300 is a lamp; however, in other embodiments, the home appliance 1300 includes another IoT home appliance, such as a refrigerator, a coffee machine, an oven, another home appliance, etc. The home appliance 1300 includes or is coupled to Figure 1 The household appliance 1300 includes one or more sensors 1304, such as Figure 1 (multiple) microphones 114. In one particular aspect, the home appliance 1300 performs one or more actions in response to the device 100 detecting a particular sound event. To illustrate, the home appliance 1300 can activate a light in response to the device 100 detecting that a door is closed.

[0092] exist Fig.13 In the example shown, the device 100 includes a base model 104, an update model 106, an SED engine 108, and a model updater 110. However, in other embodiments, when installed or used in the home appliance 1300, the device 100 omits the model updater 110. For illustration, Figure 1The remote computing device 150 can generate the update model 106. In such an embodiment, the update model 106 can be downloaded to the home appliance 1300 for use by the SED engine 108.

[0093] Fig.14 It is shown that the use Figure 1 , Figure 3 or Figure 4 1400 is a flowchart of an example of a method 1400 for generating sound event detection data using an updated sound event detection model (e.g., update model 106). Method 1400 may be initiated, controlled, or performed by device 100. For example, Figure 1 The processor(s) 120 or 132 may execute instructions 124 from the memory 130 to cause the SED engine 108 to use the updated model 106 to generate output data 408 based on audio data samples corresponding to the captured audio.

[0094] In block 1402, the method 1400 includes receiving audio data samples. For example, the microphone(s) 114 of the device 100 may receive audio data from an acoustic environment proximate to the microphone(s) 114, and the codec 142 or the SED engine 108 may generate audio data samples based on the audio data.

[0095] At block 1404, method 1400 includes providing the audio data sample to a first neural network that is trained to generate a first output corresponding to a first class count of a first set of sound classes. Figure 4 As shown, an audio data sample 406 is provided (as one or more inputs 450) to the base model 104. In this example, the base model 104 includes N nodes in the output layer 234, where N is an integer corresponding to the count of a set of sound categories that the base model 104 is configured to detect. The output layer 234 generates a first output 452, which includes one data element (e.g., N data elements) for each node of the output layer 234.

[0096] At block 1406, method 1400 includes providing the audio data sample to a second neural network to generate a second output corresponding to second class counts for a second set of sound classes, wherein the second class counts are greater than the first class counts. Figure 4 As shown, the audio data sample 406 is provided to the incremental model 302. In this example, the output layer 322 of the incremental model 302 includes N+K nodes, where K is an integer greater than or equal to 1 and corresponds to the count of sound categories that the update model 106 can detect that are not detected by the base model 104. Therefore, N+K is greater than N. The output layer 322 generates an output including one data element for each node of the output layer 322 (e.g., N+K data elements).

[0097] At block 1408, method 1400 includes providing the first output to a neural adapter to generate a third output corresponding to second category counts of the second set of sound categories. Figure 4 The base model 104 provides an output to the neural adapter 310. The neural adapter 310 generates an output having the same number of data elements as the output of the incremental model 302 (eg, N+K data elements).

[0098] In block 1410, method 1400 includes providing the second output and the third output to a merging adapter to generate sound recognition data based on the audio data sample. Figure 4 , the neural adapter 310 generates a third output 456 and the incremental model 302 generates a second output 454. In this example, the third output 456 and the second output 454 are each provided to the merge adapter 308. The aggregation layer 316 of the merge adapter 308 combines the outputs 454 and 456 to generate a merged output 458, and the output layer 320 of the merge adapter 308 generates output data 408 based on the merged output 458. The output data 408 includes an indication of the recognized sound event (e.g., sound event identification data 460).

[0099] Fig.15 It shows that the generated Figure 1 , Figure 3 or Figure 4 1500 is a flowchart of an example of a method 1500 for updating a sound event detection model. The method 1500 may be initiated, controlled, or executed by the device 100. For example, Figure 1 The processor(s) 120 or 132 of the processor 150 may execute the instructions 124 from the memory 130 to cause the model updater 110 to generate the updated model 106 based on the audio data samples 126 and 128. Alternatively, in a particular aspect, the method 1500 may be performed by Figure 1 The model updater 110 may be initiated, controlled, or executed by a remote computing device 150. For illustration, the model updater 110 may be executed at the remote computing device 150 using audio data samples 126 from the base training data 152 and audio data samples 128 sent from the device 100 to the remote computing device 150.

[0100] In block 1502, method 1500 includes generating a second neural network based on a first neural network of a first sound event classifier. The first neural network includes an input layer, a hidden layer, and a first output layer, and the second neural network includes a copy of the input layer of the first neural network, a copy of the hidden layer of the first neural network, and a second output layer coupled to the copy of the hidden layer. The first output layer includes a first output node count and the second output layer includes a second output node count, wherein the second output node count is greater than the first output node count. For example, model updater 110 generates incremental model 302 by repeating (e.g., copying) base model 104 including input layer 204, hidden layer 206, and output layer 234 and replacing or modifying output layer 234 with output layer 322. In this example, base model 104 is a first neural network that is trained to generate sound recognition data in response to an input of an audio data sample representing one or more of a first set of sound categories, and output layer 234 of base model 104 includes a node count corresponding to the number of categories of the first set of sound categories (e.g., N nodes). In addition, the incremental model 302 is a second neural network that is trained to generate sound recognition data in response to input of audio data samples representing one or more of the second set of sound categories, and the output layer 322 of the incremental model 302 includes a second node count corresponding to a second number of categories of the second set of sound categories (e.g., N+K). The second set of sound categories includes the first set of sound categories and one or more additional sound categories.

[0101] At block 1504, method 1500 includes linking the first neural network and the second neural network to one or more adapter networks. For example, Figure 1 The model updater 110 generates (multiple) adapter networks 314 and links the outputs of the base model 104 (e.g., the first neural network) and the incremental model 302 (e.g., the second neural network) to the (multiple) adapter network 314.

[0102] At block 1506, method 1500 includes providing the labeled training data as input to the first neural network and the second neural network. Figure 3 , the model updater 110 provides the labeled training data 304 as one or more inputs 350 to the base model 104 and the incremental model 302 .

[0103] In block 1508, method 1500 includes modifying outputs of the first neural network and the second neural network via one or more adapter networks. For example, in response to labeled training data 304, Figure 3The base model 104 and the incremental model 302 each provide an output to the adapter network(s) 314. For illustration, the base model 104 provides the first output(s) 352 to the neural adapter 310, and the neural adapter generates the third output(s) 356 based on the first output(s) 352. Additionally, the incremental model 302 generates the second output(s) 354. The third output(s) 356 and the second output(s) 354 are provided to the merge adapter 308, and the merge adapter 308 generates the output data 318.

[0104] In block 1510, method 1500 trains a second sound event classifier by updating the link weights of the second neural network and one or more adapter networks based on the output of the adapter network and the labels of the labeled training data. For example, model updater 110 trains update model 106 by determining updated link weights 362 (e.g., using gradient descent or another optimization search process) and providing updated link weights 362 to one or more of incremental model 302, neural adapter 310, and merge adapter 308. In this example, base link weights 238 of base model 104 are not changed.

[0105] In conjunction with the described embodiments, an apparatus includes a component for storing a sound event classifier that is trained to generate first sound recognition data in response to an input of audio data representing one or more of a first set of sound categories. For example, the component for storing includes a remote computing device 150, a device 100, a memory 130, a processor 120, (multiple) processors 132, one or more other circuits or components configured to store a trained sound event classifier (e.g., a neural network), or any combination thereof.

[0106] The apparatus also includes a component for generating an updated sound event classifier, the updated sound event classifier being trained to generate second sound recognition data in response to an input of an audio data sample representing one or more of a second set of sound categories, wherein the second set of sound categories includes the first set of sound categories and one or more additional sound categories. For example, the component for generating an updated sound event classifier includes a remote computing device 150, a device 100, instructions 124, a processor 120, (multiple) processors 132, a model updater 110, one or more other circuits or components configured to generate an updated sound event classifier, the updated sound event classifier being trained to generate second sound recognition data in response to an input of an audio data sample representing one or more of a second set of sound categories, wherein the second set of sound categories includes the first set of sound categories and one or more additional sound categories, or any combination thereof.

[0107] In conjunction with the described embodiments, an apparatus includes means for generating a first output corresponding to a first category count of a first set of sound categories. For example, the means for generating the first output includes the processor 120, the processor(s) 132, the base model 104, the update model 106, the SED engine 108, one or more other circuits or components configured to generate the first output corresponding to the first category count of the first set of sound categories, or any combination thereof.

[0108] The apparatus also includes means for generating a second output corresponding to a second category count for the second set of sound categories, the second category count being greater than the first category count. For example, the means for generating the second output includes the processor 120, the processor(s) 132, the incremental model 302, the update model 106, the SED engine 108, one or more other circuits or components configured to generate a second output corresponding to the second category count for the second set of sound categories, or any combination thereof.

[0109] The apparatus also includes means for generating a third output based on the first output, the third output corresponding to a second category count of the second set of sound categories. For example, the means for generating the third output includes processor 120, (multiple) processors 132, update model 106, (multiple) adapter network 314, neural adapter 310, SED engine 108, one or more other circuits or components configured to generate the third output based on the first output, or any combination thereof.

[0110] The apparatus also includes a component for generating sound event identification data based on the third output and the second output. For example, the component for generating sound identification data based on the third output and the second output includes processor 120, (multiple) processors 132, update model 106, (multiple) adapter network 314, merge adapter 308, SED engine 108, one or more other circuits or components configured to generate sound identification data based on the third output and the second output, or any combination thereof.

[0111] The technician will further understand that the various illustrative logic blocks, configurations, modules, circuits and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits and steps have been described above generally according to their functions. Whether such functions are implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the entire system. The technician can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0112] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disk read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. Alternatively, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.

[0113] Certain aspects of the present disclosure are described below in a first set of interrelated clauses:

[0114] According to clause 1, a device includes a processor configured to: receive an audio data sample; provide the audio data sample to a first neural network that is trained to generate a first output corresponding to a first category count of a first set of sound categories; provide the audio data sample to a second neural network to generate a second output corresponding to a second category count of a second set of sound categories, the second category count being greater than the first category count; provide the first output to a neural adapter to generate a third output corresponding to the second category count of the second set of sound categories; and provide the second output and the third output to a merging adapter to generate sound event recognition data based on the audio data sample.

[0115] Clause 2 includes an apparatus as described in clause 1, wherein the first neural network has a base topology and a first output layer and the second neural network has a base topology and a second output layer, and wherein the first output layer includes a first node count, the second output layer includes a second node count, and the second node count is greater than the first node count.

[0116] Clause 3 includes the apparatus of clause 2, wherein the neural adapter has an input layer comprising a first node count and an output layer comprising a second node count.

[0117] Clause 4 includes an apparatus as defined in any one of clauses 1 to 3, wherein the merge adapter is configured to merge the second output and the third output element-by-element to form the merged output.

[0118] Clause 5 includes the apparatus of clause 4, wherein the merge adapter is configured to generate output data including the sound event identification data based on the merged output.

[0119] Clause 6 includes an apparatus as described in any of clauses 1 to 5, wherein the audio data sample includes features extracted from the audio data.

[0120] Clause 7 includes an apparatus as defined in any one of clauses 1 to 6, wherein the audio data samples include mel-spectrogram features extracted from the audio data.

[0121] Clause 8 includes the apparatus of any one of clauses 1 to 7, and further includes one or more microphones coupled to the processor and configured to capture audio data to generate audio data samples.

[0122] Clause 9 includes the device of clause 8, wherein the processor and the one or more microphones are integrated within the mobile computing device, and the audio data represents an acoustic environment of the mobile computing device.

[0123] Clause 10 includes the apparatus of clause 8, wherein the processor and the one or more microphones are integrated within a vehicle.

[0124] Clause 11 includes the device of clause 8, wherein the processor and the one or more microphones are integrated within a wearable device, and the audio data represents an acoustic environment of the wearable device.

[0125] Clause 12 includes the device of clause 8, wherein the processor and the one or more microphones are integrated into a headset.

[0126] Clause 13 includes the apparatus of clause 8, wherein the processor is included in an integrated circuit.

[0127] Certain aspects of the present disclosure are described below in a second set of interrelated clauses:

[0128] According to clause 14, a method includes: receiving an audio data sample; providing the audio data sample by a processor to a first neural network, the first neural network being trained to generate a first output corresponding to a first category count of a first set of sound categories; providing the audio data sample by the processor to a second neural network to generate a second output corresponding to a second category count of a second set of sound categories, the second category count being greater than the first category count; providing the first output by the processor to a neural adapter to generate a third output corresponding to the second category count of the second set of sound categories; and providing the second output and the third output by the processor to a merging adapter to generate sound event recognition data based on the audio data sample.

[0129] Clause 15 includes the method of clause 14, wherein the first neural network has a base topology and a first output layer and the second neural network has a base topology and a second output layer, and wherein the first output layer includes a first node count, the second output layer includes a second node count, and the second node count is greater than the first node count.

[0130] Clause 16 includes the method of clause 15, wherein the neural adapter has an input layer comprising a first node count and an output layer comprising a second node count.

[0131] Clause 17 includes the method of any of clauses 14 to 16, wherein the merge adapter merges the second output and the third output element-by-element to form the merged output.

[0132] Clause 18 includes the method of clause 17, wherein the merge adapter generates output data including sound event identification data based on the merged output.

[0133] Clause 19 includes the method of any one of clauses 14 to 18, and further includes generating the audio data samples by extracting features from the audio data representative of the acoustic environment.

[0134] Clause 20 includes the method of any of clauses 14 to 19, and further includes capturing audio data at one or more microphones coupled to the processor, wherein the audio data samples are generated based on the captured audio data.

[0135] Clause 21 includes the method of any one of clauses 14 to 20, and further includes performing an action in response to the sound event recognition data.

[0136] Certain aspects of the present disclosure are described below in a third set of interrelated clauses:

[0137] According to clause 22, a non-transitory computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to: provide an audio data sample to a first neural network that is trained to generate a first output corresponding to a first class count of a first set of sound classes; provide the audio data sample to a second neural network to generate a second output corresponding to a second class count of a second set of sound classes, the second class count being greater than the first class count; provide the first output to a neural adapter to generate a third output corresponding to the second class count of the second set of sound classes; and provide the second output and the third output to a merging adapter to generate sound event recognition data based on the audio data sample.

[0138] Clause 23 includes the non-transitory computer-readable storage medium of clause 22, wherein the first neural network has a base topology and a first output layer and the second neural network has a base topology and a second output layer, and wherein the first output layer includes a first node count, the second output layer includes a second node count, and the second node count is greater than the first node count.

[0139] Clause 24 includes the non-transitory computer-readable storage medium of clause 22 or clause 23, wherein the instructions, when executed by the processor, further cause the processor to perform an action in response to the sound event recognition data.

[0140] Clause 25 includes the non-transitory computer-readable storage medium of any of clauses 22 to 24, wherein the merge adapter generates the sound event identification data based on a merged output that is an element-by-element merge of the third output and the second output.

[0141] Certain aspects of the present disclosure are described below in a fourth set of interrelated clauses:

[0142] According to clause 26, a device includes: a component for generating a first output based on an audio data sample, the first output having a first data element count corresponding to a first category count of a first group of sound categories; a component for generating a second output based on the audio data sample, the second output corresponding to a second category count of a second group of sound categories, the second category count being greater than the first category count; a component for generating a third output based on the first output, the third output corresponding to the second category count of the second group of sound categories; and a component for generating sound event identification data based on the third output and the second output.

[0143] Clause 27 includes an apparatus according to clause 26, wherein the component for generating a third output based on the first output includes: an input layer including a first node count and an output layer including a second node count, and wherein the first node count corresponds to a first category count and the second node count corresponds to a second category count.

[0144] Clause 28 includes the apparatus of clause 26 or clause 27, wherein the means for generating the sound event identification data based on the third output and the second output is configured to merge the second output and the third output element by element to form a merged output.

[0145] Clause 29 includes an apparatus according to any of clauses 26 to 28, wherein the means for generating sound event identification data based on the third output and the second output is configured to generate output data including the sound event identification data based on a combined output formed by the third output and the second output.

[0146] Clause 30 includes the apparatus of any of clauses 26 to 29, further comprising means for capturing audio data, wherein the audio data sample includes features extracted from the audio data.

[0147] Certain aspects of the present disclosure are described below in a fifth set of interrelated clauses:

[0148] According to clause 31, a device includes a memory storing a sound event classifier, the sound event classifier being trained to generate first sound recognition data in response to an input of audio data representing one or more of a first set of sound categories. The device also includes a processor coupled to the memory and configured to generate an updated sound event classifier, the updated sound event classifier being trained to generate second sound recognition data in response to an input of audio data representing one or more of a second set of sound categories, the second set of sound categories including the first set of sound categories and one or more additional sound categories. The updated sound event classifier includes a sound event classifier, a second sound event classifier, a neural adapter, and a merge adapter. The neural adapter includes one or more adapter layers configured to receive an input having a data element count corresponding to an output layer of the sound event classifier, and configured to generate an output having a second data element count corresponding to an output layer of the second sound event classifier. The merge adapter includes one or more aggregation layers configured to merge outputs from the neural adapter and outputs from the second sound event classifier, and includes an output layer for generating second sound recognition data.

[0149] Clause 32 includes the apparatus of clause 31, and further includes one or more microphones coupled to the processor and configured to receive audio data corresponding to the additional sound category.

[0150] Clause 33 includes a device as described in clause 31 or clause 32, and further includes one or more input devices coupled to the processor and configured to receive tag data associated with additional sound categories.

[0151] Clause 34 includes an apparatus as described in any of clauses 31 to 33, wherein the memory stores instructions corresponding to the model updater, and wherein the model updater is executable by the processor to generate an updated sound event classifier based on the sound event classifier, the first set of sound categories, and the additional sound categories.

[0152] Clause 35 includes the device of any of clauses 31 to 34, wherein the processor and the memory are integrated within the mobile computing device.

[0153] Clause 36 includes an apparatus as described in clauses 31 to 34, wherein the processor and the memory are integrated within a carrier.

[0154] Clause 37 includes a device as described in clauses 31 to 34, wherein the processor and the memory are integrated within the wearable device.

[0155] Clause 38 includes a device as described in clauses 31 to 34, wherein the processor and the memory are integrated in an augmented reality headset, a mixed reality headset, or a virtual reality headset.

[0156] Clause 39 includes an apparatus as described in clauses 31 to 38, wherein the processor is included in an integrated circuit.

[0157] Clause 40 includes the apparatus of clauses 31 to 39 and further includes one or more output devices coupled to the processor and configured to generate a sound classification output based on the second sound recognition data.

[0158] Certain aspects of the present disclosure are described below in a sixth set of interrelated clauses:

[0159] According to clause 41, a device includes a component for storing a sound event classifier, the sound event classifier being trained to generate first sound recognition data in response to an input of audio data representing one or more of a first set of sound categories, and includes a component for generating an updated sound event classifier, the updated sound event classifier being trained to generate second sound recognition data in response to an input of audio data representing one or more of a second set of sound categories; the second set of sound categories includes the first set of sound categories and one or more additional sound categories. The updated sound event classifier includes a sound event classifier, a second sound event classifier, a neural adapter, and a merge adapter. The neural adapter includes one or more adapter layers configured to receive an input having a data element count corresponding to an output layer of the sound event classifier, and configured to generate an output having a second data element count corresponding to an output layer of the second sound event classifier. The merge adapter includes one or more aggregation layers configured to merge outputs from the neural adapter and outputs from the second sound event classifier, and includes an output layer for generating second sound recognition data.

[0160] Clause 42 comprises the apparatus of clause 41 and further comprises means for receiving audio data corresponding to additional sound categories.

[0161] Clause 43 includes an apparatus as described in clause 41 or clause 42, and further includes means for receiving tag data associated with additional sound categories.

[0162] Clause 44 includes the apparatus of any of clauses 41 to 43, wherein the means for storing and the means for generating are integrated within a mobile computing device.

[0163] Clause 45 includes the apparatus of any of clauses 41 to 43, wherein the means for storing and the means for generating are integrated within a carrier.

[0164] Clause 46 includes the device of any of clauses 41 to 43, wherein the means for storing and the means for generating are integrated within a wearable device.

[0165] Clause 47 includes the apparatus of any of clauses 41 to 43, wherein the means for storing and the means for generating are integrated into an augmented reality or virtual reality headset.

[0166] Certain aspects of the present disclosure are described below in a seventh set of interrelated clauses:

[0167] According to clause 48, a method includes generating a second neural network based on a first neural network of a first sound event classifier, wherein the first neural network includes an input layer, a hidden layer, and a first output layer, and the second neural network includes a copy of the input layer of the first neural network, a copy of the hidden layer of the first neural network, and a second output layer coupled to the copy of the hidden layer. The first output layer includes a first output node count and the second output layer includes a second output node count, wherein the second output node count is greater than the first output node count. The method also includes: linking the first neural network and the second neural network to one or more adapter networks; providing labeled training data as input to the first neural network and the second neural network; modifying the outputs of the first neural network and the second neural network via the one or more adapter networks; and training the second sound event classifier by updating the link weights of the second neural network and the one or more adapter networks based on the outputs of the adapter networks and the labels of the labeled training data.

[0168] Clause 49 includes a method according to clause 48, wherein the first output node count corresponds to a first set of sound categories that the first sound event classifier is trained to detect, and the second output node count corresponds to a second set of sound categories that the second sound event classifier is trained to detect, and wherein the second set of sound categories includes the first set of sound categories and one or more additional sound categories.

[0169] Clause 50 includes the method according to clause 49 and also includes capturing audio data representing sound events of one or more additional sound categories by a microphone of the mobile device, wherein a processor of the mobile device trains a second sound event classifier based in part on audio data samples representing audio data captured by the microphone of the mobile device.

[0170] Clause 51 includes a method according to clause 49 or clause 50, wherein the labeled training data includes one or more audio data samples representing each category in the first set of sound categories and one or more audio data samples representing each category in one or more additional sound categories.

[0171] Clause 52 includes a method according to any of clauses 48 to 51, wherein the adapter network includes a neural adapter, the neural adapter including one or more adapter layers, the one or more adapter layers configured to receive an input having a first data element count and configured to generate an output having a second data element count, the first data element count corresponding to a first output node count and the second data element count corresponding to a second output node count.

[0172] Item 53 includes a method according to item 52, wherein the adapter network includes a merging adapter, the merging adapter comprising one or more aggregation layers and an output layer, wherein the one or more aggregation layers are configured to merge outputs from the neural adapter and outputs from the second neural network, and wherein the output layer is configured to generate output data identifying the sound event.

[0173] Clause 54 includes a method according to any one of clauses 48 to 53, and further includes: after training the second sound event classifier, providing the audio data sample as input to the second sound event classifier, and generating output data identifying the sound event detected by the second sound event classifier in the audio data sample.

[0174] Clause 55 includes a method according to clause 54, wherein generating output data identifying a sound event comprises providing an audio data sample to a first neural network to generate a first output; providing the audio data sample to a second neural network to generate a second output; providing the first output to a neural adapter of one or more adapter networks to generate a third output; and providing the second output and the third output to a merge adapter of one or more adapter networks to generate the output data.

[0175] Clause 56 includes the method of any of clauses 48 to 55, wherein link weights of the first neural network are not updated during training of the second sound event classifier.

[0176] Clause 57 includes the method of any of clauses 48 to 56, wherein the second sound event classifier comprises a first neural network, a second neural network, and one or more adapter networks.

[0177] Certain aspects of the present disclosure are described below in an eighth set of interrelated clauses:

[0178] According to clause 58, a non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to: generate a second neural network based on a first neural network of a first sound event classifier, wherein the first neural network includes an input layer, a hidden layer, and a first output layer, and the second neural network includes a copy of the input layer of the first neural network, a copy of the hidden layer of the first neural network, and a second output layer coupled to the copy of the hidden layer. The first output layer includes a first output node count and the second output layer includes a second output node count, wherein the second output node count is greater than the first output node count. The instructions also cause the processor to: link the first neural network and the second neural network to one or more adapter networks; provide labeled training data as input to the first neural network and the second neural network; modify outputs of the first neural network and the second neural network via the one or more adapter networks; and train the second sound event classifier by updating link weights of the second neural network and the one or more adapter networks based on the outputs of the adapter networks and the labels of the labeled training data.

[0179] Clause 59 includes a non-transitory computer-readable storage medium as described in clause 58, wherein the first sound event classifier is trained to detect a first set of sound categories and the second sound event classifier is trained to detect a second set of sound categories, the second set of sound categories including the first set of sound categories and one or more additional sound categories.

[0180] Clause 60 includes a non-transitory computer-readable storage medium according to clause 58 or clause 59, wherein the instructions, when executed by the processor, further cause the processor to: after training the second sound event classifier, provide an audio data sample representing the captured audio data as input to the second sound event classifier, and generate output data identifying a sound event detected by the second sound event classifier in the audio data sample.

[0181] The above description of the various aspects disclosed is provided to enable those skilled in the art to implement or use the various aspects disclosed. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the various aspects shown herein, but should be given as wide a scope as possible consistent with the principles and novel features defined by the appended claims.

Claims

1. A device for sound event recognition, comprising: A processor, wherein the processor is configured as: receiving audio data samples; providing the audio data samples to a first neural network trained to generate a first output corresponding to first class counts of a first set of sound classes, the first neural network having a base topology and a first output layer; providing the audio data samples to a second neural network to generate a second output corresponding to a second class count for a second set of sound classes, the second class count being greater than the first class count, and the second neural network having the base topology and a second output layer; providing the first output to a neural adapter to generate a third output corresponding to the second category counts of the second set of sound categories; as well as The second output and the third output are provided to a merging adapter to generate sound event identification data based on the audio data samples.

2. The device according to claim 1, wherein The first output layer includes a first node count, the second output layer includes a second node count, and the second node count is greater than the first node count.

3. The device according to claim 2, wherein: The neural adapter has an input layer including the first node count and an output layer including the second node count.

4. The device according to claim 1, wherein: The merge adapter is configured to merge the second output and the third output element-by-element to form a merged output.

5. The device according to claim 4, wherein: The merge adapter is configured to generate output data including the sound event identification data based on the merged output.

6. The device according to claim 1, wherein: The audio data samples include features extracted from the audio data.

7. The device according to claim 1, wherein: The audio data sample includes a mel-spectrogram feature extracted from the audio data.

8. The device of claim 1, further comprising one or more microphones coupled to the processor and configured to capture audio data to generate the audio data samples.

9. The device according to claim 8, wherein: The processor and the one or more microphones are integrated within a mobile computing device, and the audio data represents an acoustic environment of the mobile computing device.

10. The device according to claim 8, wherein: The processor and the one or more microphones are integrated within the vehicle.

11. The device according to claim 8, wherein: The processor and the one or more microphones are integrated within a wearable device, and the audio data represents an acoustic environment of the wearable device.

12. The device according to claim 8, wherein: The processor and the one or more microphones are integrated into a headset, and the audio data represents an acoustic environment of the headset.

13. The apparatus according to claim 1, wherein: The processor is included in an integrated circuit.

14. A method for sound event recognition, comprising: receiving audio data samples; providing, by a processor, the audio data samples to a first neural network trained to generate a first output corresponding to first class counts for a first set of sound classes, the first neural network having a base topology and a first output layer; providing, by the processor, the audio data samples to a second neural network to generate a second output corresponding to a second class count for a second set of sound classes, the second class count being greater than the first class count, and the second neural network having the base topology and a second output layer; providing, by the processor, the first output to a neural adapter to generate a third output corresponding to the second category counts of the second set of sound categories; as well as The second output and the third output are provided by the processor to a merging adapter to generate sound event identification data based on the audio data samples.

15. The method according to claim 14, wherein: The first output layer includes a first node count, the second output layer includes a second node count, and the second node count is greater than the first node count.

16. The method according to claim 15, wherein: The neural adapter has an input layer including the first node count and an output layer including the second node count.

17. The method according to claim 14, wherein: The merge adapter merges the second output and the third output element-by-element to form a merged output.

18. The method according to claim 17, wherein: The merge adapter generates output data including the sound event identification data based on the merged output.

19. The method of claim 14, further comprising generating audio data samples, including extracting features from the audio data representative of the acoustic environment.

20. The method of claim 14, further comprising capturing audio data at one or more microphones coupled to the processor, wherein The audio data samples are generated based on captured audio data.

21. The method of claim 14, further comprising performing an action in response to the sound event identification data.

22. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to: providing the audio data sample to a first neural network trained to generate a first output corresponding to first class counts of a first set of sound classes, the first neural network having a base topology and a first output layer; providing the audio data samples to a second neural network to generate a second output corresponding to a second class count for a second set of sound classes, the second class count being greater than the first class count, and the second neural network having the base topology and a second output layer; providing the first output to a neural adapter to generate a third output corresponding to the second category counts of the second set of sound categories; as well as The second output and the third output are provided to a merging adapter to generate sound event identification data based on the audio data samples.

23. The non-transitory computer-readable storage medium of claim 22, wherein: The first output layer includes a first node count, the second output layer includes a second node count, and the second node count is greater than the first node count.

24. The non-transitory computer-readable storage medium of claim 22, wherein: The instructions, when executed by the processor, further cause the processor to perform an action in response to the sound event identification data.

25. The non-transitory computer-readable storage medium of claim 22, wherein: The merge adapter generates the sound event identification data based on a merged output that is an element-by-element merge of the third output and the second output.

26. A device for sound event recognition, comprising: means for generating a first output based on the audio data samples, the first output corresponding to first class counts for a first set of sound classes, the means for generating a first output based on the audio data samples having a base topology and a first output layer; means for generating a second output based on the audio data samples, the second output corresponding to a second class count for a second set of sound classes, the second class count being greater than the first class count, and the means for generating a second output based on the audio data samples having the base topology and a second output layer; means for generating a third output based on the first output, the third output corresponding to the second category counts for the second set of sound categories; as well as Means for generating sound event identification data based on the third output and the second output.

27. The apparatus of claim 26, wherein: The component for generating the third output based on the first output includes: an input layer including a first node count and an output layer including a second node count, and the first node count corresponds to the first category count of the first group of sound categories, and the second node count corresponds to the second category count of the second group of sound categories.

28. The apparatus of claim 26, wherein: The means for generating the sound event identification data based on the third output and the second output is configured to merge the second output and the third output element by element to form a merged output.

29. The apparatus of claim 26, wherein: The means for generating the sound event identification data based on the third output and the second output is configured to generate output data including the sound event identification data based on a combined output formed by the third output and the second output.

30. The apparatus of claim 26, further comprising means for capturing audio data, wherein The audio data samples include features extracted from the audio data.

Images