SYSTEM AND METHOD FOR IMPROVING THE PRECISION AND RECALL OF PROTOTYPE NETWORKS FOR SOUND EVENT DETECTION

Abstract

A method for training a prototypical network for sound event detection involves receiving samples of an audio signal, including positive samples corresponding to sound events and negative samples not corresponding to sound events; determining, based on the positive samples, respective positive prototypes of a plurality of classes of sound events; determining, based on the negative samples, respective negative prototypes for respective groups of the negative samples, each of which corresponds to a combination of a plurality of the negative samples; and generating, based on comparisons between a first sample and the respective positive prototypes and each of the negative prototypes, an output signal indicating whether the first sample belongs to one of the plurality of classes of sound events.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to sound event detection (SED), and more particularly to systems and methods for operating devices implementing SED. BACKGROUND OF THE INVENTION

[0002] Sound event detection (SED) refers to the task of detecting and distinguishing individual acoustic events (e.g., bird chirping, gunshot sounds, smoke alarm calls, etc.) within a continuous audio stream. SED involves detecting the occurrence of a given event and identifying its onset and offset timestamps.

[0003] SED presents a challenge compared to other mature areas in acoustic signal processing, such as automatic speech recognition (ASR). ASR deals with spoken language, where sound events are constrained by human vocal tract morphology, whereas SED deals with acoustic events that are more diverse (e.g., both periodic and sporadic in nature). The scope of SED tasks ranges from domestic and large-scale security monitoring to bioacoustic event detection in the wild. SUMMARY

[0004] A method for training a prototypical network for sound event detection includes receiving samples of an audio signal including positive samples corresponding to sound events and negative samples not corresponding to sound events, determining, based on the positive samples, respective positive prototypes of a plurality of classes of sound events, determining, based on the negative samples, respective negative prototypes for respective groups of the negative samples, each of the negative prototypes corresponding to a combination of a plurality of the negative samples, and generating, based on comparisons between the first sample and the respective positive prototypes and each of the negative prototypes, an output signal indicating whether a first sample belongs to one of the plurality of classes of sound events.

[0005] In other features, the method further includes obtaining first embeddings of the positive samples and second embeddings of the negative samples, determining the respective positive prototypes based on the first embeddings, and determining the respective negative prototypes based on the second embeddings. Determining the respective negative prototypes includes determining a negative prototype for each of the respective groups of the negative samples. The method further includes determining, based on the comparisons, at least one probability that the first sample corresponds to the one of the plurality of classes of sound events, and generating the output based on the at least one probability. The method further includes determining at least one probability distribution and generating the output based on the at least one probability distribution.The method further includes determining at least one threshold based on the at least one probability distribution and generating the output based on a comparison between the at least one probability and the at least one threshold. Determining the at least one threshold includes determining the at least one threshold based on a first probability distribution of probabilities that the positive samples belong to respective classes of the plurality of classes of sound events and a second probability distribution that the negative samples belong to respective classes of the plurality of classes of sound events.

[0006] A computing device configured to train a prototypical network for sound event detection includes a processing device configured to execute instructions stored in a memory to receive samples of an audio signal, the samples including positive samples corresponding to sound events and negative samples not corresponding to sound events, determining, based on the positive samples, respective positive prototypes of a plurality of classes of sound events, determining, based on the negative samples, respective negative prototypes for respective groups of the negative samples, each of the negative prototypes corresponding to a combination of a plurality of the negative samples, and generating, based on comparisons between the first sample and the respective positive prototypes and each of the negative prototypes, an output signal indicatingwhether a first sample belongs to one of the multitude of classes of sound events.

[0007] In other features, the processing device is further configured to execute the instructions to obtain first embeddings of the positive samples and second embeddings of the negative samples, determine the respective positive prototypes based on the first embeddings, and determine the respective negative prototypes based on the second embeddings. Determining the respective negative prototypes includes determining a negative prototype for each of the respective groups of the negative samples. The processing device is further configured to execute the instructions to determine, based on the comparisons, at least one probability that the first sample corresponds to the one of the plurality of classes of sound events, and generate the output based on the at least one probability.The processing device is further configured to execute the instructions to determine at least one probability distribution and generate the output based on the at least one probability distribution. The processing device is further configured to execute the instructions to determine at least one threshold based on the at least one probability distribution and generate the output based on a comparison between the at least one probability and the at least one threshold.Determining the at least one threshold includes determining the at least one threshold based on a first probability distribution of probabilities that the positive samples belong to respective classes of the plurality of classes of sound events and a second probability distribution that the negative samples belong to respective classes of the plurality of classes of sound events.

[0008] A computer-controlled machine includes at least one sensor configured to generate an audio signal, a control system configured to receive a first sample of the audio signal and generate an output signal indicating whether the first sample of the audio signal belongs to one of the plurality of classes of sound events based on comparisons between the first sample and respective positive prototypes for each of a plurality of classes of sound events and respective negative prototypes for each of a plurality of groups of negative prototypes, and an actuator configured to control operation of the computer-controlled machine in response to the output of the control system.

[0009] In other features, the computer-controlled machine further includes a memory storing the respective positive prototypes and the respective negative prototypes. The respective positive prototypes correspond to a plurality of positive samples, and each of the respective negative prototypes corresponds to a combination of a plurality of negative samples. Generating the output includes calculating a probability that the first sample belongs to a first class of the plurality of classes or a first group of the plurality of groups. Generating the output includes comparing the probability to at least one threshold and generating the output based on the comparison. The at least one threshold includes a plurality of thresholds corresponding to respective classes of the plurality of classes of sound events. The computer-controlled machine includes an autonomous robot. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 generally illustrates a system for training a neural network according to the principles of the present disclosure. Fig. 2 generally illustrates a computer-implemented method for training and utilizing a neural network in accordance with the principles of the present disclosure. Fig. 3A generally illustrates an audio data labeling system according to the principles of the present disclosure. Fig. 3B generally illustrates a portion of a data acquisition system according to the principles of the present disclosure. Fig. 3C generally illustrates an alternative audio data labeling system according to the principles of the present disclosure. Fig. 4A generally illustrates an example audio stream including a support sentence and a query sentence according to the principles of the present disclosure. Fig. 4B generally illustrates clustering of a positive and negative set of samples according to the principles of the present disclosure. Fig. 4C generally illustrates an example clustering of support sets according to the principles of the present disclosure. Fig. 4D generally illustrates an example of the probability distributions obtained from positive support set samples and negative support set samples according to the principles of the present disclosure. Fig. 4E generally illustrates exemplary probability calculations according to the principles of the present disclosure. Fig. Figure 4F generally illustrates an exemplary method for determining a class of a sample x q according to the principles of the present disclosure. Fig. 5 illustrates a schematic diagram of an interaction between a computer-controlled machine and a control system according to the principles of the present disclosure. Fig. Figure 6 shows a schematic diagram of the control system of Fig. 5 configured to control a vehicle, which may be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot, in accordance with the principles of the present disclosure. Fig. Figure 7 shows a schematic diagram of the control system of Fig. 5 configured to control a manufacturing machine, such as a punch cutter, a cutter, or a pistol drill, of a manufacturing system, such as part of a production line. Fig. Figure 8 shows a schematic diagram of the control system of Fig. 5 configured to control a power tool, such as an electric drill or screwdriver, having an at least partially autonomous mode. Fig. Figure 9 shows a schematic diagram of the control system of Fig. 5 configured to control an automated personal assistant. Fig. Figure 10 shows a schematic diagram of the control system of Fig. 5 configured to control a monitoring system, such as a control access system or a monitoring system. Fig. Figure 11 shows a schematic diagram of the control system of Fig. 5 configured to control an imaging system, for example an MRI device, an X-ray imaging device or an ultrasound device. DETAILED DESCRIPTION

[0010] Embodiments of the present disclosure will now be described. It should be understood, however, that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching one skilled in the art to variously employ the embodiments. As one of ordinary skill in the art will understand, various features illustrated and described with reference to any of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described.The combinations of illustrated features provide representative embodiments for a typical application. However, various combinations and modifications of the features consistent with the teachings of this disclosure may be desirable for particular applications or implementations.

[0011] As used herein, "a," "an," "another," and "the" refer to both singular and plural referents unless the context clearly dictates otherwise. For example, "a processor" programmed to perform various functions refers to one processor programmed to perform each function individually, or more than one processor programmed together to perform each of the various functions.

[0012] Sound event detection (SED) is a challenging task due to the presence of both target and non-target sounds in the environment. Appropriate labels of the target sound are necessary for machine learning models to detect sound events from an audio stream. However, these models often require a large amount of labeled data and application-specific model tuning. Furthermore, collecting labels of the target sound events within large audio collections requires expert evaluation (e.g., to obtain annotations for long audio recordings, to mark the start and end timestamps of each event, etc.), especially when the events originate from a specific domain (e.g., wildlife, machinery, security surveillance), which contributes to the challenges in the annotation process. Therefore, SED can be costly and time-consuming.

[0013] Prototypical networks have gained popularity in recent years by employing a small number of labeled samples per class (i.e., to train deep learning SED models with less data), while demonstrating competitive recall rates in SED. However, these models often fail to distinguish between target and non-target sounds due to the diversity of background noise and the soundscape. Furthermore, these models did not aim to detect rare or previously unseen events during inference time.

[0014] Few-shot learning methods can be successful in cases where previously unseen classes can be inferred using only a few labeled samples. Some computer vision problems benefit greatly from the use of few-shot learning methods (e.g., prototypical networks, matching networks, twin networks, etc.). Some recent examples have framed the SED task with few labeled samples as a few-shot learning task, where models are trained in an episodic manner in an n-way k-shot formulation.At training time, the model learns an embedding representation on N classes in an episodic manner, where each episode uses only K labeled samples from each class to create N prototypes and classify the input samples.

[0015] For example, if a new audio sequence containing events from unseen classes is provided, the first part of the sequence is manually labeled (“support set”), while the remaining part of the sequence is used for inference (“query set”). The model labels the query set using information from the support set, where the support set has at least K labeled samples for each new class. Prototypical networks are the best performing model for few-shot SED tasks, where the model automatically finds the onset and offset timestamps of events belonging to a class unknown during training in a long audio sequence because the first K occurrences of this class are labeled in the support set.The model learns a positive prototype representative of the class of interest, along with a corresponding negative prototype, and performs binary classification at inference time for each of the classes presented in the support set.

[0016] Systems and methods according to the principles of the present disclosure identify and improve various aspects of SED systems. For example, during prototyping, more focus is typically placed on creating positive prototypes using the support set, while negative prototypes are constructed by randomly sampling segments from the entire audio sequence. While this method reduces the human effort associated with providing explicit labels for negative samples, it assumes that the target sound events in the sequence are very sparse. This may not always be the case and can lead to unreliable negative prototypes created from randomly selected positive samples.Furthermore, typically only classes of target sound events are used during training, which prevents the model from learning a suitable representation for background (i.e., non-target) sounds, as the background is often more diverse than the target sounds in the support set. However, a single negative prototype may not provide a suitable representation of the non-target sounds during inference, as there may be multiple clusters of negative samples that would naturally form different negative prototypes.

[0017] SED tasks further include a step for determining the onset and offset timestamps of target sounds using a threshold for the output predictions. Conventional SED systems use a fixed decision threshold, which may not be appropriate for all audio queries due to diverse backgrounds and events.In the context of prototypical networks applied to SED tasks, systems and methods according to the present disclosure implement (i) explicitly including non-target sounds during training to learn a background-aware embedding representation; (ii) constructing the negative prototype at inference using the non-target portions of the support set instead of randomly sampling from the entire audio sequence (support and query sentences); (iii) including clusters on the negative samples of the support set at inference time to account for variability in non-target sounds; and (iv) implementing leave-one-sample-out cross-validation on the support set to automatically identify the decision threshold that would provide the best separation between background and events of interest.

[0018] Few-shot learning is a subdomain of meta-learning that essentially studies how machines "learn to learn" and attempts to mimic the human ability to learn a new concept with exposure to very few examples. This can be very useful in detecting sound events that are less frequent or rare compared to their background. Using few-shot learning removes the burden of collecting a large number of labeled samples of such rare sound events. Models trained on regular sound events (i.e., a training set) can be tuned to identify these rare events with few examples (i.e., a support set).Models that are typically trained in a fully supervised manner can be tuned to identify these rare events using few examples from a support set.

[0019] Although few-shot learning has been widely used in computer vision, it has recently gained interest in the field of audio research. Several few-shot learning methods (e.g., twin, correspondence, prototypical, and relation networks) were investigated for sound event detection in an English speech corpus by formulating the problem as an N-way K-shot problem. In some examples, four standard convolutional blocks were used as an embedding network before the few-shot networks. Results from these examples suggested that prototypical networks outperformed other few-shot methods at inference time when K positive samples are available for the test data, and the full audio data is used to generate negative examples for the support set by random sampling.

[0020] A similar approach is used to generate the support set in other examples, which also found that prototypical networks with a convolutional embedding extractor performed better than other few-shot approaches. Using a deeper embedding network such as ResNet showed similar performance. In an effort to create a few-shot model with better generalization, another example integrates an attention similarity module on top of the few-shot models to ensure appropriate detection of the onset and offset of sound events. Furthermore, this example used synthetic background noise to create a naturalistic setup.Yet another example explicitly introduced a background class from an external dataset during training and a contrastive loss associated with the dataset, allowing the model to learn the background sound in addition to the target events. The results of this example indicated that models perform better when background samples are included in the support set during training along with a target event. Due to the synthetic nature of the background sound, domain mismatch can occur due to the difference in the environment between different datasets. Therefore, including a background sound class from the training set can reduce the mismatch problem and improve detection performance.

[0021] As described above, previous examples in few-shot sound event detection primarily focus on training an efficient model, but ignore the processing required during inference time. Further examples explore the use of clustering in the support set during inference time to modify the general framework of a prototype network to solve a computer vision problem. Other examples attempt to reduce intra-class variation within the support set. These examples are not used in few-shot sound event detection tasks, where background sounds tend to contain more variety than target sounds. A single prototype for negative samples may be insufficient, and clustering can be implemented to address this problem.

[0022] Accordingly, systems and methods according to the principles of the present disclosure implement various techniques to improve prototype networks for SED (e.g., a first technique applied during training and three techniques applied during inference) by taking into account the variability of background sounds and automatically adapting to different test scenarios.

[0023] Fig. 1 shows an exemplary system 100 for training a neural network (e.g., an ML model). The system 100 may be configured to implement (and / or include circuitry configured to implement) the systems and methods of the present disclosure described in more detail below. The system 100 may include an input interface for accessing training data 102 for the neural network. As shown in Fig. 1, the input interface may be formed, for example, by a data storage interface 104 that can access the training data 102 from the data storage 106. The data storage interface 104 may, for example, be a memory interface or a persistent storage interface, e.g., a hard disk or SSD interface, but also a personal, local, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage 106 may be an internal data storage of the system 100, such as a hard disk or SSD, but also an external data storage, e.g., a network-accessible data storage.

[0024] In some embodiments, the data store 106 may further include a data representation 108 of an untrained version of the neural network, which the system 100 may access from the data store 106. However, it should be understood that the training data 102 and the data representation 108 of the untrained neural network may each be accessed from a different data store, e.g., via a different subsystem of the data store interface 104. Each subsystem may be of a type as described above for the data store interface 104.

[0025] In some embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on design parameters for the neural network and therefore may not be explicitly stored on the data store 106. The system 100 may further include a processor subsystem 110 that may be configured to provide an iterative function as a replacement for a stack of layers of the neural network to be trained during operation of the system 100. Here, respective layers of the stack of layers being replaced may have shared weights and receive as input an output of a previous layer or, for a first layer of the stack of layers, an initial activation and a portion of the input of the stack of layers.

[0026] The processor subsystem 110 may be further configured to iteratively train the neural network using the training data 102. Here, an iteration of training by the processor subsystem 110 may include a forward propagation part and a backward propagation part. The processor subsystem 110 may be configured to perform the forward propagation part by determining, among other operations that define the forward propagation part that may be performed, an equilibrium point of the iterative function at which the iterative function converges to a fixed point, wherein determining the equilibrium point comprises using a numerical root-finding algorithm to find a root solution for the iterative function minus its input, and providing the equilibrium point as a substitute for an output of the stack of layers in the neural network.

[0027] The system 100 may further comprise an output interface for outputting a data representation 112 of the trained neural network, which data may also be referred to as trained model data 112. As also in Fig. 1, the output interface may, for example, be formed by the data storage interface 104, wherein the interface in these embodiments is an input / output ("I / O") interface via which the trained model data 112 may be stored in the data storage 106. For example, the data representation 108 defining the "untrained" neural network may be at least partially replaced by the data representation 112 of the trained neural network during or after training by adjusting the parameters of the neural network, such as weights, hyperparameters, and other types of neural network parameters, to reflect the training on the training data 102. This is also illustrated in Fig. 1 by reference numerals 108, 112, which refer to the same data set on the data storage 106. In some embodiments, the data representation 112 may be stored separately from the data representation 108 that defines the "untrained" neural network. In some embodiments, the output interface may be separate from the data storage interface 104, but may generally be of a type described above for the data storage interface 104.

[0028] Fig. 2 illustrates a data annotation / augmentation system 200 configured (and / or including circuitry configured) to implement a system for annotating and / or augmenting data. The data annotation system 200 may include at least one computing system 202 configured to implement all or portions of the systems and methods of the present disclosure discussed in more detail below. The computing system 202 may include at least one processor 204 operatively connected to a memory device 208. The processor 204 may include one or more integrated circuits implementing the functionality of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit implementing an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families.Various components of the system 200 may be implemented with the same or different circuits.

[0029] During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software that controls the operation of CPU 206 to perform the operation described herein. In some embodiments, processor 204 may be a system-on-a-chip (SoC) that integrates the functionality of CPU 206, memory unit 208, a network interface, and input / output interfaces into a single integrated device. Computing system 202 may implement an operating system to manage various aspects of operation.

[0030] The storage unit 208 may include volatile memory and non-volatile memory for storing instructions and data. The non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is deactivated or loses electrical power. The volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and Fig. 6 stores data. For example, the storage unit 208 may store a machine learning model 210 (e.g., in Fig. 2 as the ML model 210) or an algorithm, a training dataset 212 for the machine learning model 210, a raw source dataset 216, etc.

[0031] Computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. Network interface device 222 may include, for example, a wired and / or wireless Ethernet interface as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. Network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). Network interface device 222 may be further configured to provide a communication interface to an external network 224 or a cloud.

[0032] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may enable information and data to be easily exchanged between computing devices and networks. One or more servers 230 may be in communication with the external network 224.

[0033] Computing system 202 may include an input / output (I / O) interface 220 that may be configured to provide digital and / or analog inputs and outputs. I / O interface 220 may include additional serial interfaces for communicating with external devices (e.g., Universal Serial Bus (USB) interface).

[0034] Computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables system 200 to receive control input. Examples of input devices may include human interface inputs such as keyboards, mice, touchscreens, voice input devices, and other similar devices. Computing system 202 may include a display device 232. Computing system 202 may include hardware and software for outputting graphic and text information to display device 232. Display device 232 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator.The computing system 202 may be further configured to enable interaction with remote HMI and remote display devices via the network interface device 222.

[0035] System 200 may be implemented using one or more computing systems. Although the example illustrates a single computing system 202 implementing all of the described features, it is intended that various features and functions may be implemented separately and by multiple computing units communicating with each other. The particular system architecture selected may depend on a variety of factors.

[0036] The system 200 may implement a machine learning model 210 configured to analyze the raw source data set 216. For example, the CPU 206 and / or other circuitry may implement the machine learning model 210. The raw source data set 216 may include raw or unprocessed sensor data that may be representative of an input data set for a machine learning system. The raw source data set 216 may include video, video segments, images, audio, text-based information, and raw or partially processed sensor data (e.g., radar map of objects). In some embodiments, the machine learning model 210 may be a deep learning or neural network algorithm designed to perform a predetermined function. For example, the neural network algorithm may be configured to identify events or objects in video segments based on audio data.

[0037] The computer system 200 may store the training dataset 212 for the machine learning model 210. The training dataset 212 may represent a set of previously constructed data for training the machine learning model 210. For example, according to the present disclosure, the training dataset 212 may include multiple automatically collected ground truth measurements and associated data. The training dataset 212 may be used by the machine learning model 210 to learn weighting factors associated with a neural network algorithm. The training dataset 212 may include a set of source data having corresponding results or results that the machine learning model 210 attempts to duplicate through the learning process.

[0038] The machine learning model 210 may be operated in a learning mode using the training dataset 212 as input. The machine learning model 210 may be executed over a number of iterations using the data from the training dataset 212. During each iteration, the machine learning model 210 may update internal weighting factors based on the results obtained. For example, the machine learning model 210 may compare output results (e.g., annotations) with those contained in the training dataset 212. Because the training dataset 212 includes the expected results, the machine learning model 210 may determine when an operating performance is acceptable. After the machine learning model 210 has reached a predetermined operating performance level (e.g.,100% agreement with the results associated with the training dataset 212), the machine learning model 210 may be executed using data not included in the training dataset 212. The trained machine learning model 210 may be applied to new datasets to generate annotated data.

[0039] The machine learning model 210 may be configured to identify a particular feature in the raw source data 216. The raw source data 216 may include a plurality of instances or an input data set for which annotation results are desired (e.g., a video stream or a video segment that includes audio data). For example only, the machine learning model 210 may be configured to identify objects or events in a video segment based on audio data and annotate the events. The machine learning model 210 may be programmed to process the raw source data 216 to identify the presence of the particular features. The machine learning model 210 may be configured to identify a feature in the raw source data 216 as a predetermined feature. The raw source data 216 may be derived from a variety of sources.For example, the raw source data 216 may be actual input data collected by a machine learning system. The raw source data 216 may be machine-generated for testing the system. As an example, the raw source data 216 may include raw video and / or audio data from a camera, audio data from a microphone, etc.

[0040] In one example, the machine learning model 210 may process raw source data 216 and output video and / or audio data including one or more indications of an identified event. The machine learning model 210 may generate a confidence level or confidence factor for each generated output. For example, a confidence value that exceeds a predetermined high-confidence threshold may indicate that the machine learning model 210 is confident that the identified event (or feature) corresponds to the particular event. A confidence value that is less than a low-confidence threshold may indicate that the machine learning model 210 has some uncertainty that the particular feature is present.

[0041] As is generally the case in the Fig. 3A and Fig. 3B, a system 300 may include an image (e.g., video) capture device 302, an audio capture device 304, and the computing system 202. The system may receive video stream data associated with a data capture environment from the image capture device 302. The system 202 may be configured to perform video object detection to identify one or more objects in corresponding images of the video stream data. The system 202 may receive audio stream data corresponding to at least a portion of the video stream data from the audio capture device 304. The audio capture device 304 may include one or more microphones 306 or other suitable audio capture devices. The systems and methods described herein may be configured to perform video object detection using an output of at least a first machine learning model (e.g.,such as the machine learning model 210 or another suitable machine learning model configured to provide an output including one or more object or event detection predictions) to label at least some objects of the video stream data and / or audio stream data.

[0042] The system 202 may calculate at least one offset value for at least one portion of the audio stream data corresponding to at least one labeled object of the video stream data based on at least one data acquisition characteristic (e.g., using at least one probability-based function or other suitable technique or function). The system 202 may synchronize at least one portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using at least the at least one offset value.The at least one data capture characteristic may include one or more characteristics of the at least one image capture device, one or more characteristics of the at least one audio capture arrangement, one or more characteristics corresponding to a location of the at least one image capture device relative to the at least one audio capture arrangement, one or more characteristics corresponding to a movement of an object in the video stream data, one or more other suitable data capture characteristics, or a combination thereof.

[0043] The system 202 may label at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using one or more labels of the labeled objects (labeled objects or objects tagged with a label) of the video stream data and the at least one offset value. Each respective label may include an event type, an event start indicator, and an event end indicator. The system 202 may generate training data using at least a portion of the labeled portion of the audio stream data. The system 202 may train a second machine learning model using the training data. The system 202 may detect, using the second machine learning model, one or more sounds (sound events) associated with audio data provided as input to the second machine learning model.The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable function, such as those described herein with respect to . Fig. 4-11 are described.

[0044] In some embodiments, as generally in Fig. 3C, the computing system 202 may be configured to tag (i.e., label) audio data based on sensor data received from one or more sensors, such as those described herein or any other suitable sensor or combination of sensors. The system 202 may receive audio stream data associated with a data capture environment from the audio capture assembly 354 or any suitable audio capture device, such as one or more of the microphones 306 or other suitable audio capture device. It is understood that the audio capture assembly 354 may include features similar to those of the audio capture assembly 304 and may include any suitable number of audio capture devices. The system 202 may receive audio stream data from at least one sensor (e.g.,The audio pickup assembly 354 may receive sensor data associated with the data pickup environment from a sensor (such as sensor 352) that is asynchronous with respect to the audio pickup assembly 354. The sensor 354 may include at least one of an induction coil, a radar sensor, a LiDAR sensor, a sonar sensor, an image pickup device, any other suitable sensor, or a combination thereof. The audio pickup assembly 354 may be located remotely from the sensor 354, proximate to the sensor 354, or in any suitable relationship to the sensor 354.

[0045] The system 202 may identify at least some events in the sensor data using an output from at least a first machine learning model, such as the machine learning model 210 or another suitable machine learning model. The machine learning model 210 may be configured to provide an output including one or more event detection predictions based on the sensor data. The system 202 may synchronize at least a portion of the sensor data associated with the portion of the audio stream data corresponding to the at least one event of the sensor data. The system 202 may identify at least the portion of the audio stream data corresponding to the at least one event of the sensor data using one or more labels extracted for respective events of the sensor data value.Each respective label may include an event type, an event start indicator, and an event end indicator. The system 202 may generate training data using at least a portion of the labeled portion of the audio stream data. The system 202 may train a second machine learning model using the training data. The system 202 may detect, using the second machine learning model, one or more sounds associated with audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable function, such as those described herein with respect to. Fig. 4-11 are described.

[0046] Any of the systems described above and / or in more detail below may implement systems and methods of the present disclosure to perform an SED.

[0047] Prototypical networks learn the representation of different classes through support set examples. Using the embedding vectors for positive and negative samples in the support set, a prototype is formulated for each class, as shown in Equation (1) below: cpos=1|Spos|∑i=1|Spos|Θ(xipos) cneg=1|Sneg|∑i=1|Sneg|Θ(xineg),

[0048] In equation (1), Θ(·) represents a generic differentiable feature extractor that, given a sample at the input, produces its dimensional embedding as output. For a query sample x queryThe squared Euclidean distances between the query embedding and each class prototype are calculated, and the distances are passed through a softmax function to obtain the class probability distribution of the sample. The network is optimized using stochastic gradient descent of the negative log-likelihood function.

[0049] To train the model according to systems and methods of the present disclosure, an explicit background class is introduced during the training phase so that the model can learn a representation of both the target events and the background noise. Instead of creating a synthetic background class, a background class is created from the given data. The background class is composed of the unlabeled segments in X(t), X(t), t <tKend sampled. For a given ytrain={(tistart,tiend,yitrain)}, so that yitrain∈{1,…,Ctrain} is, the time slots outside the range of tiend and ti+1start marked as a background class and given a new label C train + 1. Therefore, the new training labels would be ytrain={(tistart,tiend,yitrain)}∪{(tiend,ti+1start,Ctrain+1)}, so that yitrain∈{1,…,Ctrain+1} is.

[0050] Systems and methods according to the present disclosure are further configured to construct the negative support set using unlabeled segments from the support set. For example, typically the negative support set S neg by randomly sampling segments from an entire audio stream (e.g. an audio stream 400, as in Fig. 4A, including the query set, is constructed under the assumption that the target sound events are sparse. However, the assumption that the target sound events are sparse may not be true for all cases, and therefore, systems and methods of the present disclosure construct S neg using unlabeled segments from the support set (Xtest(t),t <tKend) (i.e., from a known part of the signal). Although this approach limits the available number of negative segments for constructing S neg reduced, the model is able to produce a better negative prototype using the refined S neg to learn without being contaminated by positive events randomly sampled from the entire audio sequence.

[0051] Systems and methods according to the present disclosure are further configured to cluster the negative support set during inference. To account for the diversity in the negative support set during inference time, a k-means clustering algorithm is applied to the embedding vectors generated from the samples of the negative support set S neg obtained. In some examples, it can be assumed that the samples from the target sound events are more homogeneous than the negative samples. Fig. Figure 4B shows a two-dimensional t-SNE projection 402 obtained from the embeddings of positive and negative support sentences for an audio sequence. The negative embeddings show two distinct clusters (e.g., a negative sentence in clusters 0 and 1), while the positive embeddings are centered between the negative embeddings. By creating only one prototype for the negative class, a query sample from this sequence can easily be misclassified due to the proximity of the resulting prototypes. As an example, k-means clustering is performed on the embeddings of the S neg-Samples are performed in each of the respective clusters 0 and 1, with an exemplary value of k ranging from 2 to 10. The value of k for which a highest silhouette coefficient is obtained is then selected (e.g., 2 for the present example). Based on this observation, one positive prototype 404 and two negative prototypes 406, 408 are obtained for each audio stream. Accordingly, equation (1) can be modified as follows: cnegj=1|Γj|∑Θ(xineg)∈ΓjΘ(xineg),j∈{0…k−1}, where Γ j is the set of sample indices assigned to cluster j.

[0052] Furthermore, instead of a binary classification, the inference problem can be formulated as a ternary K-shot classification problem with one class for positive labels and two classes for negative labels (e.g., as performed during inference, not during training).

[0053] Fig. 4C further illustrates exemplary clustering of support sets according to the principles of the present disclosure. As shown at 420, positive samples x1pos,…,xNpospos, taken from the support set (e.g. from the known part of the signal) into positive embeddings e1pos,…,eNpospos embedded / encoded (e.g., by the encoder or coding circuit 422). Conversely, negative samples x1neg,…,xNnegneg, taken from the support set into negative embeddings e1neg,…,eNnegneg embedded / encoded.

[0054] As shown at 424, the positive embeddings are aggregated (e.g., by the aggregator or aggregator circuit 426) to produce a positive prototype for each class p1pos,…,pCpos For example, embeddings of positive samples from the same class can be averaged. Conversely, as shown at 428, the negative embeddings are aggregated (e.g., by the aggregator or aggregator circuit 430, which may be the same as or different from the aggregator 426) to generate a negative prototype for each group of negative samples. p1neg,…,pKneg Typically, embeddings of negative samples are averaged into a single prototype (i.e., as a group). In contrast, according to the principles of the present disclosure, the negative embeddings are clustered into K different groups, and K different negative prototypes are obtained.

[0055] Systems and methods according to the present disclosure are further configured to perform automatic threshold identification. The class probability distribution of a query sample is obtained via softmax over the inverse of the Euclidean distances between the query embeddings and class prototypes. A query is classified as a target sound event if the prediction probability for the positive class is above a threshold. In some examples, the prediction probability value is empirically set to 0.5. In systems and methods according to the present disclosure, a computational approach is used to identify a threshold that is individually adjusted for each audio sequence. For example, a leave-one-sample-out cross-validation approach is used on all samples of the support set to estimate the probability distribution.The probability for a sample to be classified as a target event is calculated for each sample in the support set.

[0056] It can be assumed that probabilities resulting from S neg would have a long tail on the right side of the distribution. Conversely, probabilities from S pos have a long tail on the left side of the distribution. Accordingly, two tail probabilities from these two distributions can be considered as the candidates for an evaluation threshold to reduce the false positive rate. In one example, a 95th percentile of the probabilities obtained from S neg is considered a candidate because the number of negative samples is larger. Conversely, a 25th percentile of the probabilities from S posconsidered as a different candidate. The selection of the 25th percentile over the 95th percentile is motivated by the limited number of positive samples in the support set.

[0057] Fig. Figure 4D shows an example of the probability distributions obtained from positive support set samples 432 and negative support set samples 434 and their respective candidate thresholds (e.g., prediction probability thresholds) 436 and 438. The candidate threshold with a highest F-measure over the support set is taken as the evaluation threshold for the specific audio.

[0058] Fig. 4E further illustrates exemplary probability calculations according to the principles of the present disclosure. For example, as shown at 440, probabilities d1pos,…,dNpospos obtained that each of the positive samples e1pos,…,eNpospos to a given class p1pos,…,pCpos The distribution of values d1pos,…,dNpospos is shown as solid line 432 in Fig. 4D. Conversely, as shown in 442, probabilities d1neg,…,dNposneg obtained that each of the negative samples e1neg,…,eNnegneg to a given group p1neg,…,pKneg The distribution of values d1neg,…,dNposneg is shown as dashed line 434 in Fig. 4D. As shown at 444, threshold values τ1pos,…,τCpos for each class based on the distributions of d1pos,…,dNpospos and d1neg,…,dNposneg The thresholds separate negative samples from positive samples.

[0059] Fig. 4F illustrates an exemplary method 450 for determining a class of a sample x q according to the principles of the present disclosure (e.g., as described in any of the Fig. 1-3 and Fig. 5-11 described systems). A coded sample or an embedding e q of the sample is obtained at 452. A probability that the sample x q belongs to a given class or group is calculated at 454 (e.g. based on the positive prototypes p1pos,…,pCpos and the negative prototype p1neg,…,pKneg. The calculated probabilities are compared with the threshold values ​​of each class τ1pos,…,τCpos at 456. Accordingly, at 456 a determination is made as to whether the sample x q belongs to a given class {1, 2, ..., K} or belongs to no class.

[0060] Fig. 5- Fig. 11 illustrate example systems and devices that may implement systems and methods according to the present disclosure. Fig. 5 illustrates a schematic diagram of an interaction between a computer-controlled machine 500 and a control system 502. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a state of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed state into sensor signals 508 and transmit the sensor signals 508 to the control system 502. Non-limiting examples of the sensor 506 include video, radar, LiDAR, ultrasonic, and motion sensors. In some embodiments, the sensor 506 is an optical sensor configured to capture optical images of an environment proximate the computer-controlled machine 500.

[0061] The control system 502 is configured to receive sensor signals 508 from the computer-controlled machine 500. As set forth below, the control system 502 may be further configured to calculate actuator control commands 510 in response to the sensor signals and to transmit actuator control commands 510 to the actuator 504 of the computer-controlled machine 500.

[0062] As in Fig. 5, the control system 502 includes a receiving unit 512. The receiving unit 512 may be configured to receive sensor signals 508 from the sensor 506 and convert sensor signals 508 into input signals x. In an alternative embodiment, sensor signals 508 are received directly as input signals x without the receiving unit 512. Each input signal x may be a portion of each sensor signal 508. The receiving unit 512 may be configured to process each sensor signal 508 to generate each input signal x. The input signal x may include data corresponding to an image recorded by the sensor 506.

[0063] The control system 502 includes a classifier 514. The classifier 514 may be configured to classify the input signals x into one or more labels using a machine learning (ML) algorithm, such as a neural network described above. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameters θ may be stored in and provided by the non-volatile memory 516. The classifier 514 is configured to determine output signals y from the input signals x. Each output signal y includes information assigning one or more labels to each input signal x. The classifier 514 may transmit output signals y to the conversion unit 518. The conversion unit 518 is configured to convert output signals y into actuator control commands 510.The control system 502 is configured to transmit actuator control commands 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control commands 510. In some embodiments, the actuator 504 is configured to actuate the computer-controlled machine 500 directly based on the output signals y.

[0064] Upon receipt of actuator control commands 510 by actuator 504, actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. Actuator 504 may include control logic configured to convert actuator control commands 510 into a second actuator control command used to control actuator 504. In one or more embodiments, actuator control commands 510 may be used to control a display instead of or in addition to an actuator.

[0065] In some embodiments, control system 502 includes a sensor 506 instead of or in addition to computer-controlled machine 500 including sensor 506. Control system 502 may also include an actuator 504 instead of or in addition to computer-controlled machine 500 including actuator 504.

[0066] As in Fig. 5, control system 502 also includes a processor 520 and a memory 522. Processor 520 may include one or more processors. Memory 522 may include one or more storage devices. Classifier 514 (e.g., ML algorithms) of one or more embodiments may be implemented by control system 502, which includes non-volatile memory 516, processor 520, and memory 522.

[0067] Non-volatile memory 516 may include one or more persistent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, cloud storage, or any other device capable of persistently storing information. Processor 520 may include one or more devices selected from high-performance computing (HPC) systems, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other devices that manipulate signals (analog or digital) based on computer-executable instructions residing in memory 522.The memory 522 may include a single memory device or a number of memory devices, including, but not limited to, random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0068] Processor 520 may be configured to read into memory 522 and execute computer-executable instructions located in non-volatile memory 516 that embody one or more ML algorithms and / or methodologies of one or more embodiments. Non-volatile memory 516 may include one or more operating systems and applications. Non-volatile memory 516 may store, compiled and / or interpreted, computer programs created using a variety of programming languages ​​and / or technologies, including, among others, and either alone or in combination, Java, C, C++, C#, Objective C, Fortran, Pascal, Java Script, Python, Perl, and PL / SQL.

[0069] When executed by processor 520, the computer-executable instructions of non-volatile memory 516 may cause control system 502 to implement one or more of the ML algorithms and / or methodologies disclosed herein. Non-volatile memory 516 may also include ML data (including data parameters) that support the functions, features, and processes of one or more embodiments described herein.

[0070] The program code embodying the algorithms and / or methodologies described herein may be distributed individually or collectively as a program product in a variety of different forms. The program code may be distributed using a computer-readable storage medium having computer-readable program instructions embodied thereon for causing a processor to perform aspects of one or more embodiments. Computer-readable storage media that are inherently non-transitory may include volatile and non-volatile, and removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.Computer-readable storage media may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technology, portable compact disc read-only memory (CD-ROM) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be read by a computer. Computer-readable program instructions may be downloaded to a computer, another type of programmable data processing apparatus, or other device from a computer-readable storage medium, or to an external computer or external storage device over a network.

[0071] Computer-readable program instructions stored on a computer-readable medium can be used to instruct a computer, other types of programmable data processing devices, or other devices to function in a particular manner, such that the instructions stored on the computer-readable medium produce an article of manufacture including instructions that implement the functions, acts, and / or operations specified in the flowcharts or diagrams. In certain alternative embodiments, the functions, acts, and / or operations specified in the flowcharts and diagrams can be reordered, processed serially, and / or processed concurrently, consistent with one or more embodiments.Moreover, any of the flowcharts and / or diagrams may include more or fewer nodes or blocks than those illustrated, consistent with one or more embodiments.

[0072] The processes, methods, or algorithms may be embodied, in whole or in part, using suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0073] Fig. 6 illustrates a schematic diagram of a control system 502 configured to control a vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or position sensors (e.g., GPS). To implement systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 is configured to detect sound events in a proximity of the vehicle 600. As one example, the control system 502 may be configured to control autonomous functions, such as navigation, of the vehicle 600 in response to detected sound events.

[0074] One or more of the one or more specific sensors may be integrated into the vehicle 600. Alternatively, or in addition to one or more of the specific sensors identified above, the sensor 506 may include a software module configured, when executed, to determine a state of the actuator 504. A non-limiting example of a software module includes a weather information software module configured to determine a current or future state of the weather in the vicinity of the vehicle 600 or another location.

[0075] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect objects in the vicinity of the vehicle 600 based on the input signals x. In such an embodiment, the output signal y may include information characterizing the proximity of objects to the vehicle 600. The actuator control command 510 may be determined according to this information. The actuator control command 510 may be used to avoid collisions with the detected objects.

[0076] In some embodiments, the vehicle 600 is an at least partially autonomous vehicle; the actuator 504 may be embodied in a brake, drive system, engine, drivetrain, or steering system of the vehicle 600. The actuator control commands 510 may be determined such that the actuator 504 is controlled such that the vehicle 600 avoids collisions with detected objects. Detected objects may also be classified according to what the classifier 514 considers most likely, such as pedestrians or trees. The actuator control commands 510 may be determined depending on the classification. In a scenario where an adversary attack may occur, the system described above may be further trained to better detect objects or to identify a change in lighting conditions or angle for a sensor or camera on the vehicle 600.

[0077] In some embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and pedaling. The mobile robot may be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In such embodiments, the actuator control command 510 may be determined such that a drive unit, steering unit, and / or braking unit of the mobile robot may be controlled such that the mobile robot can avoid collisions with identified objects.

[0078] In some embodiments, the vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, the vehicle 600 may use an optical sensor as sensor 506 to determine a condition of plants in an environment near the vehicle 600. The actuator 504 may be a nozzle configured to spray chemicals. Depending on an identified species and / or condition of the plants, the actuator control command 510 may be determined to cause the actuator 504 to spray the plants with an appropriate amount of suitable chemicals.

[0079] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect a condition of an object to be processed by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor 506 may detect a condition of the laundry within the washing machine. The actuator control command 510 may be determined based on the detected condition of the laundry.

[0080] Fig. 7 illustrates a schematic diagram of control system 502 configured to control a system 700 (e.g., a manufacturing machine), such as a punch cutter, a cutter, or a pistol drill, of a manufacturing system 702, such as a portion of a production line. Control system 502 may be configured to control actuator 504 configured to control system 700 (e.g., a manufacturing machine).

[0081] The sensor 506 of the system 700 (e.g., a manufacturing machine) may be an optical sensor configured to detect one or more characteristics of the manufactured product 704. To implement systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 is configured to detect sound events related to the operation of the manufacturing machine in a manufacturing facility. The classifier 514 may be configured to determine a state of the manufactured product 704 from one or more of the captured characteristics, such as detected sound events. The actuator 504 may be configured to control the system 700 (e.g., a manufacturing machine) for a subsequent manufacturing step of the manufactured product 704 depending on the determined state of the manufactured product 704.The actuator 504 may be configured to control functions of the system 700 (e.g., a manufacturing machine) on the subsequent manufactured product 706 of the system 700 (e.g., a manufacturing machine) depending on the determined state of the manufactured product 704.

[0082] Fig. Figure 8 illustrates a schematic diagram of a control system 502 configured to control a power tool 800, such as a power drill or screwdriver, having an at least partially autonomous mode. The control system 502 may be configured to control the actuator 504 configured to control the power tool 800.

[0083] The sensor 506 of the power tool 800 may be an optical sensor configured to detect one or more characteristics of the work surface 802 and / or the fastener 804 being driven into the work surface 802. To implement systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 is configured to detect sound events related to the operation of the power tool 800. The classifier 514 may be configured to determine a condition of the work surface 802 and / or the fastener 804 relative to the work surface 802 from one or more of the detected characteristics, such as detected sound events. The condition may be that the fastener 804 is flush with the work surface 802. Alternatively, the condition may be the hardness of the work surface 802.The actuator 504 may be configured to control the power tool 800 such that the drive function of the power tool 800 is adjusted depending on the determined condition of the fastener 804 relative to the work surface 802 or one or more sensed properties of the work surface 802. For example, the actuator 504 may interrupt the drive function when the condition of the fastener 804 relative to the work surface 802 is flush. As another non-limiting example, the actuator 504 may apply additional or less torque depending on the hardness of the work surface 802.

[0084] Fig. 9 illustrates a schematic diagram of control system 502 configured to control automated personal assistant 900. Control system 502 may be configured to control actuator 504, which is configured to control automated personal assistant 900. Automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, a stove, an oven, a microwave, or a dishwasher.

[0085] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive video images of gestures 904 of user 902. The audio sensor may be configured to receive a voice command from user 902. To implement systems and methods according to the present disclosure, control system 502 is configured to detect sound events in a proximity of automated personal assistant 900, such as sound events associated with the operation of devices controlled by automated personal assistant 900.

[0086] The control system 502 of the automated personal assistant 900 may be configured to determine actuator control commands 510 configured to control the system 502. The control system 502 may be configured to determine actuator control commands 510 according to sensor signals 508 from the sensor 506. The automated personal assistant 900 is configured to transmit sensor signals 508 to the control system 502. The classifier 514 of the control system 502 may be configured to execute a gesture recognition algorithm to identify the gesture 904 performed by the user 902, to determine actuator control commands 510, and to transmit the actuator control commands 510 to the actuator 504.The classifier 514 may be configured to retrieve information from the non-volatile memory in response to the gesture 904 and to output the retrieved information in a form suitable for receipt by the user 902.

[0087] Fig. 10 illustrates a schematic diagram of control system 502 configured to control surveillance system 1000. Surveillance system 1000 may be configured to physically control access through door 1002. Sensor 506 may be configured to detect a scene relevant to determining whether access is granted. Sensor 506 may be an optical sensor configured to generate and transmit image and / or video data. Such data may be used by control system 502 to detect a person's face. To implement systems and methods according to the present disclosure, sensor 506 may include at least one audio sensor (e.g., a microphone), and control system 502 is configured to detect sound events in a proximity of surveillance system 1000, such as sound events caused by people attempting to access door 1002.As an example, the control system 502 may be configured to prevent or allow access through the door 1002 in response to detected sound events.

[0088] The classifier 514 of the control system 502 of the surveillance system 1000 may be configured to interpret the image and / or video data by matching identities of known individuals stored in the non-volatile memory 516, thereby determining an identity of a person. The classifier 514 may be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 may be configured to lock or unlock the door 1002 in response to the actuator control command 510. In some embodiments, non-physical, logical access control is also possible.

[0089] The monitoring system 1000 may also be a surveillance system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene being monitored, and the control system 502 is configured to control the display 1004. The classifier 514 is configured to determine a classification of a scene, e.g., whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to transmit an actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight an object deemed suspicious by the classifier 514.Using an embodiment of the disclosed system, the surveillance system can predict objects appearing at specific times in the future.

[0090] Fig.11 illustrates a schematic diagram of the control system 502 configured to control the imaging system 1100, for example, an MRI device, an x-ray imaging device, or an ultrasound device. The sensor 506 may, for example, be an imaging sensor. To implement systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 is configured to detect sound events in a proximity of the imaging system 1100, such as sounds caused by the operation of the imaging system 1100, users of the imaging system 1100, etc. As an example, the control system 502 may be configured to pause, interrupt, resume, etc., an imaging process in response to detected sound events.Classifier 514 may be configured to determine a classification of all or a portion of the acquired image. Classifier 514 may be configured to determine or select an actuator control command 510 in response to the classification obtained by the trained neural network. For example, classifier 514 may interpret a region of a acquired image as potentially anomalous. In this case, actuator control command 510 may be determined or selected to cause display 1102 to display the imaging and highlight the potentially anomalous region.

[0091] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms encompassed by the claims. The terms used in the description are words of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the disclosure. As described above, the features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated.Although various embodiments may be described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, one of ordinary skill in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system attributes depending on the particular application and implementation. These attributes may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, serviceability, weight, manufacturability, ease of assembly, etc.Thus, to the extent that embodiments are described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics, these embodiments are not outside the scope of the disclosure and may be desirable for certain applications.

Claims

[1] A method for training a prototype network for sound event detection, the method comprising: Receiving samples of an audio signal, the samples including positive samples corresponding to sound events and negative samples corresponding to non-sound events; Determining, based on the positive samples, respective positive prototypes of a plurality of classes of sound events; Determining, based on the negative samples, respective negative prototypes for respective groups of the negative samples, wherein each of the negative prototypes corresponds to a combination of a plurality of the negative samples; and Generating, based on comparisons between (i) a first sample and (ii) the respective positive prototypes and each of the negative prototypes, an output signal indicating whether a first sample belongs to one of the plurality of classes of sound events. [2] The method of claim 1, further comprising: Obtaining first embeddings of the positive samples and second embeddings of the negative samples; Determining the respective positive prototypes based on the first embeddings; and Determine the respective negative prototypes based on the second embeddings. [3] The method of claim 1, wherein determining the respective negative prototypes includes determining a negative prototype for each of the respective groups of negative samples. [4] The method of claim 1, further comprising: Determining, based on the comparisons, at least one probability that the first sample corresponds to one of the plurality of classes of sound events; and Generating the output based on the at least one probability. [5] The method of claim 4, further comprising: Determining at least one probability distribution; and Generating the output based on the at least one probability distribution. [6] The method of claim 5, further comprising: Determining at least one threshold based on the at least one probability distribution; and Generating the output based on a comparison between the at least one probability and the at least one threshold. [7] The method of claim 6, wherein determining the at least one threshold includes determining the at least one threshold based on (i) a first probability distribution of probabilities that the positive samples belong to respective classes of the plurality of classes of sound events, and (ii) a second probability distribution that the negative samples belong to respective classes of the plurality of classes of sound events. [8] A computing device configured to train a prototypical network for sound event detection, the computing device including a processing device configured to execute instructions stored in a memory to: Receiving samples of an audio signal, the samples including positive samples corresponding to sound events and negative samples corresponding to non-sound events; Determining, based on the positive samples, respective positive prototypes of a plurality of classes of sound events; Determining, based on the negative samples, respective negative prototypes for respective groups of the negative samples, wherein each of the negative prototypes corresponds to a combination of a plurality of the negative samples; and Generating, based on comparisons between (i) a first sample and (ii) the respective positive prototypes and each of the negative prototypes, an output signal indicating whether the first sample belongs to one of the plurality of classes of sound events. [9] The computing device of claim 8, wherein the processing device is further configured to execute the instructions to: Obtaining first embeddings of the positive samples and second embeddings of the negative samples; Determining the respective positive prototypes based on the first embeddings; and Determine the respective negative prototypes based on the second embeddings. [10] The computing device of claim 8, wherein determining the respective negative prototypes includes determining a negative prototype for each of the respective groups of negative samples. [11] The computing device of claim 8, wherein the processing device is further configured to execute the instructions to: Determining, based on the comparisons, at least one probability that the first sample corresponds to one of the plurality of classes of sound events; and Generating the output based on the at least one probability. [12] The computing device of claim 11, wherein the processing device is further configured to execute the instructions to: Determining at least one probability distribution; and Generating the output based on the at least one probability distribution. [13] The computing device of claim 12, wherein the processing device is further configured to execute the instructions to: Determining at least one threshold based on the at least one probability distribution; and Generating the output based on a comparison between the at least one probability and the at least one threshold. [14] The computing device of claim 13, wherein determining the at least one threshold includes determining the at least one threshold based on (i) a first probability distribution of probabilities that the positive samples belong to respective classes of the plurality of classes of sound events, and (ii) a second probability distribution that the negative samples belong to respective classes of the plurality of classes of sound events. [15] Computer-controlled machine comprising: at least one sensor configured to generate an audio signal; a control system configured to Receiving a first sample of the audio signal, and Generating, based on comparisons between (i) the first sample and (ii) respective positive prototypes for each of a plurality of classes of sound events and respective negative prototypes for each of a plurality of groups of negative prototypes, an output signal indicating whether the first sample of the audio signal belongs to one of the plurality of classes of sound events; and an actuator configured to control operation of the computer-controlled machine in response to the output of the control system. [16] A computer-controlled machine according to claim 15, further comprising a memory storing the respective positive prototypes and the respective negative prototypes, wherein: the respective positive prototypes correspond to a large number of positive samples; and each of the respective negative prototypes corresponds to a combination of a large number of negative samples. [17] The computer-controlled machine of claim 15, wherein generating the output includes calculating a probability that the first sample belongs to a first class of the plurality of classes of sound events or a first group of the plurality of groups. [18] The computer controlled machine of claim 17, wherein generating the output includes comparing the probability to at least one threshold and generating the output based on the comparison. [19] The computer-controlled machine of claim 18, wherein the at least one threshold includes a plurality of thresholds corresponding to respective classes of the plurality of classes of sound events. [20] A computer-controlled machine according to claim 15, wherein the computer-controlled machine includes an autonomous robot.

Images