System and method for improving accuracy and recall rate of prototype networks for sound event detection

Abstract

Systems and methods for improving the accuracy and recall rate of prototype networks for sound event detection. A method of training a prototype network for sound event detection includes: receiving samples of an audio signal, the samples including positive samples corresponding to sound events and negative samples not corresponding to sound events; determining corresponding positive prototypes of the plurality of sound event categories based on the positive samples; determining, based on the negative samples, a respective negative prototype for a respective negative sample group, each negative prototype corresponding to a combination of a plurality of negative samples; and generating an output signal indicating whether the first sample belongs to one of the plurality of sound event categories based on a comparison between the first sample and the respective positive prototype and each negative prototype.

Description

Technical Field

[0001] The present disclosure relates to acoustic event detection (SED), and more particularly to systems and methods of operating devices that implement SED. Background Art

[0002] Sound event detection (SED) refers to the task of identifying and distinguishing individual acoustic events within a continuous audio stream (e.g., birdsong, gunshots, smoke alarms, etc.). SED involves detecting the occurrence of a given event, as well as identifying the start and offset timestamps.

[0003] SED faces challenges compared to other mature areas of acoustic signal processing, such as automatic speech recognition (ASR). ASR processes spoken language, where sound events are constrained by the morphology of the human vocal tract, while SED processes more diverse (e.g., both periodic and episodic in nature) acoustic events. SED task domains range from home and large-scale security surveillance to bioacoustic event detection in the wild. Summary of the invention

[0004] A method for training a prototype network for sound event detection, comprising: receiving samples of an audio signal, the samples comprising positive samples corresponding to sound events and negative samples not corresponding to sound events; determining corresponding positive prototypes of a plurality of sound event categories based on the positive samples; determining corresponding negative prototypes of corresponding negative sample groups based on the negative samples, each negative prototype corresponding to a combination of a plurality of negative samples; and generating an output signal indicating whether the first sample belongs to one of the plurality of sound event categories based on a comparison between the first sample and the corresponding positive prototype and each negative prototype.

[0005] In other features, the method also includes obtaining a first embedding of a positive sample and a second embedding of a negative sample, determining a corresponding positive prototype based on the first embedding, and determining a corresponding negative prototype based on the second embedding. Determining the corresponding negative prototype includes determining a negative prototype for each of the corresponding groups of negative samples. The method also includes determining at least one probability that the first sample corresponds to one of a plurality of sound event categories based on the comparison, and generating an output based on the at least one probability. The method also includes determining at least one probability distribution, and generating an output based on the at least one probability distribution. The method also includes determining at least one threshold based on the at least one probability distribution, and generating an output based on a comparison between the at least one probability and the at least one threshold. Determining the at least one threshold includes determining the at least one threshold based on a first probability distribution of probabilities of the positive sample belonging to a corresponding category in a plurality of sound event categories and a second distribution of probabilities of the negative sample belonging to a corresponding category in a plurality of sound event categories.

[0006] A computing device configured to train a prototype network for sound event detection includes a processing device configured to execute instructions stored in a memory to: receive samples of an audio signal, the samples including positive samples corresponding to sound events and negative samples not corresponding to sound events; based on the positive samples, determine corresponding positive prototypes of multiple sound event categories; based on the negative samples, determine corresponding negative prototypes of corresponding negative sample groups, each negative prototype corresponding to a combination of multiple negative samples; and based on a comparison between the first sample and the corresponding positive prototype and each negative prototype, generate an output signal indicating whether the first sample belongs to one of the multiple sound event categories.

[0007] In other features, the processing device is further configured to execute the instructions to obtain a first embedding of a positive sample and a second embedding of a negative sample, determine a corresponding positive prototype based on the first embedding, and determine a corresponding negative prototype based on the second embedding. Determining the corresponding negative prototype includes determining a negative prototype for each of the corresponding negative sample groups. The processing device is further configured to execute the instructions to determine at least one probability that the first sample corresponds to one of a plurality of sound event categories based on the comparison, and to generate an output based on the at least one probability. The processing device is further configured to execute the instructions to determine at least one probability distribution, and to generate an output based on the at least one probability distribution. The processing device is further configured to execute the instructions to determine at least one threshold based on the at least one probability distribution, and to generate an output based on a comparison between the at least one probability and the at least one threshold. Determining the at least one threshold includes determining the at least one threshold based on a first probability distribution of probabilities of the positive sample belonging to a corresponding category in a plurality of sound event categories and a second distribution of probabilities of the negative sample belonging to a corresponding category in a plurality of sound event categories.

[0008] A computer-controlled machine comprises: at least one sensor configured to generate an audio signal; a control system configured to receive a first sample of the audio signal and generate an output signal indicating whether the first sample of the audio signal belongs to one of a plurality of sound event categories based on a comparison between the first sample and a corresponding positive prototype for each of a plurality of sound event categories and a corresponding negative prototype for each of a plurality of negative prototype groups; and an actuator configured to control the operation of the computer-controlled machine in response to an output of the control system.

[0009] In other features, the computer-controlled machine also includes a memory storing corresponding positive prototypes and corresponding negative prototypes. The corresponding positive prototypes correspond to multiple positive samples, and each of the corresponding negative prototypes corresponds to a combination of multiple negative samples. Generating an output includes calculating a probability that a first sample belongs to a first category in a plurality of categories or a first group in a plurality of groups. Generating an output includes comparing the probability to at least one threshold and generating an output based on the comparison. The at least one threshold includes a plurality of thresholds corresponding to corresponding categories in a plurality of sound event categories. The computer-controlled machine includes an autonomous robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A system for training a neural network according to the principles of the present disclosure is generally illustrated.

[0011] Figure 2 A computer-implemented method for training and utilizing a neural network in accordance with the principles of the present disclosure is generally illustrated.

[0012] Figure 3A An audio data tagging system according to the principles of the present disclosure is generally illustrated.

[0013] Figure 3B A portion of a data capture system in accordance with the principles of the present disclosure is generally illustrated.

[0014] Figure 3C An alternative audio data labeling system in accordance with the principles of the present disclosure is generally illustrated.

[0015] Figure 4A An example audio stream including a support set and a query set in accordance with the principles of the present disclosure is generally illustrated.

[0016] Figure 4B Positive and negative clustering of samples in accordance with the principles of the present disclosure are generally illustrated.

[0017] Figure 4C An example clustering of support sets in accordance with the principles of the present disclosure is generally illustrated.

[0018] Figure 4D

[0026] Illustrated generally are examples of probability distributions obtained from positive support set samples and negative support set samples in accordance with the principles of the present disclosure.

[0019] Figure 4E An example probability calculation in accordance with the principles of the present disclosure is generally illustrated.

[0020] Figure 4F The method for determining a sample x according to the principles of the present disclosure is generally illustrated. q An example method for a class.

[0021] Figure 5 Depicted is a schematic diagram of the interaction between a computer-controlled machine and a control system in accordance with the principles of the present disclosure.

[0022] Figure 6 Depicted is a schematic diagram of a device according to the principles of the present disclosure. Figure 5 Schematic diagram of a control system configured to control a vehicle, which can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0023] Figure 7 Depicted Figure 5 Schematic diagram of a control system configured to control a manufacturing machine (such as a punch cutter, a cutting machine, or a gun drill) of a manufacturing system (such as part of a production line).

[0024] Figure 8 Depicted Figure 5 Schematic diagram of a control system configured to control a power tool (such as a drill or driver) having at least a partially autonomous mode.

[0025] Fig. 9 Depicted Figure 5 Schematic diagram of a control system configured to control an automated personal assistant.

[0026] Fig.10 Depicted Figure 5 Schematic diagram of a control system configured to control a monitoring system (such as a controlled access system or a surveillance system).

[0027] Fig.11 Depicted Figure 5 Schematic diagram of a control system configured to control an imaging system (eg, an MRI device, an x-ray imaging device, or an ultrasound device). DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be magnified or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the figures may be combined with the features illustrated in one or more other figures to produce embodiments that are not explicitly illustrated or described. The combination of the illustrated features provides representative embodiments for typical applications. However, for specific applications or embodiments, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable.

[0029] As used herein, "a", "an", and "the" refer to both singular and plural referents unless the context clearly dictates otherwise. As an example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor that are collectively programmed to perform each of the various functions.

[0030] Due to the presence of both target sounds and non-target sounds in the environment, sound event detection (SED) is a challenging task. Correct target sound labels are necessary for machine learning models to detect sound events from audio streams. However, these models usually require a large amount of labeled data and application-specific model fine-tuning. In addition, collecting labels for target sound events in large audio collections requires professional raters (e.g., to obtain annotations for long audio recordings, mark the start and end timestamps of each event, etc.), especially when the event comes from a specific field (e.g., wildlife, machines, safety supervision), which increases the challenges in the annotation process. Therefore, SED may be expensive and time-consuming.

[0031] In recent years, prototypical networks have gained popularity by utilizing a small number of labeled samples per class (i.e., training deep learning SED models with less data), while showing competitive recall rates in SED that can be referenced. However, due to the diversity of background noise and soundscapes, these models often fail to distinguish between target and non-target sounds. In addition, these models are not designed to detect rare or previously unseen events during inference time.

[0032] Few-shot learning methods can be successful in cases where previously unseen classes can be inferred with only a few labeled samples. Several computer vision problems benefit greatly from the use of few-shot learning methods (e.g., prototype networks, matching networks, twin networks, etc.). Some recent examples frame the SED task with a small number of labeled samples as a few-shot learning task, where the model is trained in an episodic manner in an N-way K-shot formulation. During training, the model learns embedding representations over N classes in an episodic manner, where each episode uses only K labeled samples from each class to build N prototypes and classify the input sample.

[0033] For example, when provided with a new audio sequence containing events from unseen classes, the first part of the sequence is manually labeled (the "support set"), while the remainder of the sequence is used for inference (the "query set"). The model uses information from the support set to label the query set, which has at least K labeled samples for each new class. Prototypical networks are the best performing models for the few-shot SED task, where the model automatically finds the start and offset timestamps of events belonging to unknown classes during training in long audio sequences, assuming that the first K occurrences of that class are labeled in the support set. The model learns positive prototypes representing the class of interest along with the corresponding negative prototypes, and performs binary classification for each class presented in the support set at inference time.

[0034] Systems and methods according to the principles of the present disclosure identify and improve various aspects of SED systems. For example, when building prototypes, more attention is usually paid to creating positive prototypes using support sets, while negative prototypes are constructed by randomly sampling segments from the entire audio sequence. Although this approach reduces manual effort in providing explicit labels for negative samples, it assumes that the target sound events are very sparse in the sequence. This may not always be the case and may result in unreliable negative prototypes built on randomly selected positive samples. In addition, only the categories of target sound events are usually used during training, which prevents the model from learning the correct representation of background (i.e., non-target) sounds because the background is usually more diverse than the target sounds in the support set. However, a single negative prototype may not provide a correct representation of non-target sounds during inference because there may be multiple clusters of negative samples that will naturally form different negative prototypes.

[0035] The SED task also includes a step of determining the start and offset timestamps of the target sound using a threshold on the output prediction. Conventional SED systems use a fixed value for the decision threshold, which may not be applicable to all audio queries due to different contexts and events. In the context of a prototype network applied to the SED task, the systems and methods according to the present disclosure (i) implement explicit inclusion of non-target sounds during training to learn context-aware embedding representations; (ii) construct negative prototypes at inference time using the non-target portion of the support set rather than randomly sampling from the entire audio sequence (support set and query set); (iii) incorporate clustering of negative samples of the support set at inference time to accommodate the variability of non-target sounds; and (iv) implement leave-one-out cross-validation on the support set to automatically identify the decision threshold that will provide the best separation between background and events of interest.

[0036] Few-shot learning is a subfield of meta-learning that essentially explores how machines "learn to learn" and attempts to mimic the human ability to learn new concepts by being exposed to very few examples. This can be very useful in detecting sound events that are less frequent or rare compared to their background. The use of few-shot learning eliminates the burden of collecting a large number of labeled samples of such infrequent sound events. A model trained with regular sound events (i.e., the training set) can be tuned to identify these infrequent events with very few examples (i.e., the support set). Models that are typically trained in a fully supervised manner can be tuned to identify these infrequent events with very few examples from the support set.

[0037] Although few-shot learning has been widely used in computer vision, it has recently gained interest in the field of audio research. Several few-shot learning methods (e.g., twin networks, matching networks, prototype networks, and relational networks) have been explored for sound event detection in an English speech corpus by formulating the problem as an N-way K-shot problem. In some examples, four standard convolutional blocks are used as an embedding network before the few-shot network. The results of these examples show that the prototype network outperforms other few-shot methods at inference time when K positive samples are available for test data, and the complete audio data is used to generate negative samples of the support set by random sampling.

[0038] In other examples, similar methods are used to generate support sets, which also find that a prototype network with a convolutional embedding extractor performs better than other few-shot methods. Using deeper embedding networks such as ResNet shows similar performance. In order to create a few-shot model with better generalization, another example incorporates an attention similarity module on top of the few-shot model to ensure that the start and offset of the sound event are correctly detected. In addition, this example uses synthetic background noise to create a natural setting. Yet another example explicitly introduces background categories from an external dataset during training and a contrastive loss associated with the dataset so that the model can learn background sounds in addition to the target event. The results of this example indicate that the model performs better when background samples are included in the support set during training along with the target event. Due to the synthetic nature of background sounds, there may be domain mismatches due to differences in the environment in various datasets. Therefore, including background sound categories derived from the training set can reduce the mismatch problem and improve detection performance.

[0039] As described above, previous examples in few-shot sound event detection have mostly focused on training efficient models, but have ignored the processing required during inference time. Additional examples have explored the general framework of modifying prototype networks using clustering in the support set during inference time to solve computer vision problems. Other examples attempt to reduce intra-class variation within the support set. These examples are not used for few-shot sound event detection tasks, where background sounds often contain more variation than target sounds. A single prototype for negative samples may be insufficient, and clustering can be implemented to address this issue.

[0040] Thus, systems and methods according to the principles of the present disclosure implement various techniques to improve the prototype network of SED (e.g., the first technique applied during training and the three techniques applied during inference) by taking into account the variability of background sounds and automatically adapting to different test scenarios.

[0041] Figure 1 An example system 100 for training a neural network (e.g., of an ML model) is shown. System 100 may be configured to implement (and / or include circuitry configured to implement) the systems and methods of the present disclosure as described in more detail below. System 100 may include an input interface for accessing training data 102 for the neural network. For example, Figure 1 As illustrated in , the input interface may be constituted by a data storage interface 104 that can access the training data 102 from a data storage device 106. For example, the data storage interface 104 may be a memory interface or a persistent storage interface, such as a hard disk or SSD interface, but may also be a private, local or wide area network interface, such as a Bluetooth, Zigbee or Wi-Fi interface or an Ethernet or fiber optic interface. The data storage device 106 may be an internal data storage device of the system 100, such as a hard disk drive or SSD, but may also be an external data storage device, for example, a network accessible data storage device.

[0042] In some embodiments, data storage 106 may also include a data representation 108 of an untrained version of the neural network, which system 100 may access from data storage 106. However, it will be appreciated that training data 102 and data representation 108 of the untrained neural network may each be accessed from a different data storage, e.g., via a different subsystem of data storage interface 104. Each subsystem may have a data storage interface 104 of the type described above.

[0043] In some embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on the design parameters of the neural network, and thus may not be explicitly stored on the data storage device 106. The system 100 may also include a processor subsystem 110 that may be configured to provide an iterative function as a replacement for the stack of neural network layers to be trained during operation of the system 100. Here, the corresponding layers of the replaced layer stack may have mutually shared weights and may receive as input the output of the previous layer, or, for the first layer of the layer stack, receive as input an initial activation and a portion of the input of the layer stack.

[0044] The processor subsystem 110 may also be configured to iteratively train the neural network using the training data 102. Here, the training iterations of the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by, in addition to defining other operations of the forward propagation portion that may be performed, determining an equilibrium point of the iterative function at which the iterative function converges to a fixed point, wherein determining the equilibrium point includes using a numerical root finding algorithm to find a root solution of the iterative function minus its input; and by providing the equilibrium point as a surrogate for the output of a layer stack in the neural network.

[0045] The system 100 may also include an output interface for outputting a data representation 112 of the trained neural network, which may also be referred to as trained model data 112. Figure 1 , the output interface may be comprised of a data storage interface 104, which in these embodiments is an input / output ("IO") interface, via which trained model data 112 may be stored in a data storage device 106. For example, a data representation 108 defining an "untrained" neural network may be replaced, at least in part, with a data representation 112 of a trained neural network during or after training, as parameters of the neural network (such as weights, hyperparameters, and other types of parameters of the neural network) may be adapted to reflect the training on the training data 102. Figure 1 , this is also illustrated by reference numerals 108, 112 referring to the same data records on the data storage device 106. In some embodiments, the data representation 112 may be stored separately from the data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from the data storage interface 104, but may generally be of the type described above for the data storage interface 104.

[0046] Figure 2A data annotation / augmentation system 200 is depicted that is configured to implement a system for annotating and / or augmenting data (and / or includes circuitry configured to implement a system for annotating and / or augmenting data). The data annotation system 200 may include at least one computing system 202 that is configured to implement all or part of the systems and methods of the present disclosure explained in more detail below. The computing system 202 may include at least one processor 204 that is operatively connected to a memory unit 208. The processor 204 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit that implements an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families. The various components of the system 200 may be implemented with the same or different circuitry.

[0047] During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software that controls the operation of CPU 206 to perform the operations described herein. In some embodiments, processor 204 may be a system on a chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interfaces, and input / output interfaces into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation.

[0048] The memory unit 208 may include volatile memory and nonvolatile memory for storing instructions and data. Nonvolatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses power. Volatile memory may include memory that stores program instructions and data. Figure 6 Static and dynamic random access memory (RAM) for data. For example, memory unit 208 can store machine learning model 210 (e.g., in Figure 2 210), or an algorithm, a training data set 212 of the machine learning model 210, and an original source data set 216.

[0049] The computing system 202 may include a network interface device 222 that is configured to provide communications with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface as defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards. The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may also be configured to provide a communication interface to an external network 224 or a cloud.

[0050] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 may communicate with the external network 224.

[0051] Computing system 202 may include input / output (I / O) interface 220, which may be configured to provide digital and / or analog input and output. I / O interface 220 may include an additional serial interface (eg, a universal serial bus (USB) interface) for communicating with external devices.

[0052] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system 200 to receive control inputs. Examples of input devices may include human-machine interface inputs, such as keyboards, mice, touch screens, voice input devices, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphics and text information to the display device 232. The display device 232 may include an electronic display screen, a projector, a printer, or other suitable devices for displaying information to a user or operator. The computing system 202 may also be configured to allow interaction with a remote HMI and a remote display device via a network interface device 222.

[0053] System 200 can be implemented using one or more computing systems. Although this example depicts a single computing system 202 that implements all described features, it is intended that various features and functions be separated and implemented by multiple computing units that communicate with each other. The specific system architecture selected may depend on various factors.

[0054] The system 200 can implement a machine learning model 210 configured to analyze a raw source data set 216. For example, the CPU 206 and / or other circuits can implement the machine learning model 210. The raw source data set 216 can include raw or unprocessed sensor data that can represent an input data set for a machine learning system. The raw source data set 216 can include video, video clips, images, audio, text-based information, and raw or partially processed sensor data (e.g., a radar map of an object). In some embodiments, the machine learning model 210 can be a deep learning or neural network algorithm designed to perform a predetermined function. For example, a neural network algorithm can be configured to identify an event or object in a video clip based on audio data.

[0055] The computer system 200 may store a training data set 212 for the machine learning model 210. The training data set 212 may represent a previously constructed data set for training the machine learning model 210. For example, the training data set 212 according to the present disclosure may include a plurality of automatically collected ground truth measurements and associated data. The machine learning model 210 may use the training data set 212 to learn weighting factors associated with a neural network algorithm. The training data set 212 may include a source data set having a corresponding outcome or result that the machine learning model 210 attempts to replicate via a learning process.

[0056] The machine learning model 210 can operate in a learning mode using the training data set 212 as input. The machine learning model 210 can be executed in multiple iterations using data from the training data set 212. For each iteration, the machine learning model 210 can update the internal weighting factors based on the achieved results. For example, the machine learning model 210 can compare the output results (e.g., annotations) with those included in the training data set 212. Because the training data set 212 includes expected results, the machine learning model 210 can determine when the performance is acceptable. After the machine learning model 210 reaches a predetermined performance level (e.g., 100% consistent with the results associated with the training data set 212), the machine learning model 210 can be executed using data that is not in the training data set 212. The trained machine learning model 210 can be applied to a new data set to generate annotated data.

[0057] The machine learning model 210 may be configured to identify specific features in the raw source data 216. The raw source data 216 may include multiple instances or input data sets (e.g., video streams or segments including audio data) for which annotation results are desired. By way of example only, the machine learning model 210 may be configured to identify objects or events in a video segment based on audio data and annotate these events. The machine learning model 210 may be programmed to process the raw source data 216 to identify the presence of specific features. The machine learning model 210 may be configured to identify features in the raw source data 216 as predetermined features. The raw source data 216 may be derived from a variety of sources. For example, the raw source data 216 may be actual input data collected by a machine learning system. The raw source data 216 may be machine generated for the purpose of testing the system. As an example, the raw source data 216 may include raw video and / or audio data from a camera, audio data from a microphone, and the like.

[0058] In one example, the machine learning model 210 can process the raw source data 216 and output video and / or audio data that includes one or more indications of identified events. The machine learning model 210 can generate a confidence level or factor for each generated output. For example, a confidence value that exceeds a predetermined high confidence threshold can indicate that the machine learning model 210 is confident that the identified event (or feature) corresponds to a particular event. A confidence value that is less than a low confidence threshold can indicate that the machine learning model 210 has some uncertainty that a particular feature exists.

[0059] like Figure 3A and Figure 3B As generally illustrated, system 300 may include an image (e.g., video) capture device 302, an audio capture array 304, and a computing system 202. The system may receive video stream data associated with a data capture environment from image capture device 302. System 202 may be configured to perform video object detection to identify one or more objects in a corresponding image of the video stream data. System 202 may receive audio stream data corresponding to at least a portion of the video stream data from audio capture array 304. Audio capture array 304 may include one or more microphones 306 or other suitable audio capture devices. The systems and methods described herein may be configured to use output from at least a first machine learning model (e.g., such as machine learning model 210 or other suitable machine learning model configured to provide output including one or more object or event detection predictions) to label at least some objects of video stream data and / or audio stream data.

[0060] The system 202 may calculate (e.g., using at least one probability-based function or other suitable technique or function) at least one offset value for at least a portion of the audio stream data corresponding to at least one tagged object of the video stream data based on at least one data capture characteristic. The system 202 may synchronize at least a portion of the video stream data with a portion of the audio stream data corresponding to at least one tagged object of the video stream data using the at least one offset value. The at least one data capture characteristic may include one or more characteristics of at least one image capture device, one or more characteristics of at least one audio capture array, one or more characteristics corresponding to a position of at least one image capture device relative to at least one audio capture array, one or more characteristics corresponding to movement of an object in the video stream data, one or more other suitable data capture characteristics, or a combination thereof.

[0061] System 202 may use one or more tags and at least one offset value of the labeled objects of the video stream data to mark at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data. Each corresponding tag may include an event type, an event start indicator, and an event end indicator. System 202 may use at least some of the labeled portions of the audio stream data to generate training data. System 202 may use the training data to train a second machine learning model. System 202 may use the second machine learning model to detect one or more sounds associated with the audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable functions, such as those described herein with respect to Figures 4-11.

[0062] In some embodiments, Figure 3C As generally illustrated in FIG. 1 , the computing system 202 can be configured to tag audio data based on sensor data received from one or more sensors, such as the sensors described herein or any other suitable sensor or sensor combination. The system 202 can receive audio stream data associated with the data capture environment from an audio capture array 354 or any suitable audio capture device, such as one or more microphones 306 or other suitable audio capture devices. It should be understood that the audio capture array 354 may include features similar to those of the audio capture array 304, and may include any suitable number of audio capture devices. The system 202 can receive sensor data associated with the data capture environment from at least one sensor (e.g., such as sensor 352) that is asynchronous with respect to the audio capture array 354. The sensor 354 may include at least one of an induction coil, a radar sensor, a lidar sensor, a sonar sensor, an image capture device, any other suitable sensor, or a combination thereof. The audio capture array 354 may be remotely located from the sensor 354, located close to the sensor 354, or located in any suitable relationship with the sensor 354.

[0063] The system 202 may use output from at least a first machine learning model (such as the machine learning model 210 or other suitable machine learning model) to identify at least some events in the sensor data. The machine learning model 210 may be configured to provide an output including one or more event detection predictions based on the sensor data. The system 202 may synchronize at least a portion of the sensor data associated with a portion of the audio stream data corresponding to at least one event of the sensor data. The system 202 may use one or more tags extracted for the corresponding event of the sensor data value to mark at least a portion of the audio stream data corresponding to at least one event of the sensor data. Each corresponding tag may include an event type, an event start indicator, and an event end indicator. The system 202 may use at least some of the marked portions of the audio stream data to generate training data. The system 202 may use the training data to train a second machine learning model. The system 202 may use the second machine learning model to detect one or more sounds associated with the audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable functions, such as those described herein with respect to Figures 4-11.

[0064] Any of the systems described above and / or in greater detail below may implement the systems and methods of the present disclosure to perform SED.

[0065] The prototype network learns representations of different categories through support set examples. Using the embedding vectors of positive and negative samples in the support set, the prototype is formulated for each category as shown in equation (1) below:

[0066]

[0067] In Equation (1), Θ() represents a general differentiable feature extractor that, given a sample at input, produces its dimensional embedding as output. For a query sample x query , the squared Euclidean distance between the query embedding and each category prototype is calculated, and this distance is passed through a softmax function to obtain the category probability distribution of the sample. The network is optimized by stochastic gradient descent of the negative log-likelihood function.

[0068] In order to train a model according to the systems and methods of the present disclosure, an explicit background category is introduced during the training phase so that the model can learn representations of both target events and background sounds. Instead of creating synthetic background categories, background categories are created based on given data. Background categories are derived from X(t), Sampling from unlabeled fragments in . Given Make and The time windows outside the range are marked as background categories, and they are given new labels C train +1. Therefore, the new training labels will be Make

[0069] The systems and methods according to the present disclosure are also configured to construct a negative support set using unlabeled segments from the support set. neg is obtained by extracting the entire audio stream (e.g. Figure 4A The target sound events are constructed by randomly sampling segments from the audio stream 400 shown in FIG. 4 , including the query set), assuming that the target sound events are sparse. However, the assumption that the target sound events are sparse may not apply to all cases, and therefore the system and method of the present disclosure uses the support set (X test (t), ) (i.e., from the known part of the signal) to construct S neg Although this approach reduces the time required to construct S neg The available number of negative fragments, but the model is able to use the refined S neg to learn better negative prototypes without being contaminated by positive events randomly sampled from the entire audio sequence.

[0070] The systems and methods according to the present disclosure are also configured to cluster the negative support set during inference. In order to accommodate the diversity of the negative support set during inference, we cluster the negative support set S neg A k-means clustering algorithm is applied to the embedding vectors obtained from the samples of . In some examples, it can be assumed that the samples from the target sound event are more uniform than the negative samples. Figure 4B A two-dimensional t-SNE projection 402 is shown obtained from the embeddings of the positive and negative support sets of an audio sequence. The negative embeddings exhibit two distinct clusters (e.g., the negative set in clusters 0 and 1), while the positive embeddings are concentrated between the negative embeddings. By constructing only one prototype for the negative class, query samples from this sequence may be easily misclassified due to the proximity of the resulting prototypes. As an example, k-means clustering is performed on the S in each of the corresponding clusters 0 and 1. neg The embedding of the samples is performed with example values ​​of k ranging from 2 to 10. The value of k that yields the highest Silhouette coefficient is then selected (e.g., 2 for this example). Based on this observation, one positive prototype 404 and two negative prototypes 406, 408 are obtained for each audio stream. Therefore, equation (1) can be modified as follows:

[0071]

[0072] where Γ j is the set of sample indices assigned to cluster j.

[0073] Additionally, instead of binary classification, the inference problem can be formulated as a K-sample ternary classification problem with one class for positive labels and two classes for negative labels (e.g., performed during inference rather than during training).

[0074] Figure 4C Also illustrated is an example clustering of a support set in accordance with the principles of the present disclosure. As shown at 420, positive samples taken from the support set (eg, from a known portion of the signal) are is embedded / encoded (eg, by an encoder or encoding circuit 422) into the In contrast, negative samples from the support set Embedded / encoded into negative embedding middle.

[0075] As shown at 424, the positive embeddings are aggregated (eg, by an aggregator or aggregator circuit 426) to obtain For example, the embeddings of positive samples from the same class may be averaged. Conversely, as shown at 428, negative embeddings are aggregated (e.g., by an aggregator or aggregator circuit 430, which may be the same as or different from aggregator 426) to obtain each set of negative samples. Typically, the embeddings of negative samples are averaged into a single prototype (i.e., as a group). In contrast, according to the principles of the present disclosure, negative embeddings are clustered into K different groups, and K different negative prototypes are obtained.

[0076] The system and method according to the present disclosure are also configured to perform automatic threshold identification. The category probability distribution of the query sample is obtained via softmax on the opposite of the Euclidean distance between the query embedding and the category prototype. If the predicted probability of the positive category exceeds the threshold, the query is classified as a target sound event. In some examples, the predicted probability value is set to 0.5 based on experience. In the system and method according to the present disclosure, a computational method is used to identify a threshold customized for each audio sequence. For example, a leave-one-sample cross-validation method is used to estimate the probability distribution for all samples in the support set. The probability of the sample being classified as a target event is calculated for each sample in the support set.

[0077] It can be assumed that from S neg The probability obtained will have a long tail on the right side of the distribution. pos The probability of will have a long tail on the left side of the distribution. Therefore, the two tail probabilities from these two distributions can be considered as candidates for evaluation thresholds to reduce the false alarm rate. In one example, from S negThe 95th percentile of the obtained probabilities are considered candidates because the negative samples are larger in number. pos The 25th percentile of the probability of is considered as another candidate. The choice of the 25th percentile over the 95th percentile is motivated by the limited number of positive examples in the support set.

[0078] Figure 4D Shown are examples of probability distributions obtained from positive support set samples 432 and negative support set samples 434 and their corresponding candidate thresholds (eg, predicted probability thresholds) 436 and 438. The candidate threshold with the highest F-measure on the support set is taken as the evaluation threshold for the particular audio.

[0079] Figure 4E Also illustrated is an example probability calculation according to the principles of the present disclosure. For example, as shown at 440, a positive sample is obtained. Each of the Probability value The distribution of Figure 4D As shown by the solid line 432 in . On the contrary, as shown at 442, a negative sample is obtained Each of the Probability value The distribution of Figure 4D As shown by the dashed line 434 in FIG. As shown at 444, based on and The distribution of each category is obtained The threshold separates the negative samples from the positive samples.

[0080] Figure 4F FIG. 1 illustrates a method for determining a sample x according to the principles of the present disclosure. q Example method 450 of the category (e.g., as described by Figure 1 - Figure 3 and Figure 5-Figure 11 At 452, the encoded sample or the embedded sample is obtained. q At 454, the sample x is calculated q The probability of belonging to a given class or group (e.g., based on a positive prototype and negative prototype At 456, the calculated probability is compared with the threshold value of each category. Therefore, at 456, a comparison is made about sample x q Determination of whether it belongs to a given category {1, 2, ..., K} or does not belong to any category.

[0081] Figure 5-Figure 11 Example systems and devices are depicted in which systems and methods consistent with the present disclosure may be implemented. Figure 5 A schematic diagram of the interaction between a computer-controlled machine 500 and a control system 502 is depicted. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a condition of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of the sensor 506 include video, radar, lidar, ultrasonic, and motion sensors. In some embodiments, the sensor 506 is an optical sensor configured to sense an optical image of an environment proximate to the computer-controlled machine 500.

[0082] The control system 502 is configured to receive sensor signals 508 from the computer controlled machine 500. As explained below, the control system 502 may also be configured to calculate actuator control commands 510 depending on the sensor signals and transmit the actuator control commands 510 to the actuators 504 of the computer controlled machine 500.

[0083] like Figure 5 As shown in , the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive the sensor signal 508 from the sensor 506 and transform the sensor signal 508 into an input signal x. In an alternative embodiment, the sensor signal 508 is directly received as the input signal x without the receiving unit 512. Each input signal x can be a part of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signal x can include data corresponding to the image recorded by the sensor 506.

[0084] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine (ML) learning algorithm, such as the neural network described above. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by a non-volatile storage device 516. The classifier 514 is configured to determine an output signal y from the input signal x. Each output signal y includes information that assigns one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control command 510. In some embodiments, actuator 504 is configured to actuate computer-controlled machine 500 directly based on output signal y.

[0085] When the actuator 504 receives the actuator control command 510, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to transform the actuator control command 510 into a second actuator control command for controlling the actuator 504. In one or more embodiments, the actuator control command 510 may be used to control a display instead of or in addition to an actuator.

[0086] In some embodiments, the control system 502 includes the sensor 506 instead of or in addition to the computer-controlled machine 500 including the sensor 506. The control system 502 may also include the actuator 504 instead of or in addition to the computer-controlled machine 500 including the actuator 504.

[0087] like Figure 5 As shown in , the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. The classifier 514 (e.g., ML algorithm) of one or more embodiments may be implemented by the control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522.

[0088] The non-volatile storage 516 may include one or more persistent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, a cloud storage device, or any other device capable of persistently storing information. The processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including a high-performance core, a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates signals (analog or digital) based on computer executable instructions residing in the memory 522. The memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0089] The processor 520 may be configured to read into the memory 522 and execute computer executable instructions residing in the non-volatile storage 516 and embodying one or more ML algorithms and / or methods of one or more embodiments. The non-volatile storage 516 may include one or more operating systems and applications. The non-volatile storage 516 may store computer programs created using various programming languages ​​and / or technologies to compile and / or interpret computer programs, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, alone or in combination.

[0090] When executed by the processor 520, the computer executable instructions of the non-volatile storage device 516 may cause the control system 502 to implement one or more of the ML algorithms and / or methods as disclosed herein. The non-volatile storage device 516 may also include ML data (including data parameters) that support the functions, features, and processes of one or more embodiments described herein.

[0091] The program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for enabling a processor to implement aspects of one or more embodiments. Computer-readable storage media that are inherently non-transitory may include volatile and non-volatile and removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media may also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage device, cassette, tape, disk storage device or other magnetic storage device, or any other medium that can be used to store the desired information and can be read by a computer. Computer-readable program instructions can be downloaded from a computer-readable storage medium to a computer, another type of programmable data processing device, or another device via a network, or to an external computer or external storage device.

[0092] Computer-readable program instructions stored in a computer-readable medium can be used to guide a computer, other types of programmable data processing devices, or other devices to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture, including instructions that implement the functions, actions, and / or operations specified in the flowchart or diagram. In certain alternative embodiments, the functions, actions, and / or operations specified in the flowchart and diagram can be reordered, processed serially, and / or processed in parallel in accordance with one or more embodiments. In addition, any of the flowchart and / or diagram may include more or fewer nodes or boxes than those illustrated in accordance with one or more embodiments.

[0093] A process, method or algorithm may be embodied in whole or in part using suitable hardware components (such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a state machine, a controller or other hardware components or devices) or a combination of hardware, software and firmware components.

[0094] Figure 6A schematic diagram of a control system 502 configured to control a vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot, is depicted. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, lidar sensors, and / or position sensors (e.g., GPS). To implement systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 is configured to detect sound events near the vehicle 600. As an example, the control system 502 may be configured to control an autonomous function (such as navigation) of the vehicle 600 in response to a detected sound event.

[0095] One or more of the one or more specific sensors may be integrated into the vehicle 600. Alternatively or in addition to the one or more specific sensors identified above, the sensor 506 may include a software module that, when executed, is configured to determine the state of the actuator 504. One non-limiting example of a software module includes a weather information software module that is configured to determine the present or future state of the weather proximate to the vehicle 600 or other location.

[0096] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect objects near the vehicle 600 depending on the input signal x. In such an embodiment, the output signal y may include information characterizing the objects near the vehicle 600. The actuator control command 510 may be determined based on the information. The actuator control command 510 may be used to avoid collision with the detected object.

[0097] In some embodiments, the vehicle 600 is at least partially autonomous and the actuator 504 may be embodied in the brakes, propulsion system, engine, drive train, or steering of the vehicle 600. Actuator control commands 510 may be determined such that the actuator 504 is controlled such that the vehicle 600 avoids colliding with a detected object. The detected objects may also be classified according to what the classifier 514 thinks they are most likely to be, such as a pedestrian or a tree. Depending on the classification, the actuator control commands 510 may be determined. In scenarios where a hostile attack may occur, the systems described above may also be trained to better detect objects or identify changes in lighting conditions or angles of sensors or cameras on the vehicle 600.

[0098] In some embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In such embodiments, the actuator control commands 510 may be determined such that a propulsion unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot may avoid colliding with the identified object.

[0099] In some embodiments, vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such embodiments, vehicle 600 may use an optical sensor as sensor 506 to determine the state of plants in an environment close to vehicle 600. Actuator 504 may be a nozzle configured to spray a chemical. Depending on the identified species and / or the identified state of the plant, actuator control commands 510 may be determined to cause actuator 504 to spray the plant with an appropriate amount of an appropriate chemical.

[0100] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor 506 may detect the state of the laundry inside the washing machine. The actuator control command 510 may be determined based on the detected state of the laundry.

[0101] Figure 7 A schematic diagram of a control system 502 of a system 700 (e.g., a manufacturing machine) configured to control a manufacturing system 702 (e.g., a portion of a production line), such as a punch cutter, a cutter, or a gun drill, is depicted. The control system 502 may be configured to control an actuator 504 configured to control the system 700 (e.g., a manufacturing machine).

[0102] The sensor 506 of the system 700 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more properties of the manufactured product 704. To implement the systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 is configured to detect sound events related to the operation of the manufacturing machine in the manufacturing facility. The classifier 514 may be configured to determine the state of the manufactured product 704 from one or more captured properties (such as the detected sound events). The actuator 504 may be configured to control the system 700 (e.g., a manufacturing machine) for subsequent manufacturing steps of the manufactured product 704 depending on the determined state of the manufactured product 704. The actuator 504 may be configured to control the function of the system 700 (e.g., a manufacturing machine) on the subsequent manufactured product 706 of the system 700 (e.g., a manufacturing machine) depending on the determined state of the manufactured product 704.

[0103] Figure 8 A schematic diagram of a control system 502 configured to control a power tool 800 (such as a drill or driver) having an at least partially autonomous mode is depicted. The control system 502 may be configured to control an actuator 504 that is configured to control the power tool 800.

[0104] The sensor 506 of the power tool 800 may be an optical sensor configured to capture one or more properties of the work surface 802 and / or the fastener 804 being driven into the work surface 802. To implement the systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 is configured to detect sound events associated with the operation of the power tool 800. The classifier 514 may be configured to determine the state of the work surface 802 and / or the state of the fastener 804 relative to the work surface 802 from one or more captured properties (such as detected sound events). The state may be that the fastener 804 is flush with the work surface 802. Alternatively, the state may be the hardness of the work surface 802. The actuator 504 may be configured to control the power tool 800 so that the driving function of the power tool 800 is adjusted depending on the determined state of the fastener 804 relative to the work surface 802 or one or more captured properties of the work surface 802. For example, if the state of fastener 804 is flush relative to working surface 802, actuator 504 may no longer continue the driving function. As another non-limiting example, actuator 504 may apply additional or less torque depending on the hardness of working surface 802.

[0105] Fig. 9A schematic diagram of a control system 502 configured to control an automated personal assistant 900 is depicted. The control system 502 may be configured to control an actuator 504 configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0106] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive a video image of a gesture 904 of user 902. The audio sensor may be configured to receive a voice command of user 902. To implement the systems and methods according to the present disclosure, control system 502 is configured to detect sound events near automated personal assistant 900, such as sound events associated with the operation of an appliance controlled by automated personal assistant 900.

[0107] The control system 502 of the automated personal assistant 900 can be configured to determine an actuator control command 510 for which the control system 502 is configured. The control system 502 can be configured to determine the actuator control command 510 based on the sensor signal 508 of the sensor 506. The automated personal assistant 900 is configured to transmit the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 can be configured to execute a gesture recognition algorithm to identify a gesture 904 made by the user 902, determine the actuator control command 510, and transmit the actuator control command 510 to the actuator 504. The classifier 514 can be configured to retrieve information from a non-volatile storage device in response to the gesture 904 and output the retrieved information in a form suitable for receipt by the user 902.

[0108] Fig.10 A schematic diagram of a control system 502 configured to control a monitoring system 1000 is depicted. The monitoring system 1000 may be configured to physically control access through a door 1002. A sensor 506 may be configured to detect scenes relevant to a decision as to whether access is authorized. The sensor 506 may be an optical sensor configured to generate and transmit image and / or video data. The control system 502 may use such data to detect the face of a person. To implement systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 may be configured to detect sound events near the monitoring system 1000, such as sound events caused by a person attempting to enter the door 1002. As an example, the control system 502 may be configured to block or allow entry through the door 1002 in response to a detected sound event.

[0109] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to interpret the image and / or video data by matching the identities of known persons stored in the non-volatile storage device 516 to determine the identity of the person. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock the door 1002 in response to the actuator control command 510. In some embodiments, non-physical logical access control may also be performed.

[0110] The monitoring system 1000 may also be a surveillance system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene under surveillance, and the control system 502 is configured to control the display 1004. The classifier 514 is configured to determine a classification of the scene, for example, whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to: transmit an actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight an object that the classifier 514 considers suspicious. Using embodiments of the disclosed system, the surveillance system can predict the appearance of an object at a specific time in the future.

[0111] Fig.11 A schematic diagram of a control system 502 is depicted, the control system 502 being configured to control an imaging system 1100, such as an MRI device, an x-ray imaging device, or an ultrasound device. For example, the sensor 506 may be an imaging sensor. To implement the systems and methods according to the present disclosure, the sensor 506 may include at least one audio sensor (e.g., a microphone), and the control system 502 is configured to detect sound events near the imaging system 1100, such as sounds caused by the operation of the imaging system 1100, a user of the imaging system 1100, etc. As an example, the control system 502 may be configured to pause, interrupt, continue, etc. the imaging process in response to the detected sound events. The classifier 514 may be configured to determine a classification of all or part of the sensed image. The classifier 514 may be configured to determine or select an actuator control command 510 in response to a classification obtained by a trained neural network. For example, the classifier 514 may interpret an area of ​​the sensed image as a potential abnormality. In this case, the actuator control command 510 may be determined or selected to cause the display 1102 to display the imaging and highlight the potential abnormal area.

[0112] Although exemplary embodiments are described above, it is not intended that these embodiments describe all possible forms covered by the claims. The words used in this specification are descriptive words, not restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As previously described, the features of various embodiments can be combined to form other embodiments of the present invention, and these other embodiments may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations with respect to one or more desired characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. As such, to the extent that any embodiment is described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method for training a prototype network for sound event detection, the method comprising: receiving samples of an audio signal, wherein the samples include positive samples corresponding to sound events and negative samples not corresponding to sound events; Based on the positive samples, corresponding positive prototypes of multiple sound event categories are determined; Based on the negative samples, determining corresponding negative prototypes of the corresponding negative sample groups, wherein each negative prototype corresponds to a combination of multiple negative samples; and Based on a comparison between (i) the first sample and (ii) the corresponding positive prototype and each negative prototype, an output signal is generated indicating whether the first sample belongs to one of a plurality of sound event categories.

2. The method according to claim 1, further comprising: Obtain the first embedding of the positive sample and the second embedding of the negative sample; determining a corresponding positive prototype based on the first embedding; and A corresponding negative prototype is determined based on the second embedding. The method according to claim 1 , wherein determining the corresponding negative prototype comprises determining a negative prototype for each of the corresponding negative sample groups.

4. The method according to claim 1, further comprising: determining, based on the comparing, at least one probability that the first sample corresponds to one of a plurality of sound event categories; and An output is generated based on the at least one probability.

5. The method according to claim 4, further comprising: determining at least one probability distribution; and An output is generated based on the at least one probability distribution.

6. The method according to claim 5, further comprising: determining at least one threshold value based on the at least one probability distribution; and An output is generated based on a comparison between the at least one probability and the at least one threshold value.

7. A method according to claim 6, wherein determining the at least one threshold value includes determining the at least one threshold value based on (i) a first probability distribution of probabilities that positive samples belong to corresponding categories among multiple sound event categories and (ii) a second distribution of probabilities that negative samples belong to corresponding categories among multiple sound event categories.

8. A computing device configured to train a prototype network for sound event detection, the computing device comprising a processing device configured to execute instructions stored in a memory to: receiving samples of an audio signal, wherein the samples include positive samples corresponding to sound events and negative samples not corresponding to sound events; Based on the positive samples, corresponding positive prototypes of multiple sound event categories are determined; Based on the negative samples, determining corresponding negative prototypes of the corresponding negative sample groups, wherein each negative prototype corresponds to a combination of multiple negative samples; and Based on a comparison between (i) the first sample and (ii) the corresponding positive prototype and each negative prototype, an output signal is generated indicating whether the first sample belongs to one of a plurality of sound event categories.

9. The computing device of claim 8, wherein the processing device is further configured to execute instructions to: Obtain the first embedding of the positive sample and the second embedding of the negative sample; determining a corresponding positive prototype based on the first embedding; and A corresponding negative prototype is determined based on the second embedding. 10 . The computing device of claim 8 , wherein determining the corresponding negative prototype comprises determining a negative sample for each of the corresponding groups of negative samples.

11. The computing device of claim 8, wherein the processing device is further configured to execute instructions to: Based on the comparing, determining at least one probability that the first sample corresponds to one of a plurality of sound event categories; and An output is generated based on the at least one probability.

12. The computing device of claim 11, wherein the processing device is further configured to execute instructions to: determining at least one probability distribution; and An output is generated based on the at least one probability distribution.

13. The computing device of claim 12, wherein the processing device is further configured to execute instructions to: determining at least one threshold value based on the at least one probability distribution; and An output is generated based on a comparison between the at least one probability and the at least one threshold value.

14. A computing device according to claim 13, wherein determining the at least one threshold value includes determining the at least one threshold value based on (i) a first probability distribution of probabilities that positive samples belong to corresponding categories among multiple sound event categories and (ii) a second distribution of probabilities that negative samples belong to corresponding categories among multiple sound event categories.

15. A computer-controlled machine comprising: at least one sensor configured to generate an audio signal; a control system configured to receive a first sample of an audio signal, and generating an output signal indicating whether a first sample of the audio signal belongs to one of a plurality of sound event categories based on a comparison between (i) the first sample and (ii) a corresponding positive prototype for each of a plurality of sound event categories and a corresponding negative prototype for each of a plurality of sets of negative prototypes; and An actuator is configured to control the operation of the computer-controlled machine in response to an output of the control system.

16. The computer controlled machine of claim 15, further comprising a memory storing the corresponding positive prototype and the corresponding negative prototype, wherein: The corresponding positive prototype corresponds to multiple positive samples; and Each of the corresponding negative prototypes corresponds to a combination of multiple negative samples.

17. The computer-controlled machine of claim 15, wherein generating an output comprises calculating a probability that the first sample belongs to a first one of a plurality of sound event classes or a first one of a plurality of groups.

18. The computer controlled machine of claim 17, wherein generating an output comprises comparing the probability to at least one threshold value and generating an output based on the comparison.

19. The computer-controlled machine of claim 18, wherein the at least one threshold comprises a plurality of thresholds corresponding to respective ones of a plurality of sound event categories.

20. The computer controlled machine of claim 15, wherein the computer controlled machine comprises an autonomous robot.

Images