Sound event detection based on modeling of sequences of event subparts

Abstract

The present disclosure relates to sound event detection based on modeling of sequences of event sub-portions. A technique for sound event detection is provided. A method for implementing the technique according to an embodiment includes extracting sound features from a received audio signal. The sound features may include, for example, one or more short-term Fourier transform frames or other spectral energy characteristics of the audio signal. The method also includes applying a trained classifier to the extracted sound features to identify and mark the sound event sub-portions of the audio signal and generate scores associated with these sub-portions. The method also includes performing sequence decoding of the sound event sub-portions and associated scores to detect target sound events of interest based on the scores and time ordering of the event sub-portions. The classifier is trained on the sound event sub-portions generated by an unsupervised subspace clustering technique applied to training data including the target sound events.

Description

Technical Field

[0001] Embodiments herein generally relate to sound event detection based on modeling sequences of event sub-parts. Background Art

[0002] Sound event detection plays an increasingly important role in applications related to the Internet of Things (IoT), smart home technology, and digital surveillance systems. It is often useful for a detection / recognition system to be able to respond to selected sound events of interest (e.g., shooting, breaking glass, crying babies, or other sounds that indicate situations that may require the attention of interested parties). Existing recognition systems generally need to be trained on sound training data sequences that contain the event types of interest. These events need to be labeled and their locations in the data sequence must be provided, which is often a time-consuming and expensive task. In addition, these existing systems typically cannot provide the desired level of accuracy (e.g., in terms of detection rate and false alarm rate) required for some of these applications. Summary of the invention

[0003] According to one aspect of the present disclosure, a processor-implemented method for detecting sound events is provided, the method comprising: extracting one or more sound features from an audio signal by a processor-based system; applying a trained classifier to the extracted one or more sound features by the processor-based system to identify sound event subportions of the audio signal and generate scores associated with the subportions; and performing sequential decoding of the sound event subportions and associated scores by the processor-based system to detect sound events.

[0004] According to another aspect of the present disclosure, a system for sound event detection is provided, the system comprising: a feature extraction circuit that extracts sound features from an audio signal; a classifier circuit that identifies sound event subportions of the audio signal based on the extracted sound features and generates scores associated with the subportions; and a sequence decoder circuit that performs sequence decoding of the sound event subportions and associated scores to detect sound events.

[0005] According to another aspect of the present disclosure, at least one non-transitory computer-readable storage medium is provided, having instructions encoded thereon, which, when executed by one or more processors, result in operations for detecting sound events, the operations comprising: extracting one or more sound features from an audio signal; applying a trained classifier to the extracted one or more sound features to identify sound event subportions of the audio signal and generate scores associated with the subportions; and performing sequential decoding of the sound event subportions and associated scores to detect sound events. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The patent or application file contains at least one drawing executed in color. Copies of the patent or patent application publication and the color drawing will be provided by the Office upon request and payment of the necessary fee.

[0007] Features and advantages of embodiments of the claimed subject matter will become apparent as the following detailed description proceeds, with reference to the drawings, wherein like reference numerals depict similar parts.

[0008] Figure 1 is a top-level block diagram of a sound event detection system configured in accordance with certain embodiments of the present disclosure.

[0009] Figure 2 is a more detailed block diagram of a clustering circuit configured in accordance with certain embodiments of the present disclosure.

[0010] Figure 3 Acoustic waveforms, sound signatures, and indicia are shown according to certain embodiments of the present disclosure.

[0011] Figure 4 The input and output of a trained classifier configured according to certain embodiments of the present disclosure are shown.

[0012] Figure 5 A graphical comparison of event markers and cluster markers according to certain embodiments of the present disclosure is provided.

[0013] Figure 6 2 shows the states of a sequence decoder circuit configured in accordance with certain embodiments of the present disclosure.

[0014] Figure 7 is a flow chart illustrating a method for sound event detection according to certain embodiments of the present disclosure.

[0015] Figure 8 is a block diagram schematically illustrating a computing platform configured to perform sound event detection according to certain embodiments of the present disclosure.

[0016] Although the following detailed description will proceed with reference to illustrative embodiments, many alternatives, modifications, and variations thereof will become apparent in light of this disclosure. DETAILED DESCRIPTION

[0017] In general, the present disclosure provides a technique for detecting sound events based on modeling a sequence of sub-parts of an event of interest. Sound events generally include multiple sequential sub-parts, each of which can have different identifiable characteristics. It will be understood from the present disclosure that the number of sub-parts varies depending on the situation, but generally includes more than two different identifiable sub-parts. For example, the sound of breaking glass can include a sharp impact sound associated with the initial break, then a crackling sound that spreads through the glass as the breakage occurs, and then a third different sound as the fragments break and fall to the surrounding area. According to the present disclosure, any number of other such multi-part sound events will be obvious. The technology generally includes a training phase and an inference or application phase, and can be implemented in, for example, a computing system or a software product that can be controlled or otherwise executed by such a system (although other embodiments will be obvious). The system or product is configured to detect one or more interesting sound events (e.g., target sound events) that may appear in a given overall audio signal. Note that the overall audio signal itself may include one or more actual sound events, each of which includes multiple sub-parts.

[0018] According to one embodiment, a classifier that recognizes each sound sub-portion of an event of interest and scores these sound sub-portions is trained. The classifier can be, for example, a deep neural network. The trained classifier is then applied to the sound features extracted from the signal provided for analysis, and a sequence of sub-portion scores is inferred or otherwise generated. As previously described, the sound features may include a short-term Fourier transform or any other spectral energy characteristic of the audio signal. The sequence decoder then performs sequence decoding of the sound event sub-portions and the associated scores to identify the correct time sequence of these sub-portions, based on which event detection can be determined. Since the classifier effectively models the different characteristics of multiple sub-portions of a given sound event instead of trying to identify the event as a whole, event detection performance is improved (e.g., fewer errors or incorrect detections). In addition, since sequence decoding imposes a temporal order on the sub-portions, the detection performance is more reliable and robust.

[0019] In some such embodiments, a classifier is trained on sub-portions of sound events generated by applying an unsupervised subspace clustering technique to training data including target sound events, as will be explained in more detail below. The use of the unsupervised subspace clustering technique provides an additional benefit in that the training data does not need to be annotated with the start and stop points of the sound events, as will be explained below. This benefit allows the generation of training data to be more efficient and cost-effective.

[0020] It will be appreciated that the techniques described herein allow for improved sound event detection based on modeling event sub-portions with a finer level of granularity, which increases the discrimination of the classifier (compared to existing systems that attempt to recognize events as a whole). The disclosed techniques can be implemented on a variety of platforms including laptops, tablet computers, smart phones, workstations, embedded systems or devices. These techniques are particularly suitable for use on digital signal processors (DSPs) or other dedicated ultra-low power hardware. These techniques can also be implemented in hardware or software or a combination thereof.

[0021] Figure 1 1 is a top-level block diagram of a sound event detection system 100 configured according to certain embodiments of the present disclosure. The system 100 is shown to include a feature extraction circuit 109, a clustering circuit 102, a classifier training circuit 104, a trained classifier 106, and a sequence decoder circuit 108. At a high level, the sound event detection system 100 receives training data 110 containing target sound events and performs classifier training on sub-portions 120 of these events using unsupervised subspace clustering, as explained in more detail below. The trained classifier 106 can then be used in an operating mode to detect sound events of interest (e.g., target events) in a received audio signal 130 based on the clustered sub-portions. The feature extraction circuit 109, the clustering circuit 102, and the classifier training circuit 104 are used in the training mode of the sound event detection system 100. The feature extraction circuit 109, the trained classifier 106, and the sequence decoder circuit 108 are used in the operating mode of the sound event detection system 100.

[0022] The feature extraction circuit 109 is configured to extract sound features from a provided audio signal (e.g., training data or training signal 110 containing a target sound event, and an audio signal 130 on which event detection will be performed). In some embodiments, the extracted sound features include short-term Fourier transform (STFT) frames of the audio signal. In some embodiments, according to the present disclosure, the sound features may include filter bank features, Mel-frequency cepstral coefficients (MFCCs), spectral shape descriptors, or other known sound feature metrics. Figure 3 An example sound feature 160 is shown extracted from a signal containing target sound event data 220. In some embodiments, the STFT interval or frame length 306 may be approximately 50 milliseconds (ms).

[0023] As can be seen in the sound signature 160, the characteristics of the features at the beginning of the event are significantly different from those in the middle and end of the event. Therefore, by increasing the granularity of modeling down to the sub-part level, the classifier can be trained to better fit the data.

[0024] The clustering circuit 102 is configured to perform unsupervised subspace clustering of the extracted features to generate a sound event sub-portion 120, which is used by the classifier training circuit 104 to train a classifier for identifying a target sound event. Figure 2 and Figure 3 The operation of clustering circuit 102 and classifier training circuit 104 is described in more detail.

[0025] In an operational mode, feature extraction circuit 109 is configured to extract features from audio signal 130 in the same manner as described above for extracting features from training data 110. Audio signal 130 is typically (although not necessarily) an unknown signal, and may or may not contain target sound events (e.g., events of interest). Trained classifier 106 is configured to identify sound event sub-portions of the audio signal based on the extracted sound features, and generate scores 140 associated with these sub-portions.

[0026] In some embodiments, the classifier circuit is implemented as a deep neural network (DNN), a support vector machine, a decision tree classifier, a k-nearest neighbor classifier, a random forest classifier, a gradient boosting classifier, a logistic regression-based classifier, or using other known techniques in accordance with the present disclosure. Figure 4 A deep neural network implementation of a trained classifier 106 is shown. An input layer 410 receives the extracted features 170 of an audio signal. These sound features are operated on by nodes of each layer, including a hidden layer 420 and a final output layer 430. In this example, each output layer node is associated with one of the target event outputs 402 corresponding to a sub-portion of a target sound event or a non-target event output 404 that forms a rejection model. These output nodes generate tags and scores for the classification output from the node. These tags identify the sound event sub-portion. The score can be a relative value on a normalized scale indicating the confidence of the classification output.

[0027] This is Figure 5 It is shown in Figure 5 An audio signal 130 , extracted sound features 170 , and outputs 504 , 506 of the DNN including scores and labels are shown. Figure 5 A graphical comparison of event labels 504 and cluster labels 506 is provided. Graph 504 shows the output of a DNN classifier trained to recognize the entire sound event 510 labeled "Output 1", with scores on the vertical axis. It can be seen that such a classifier trained on event level labels is more sensitive to the middle portion of the event, and therefore may confuse the target event with other sounds that have features similar to the middle portion, resulting in more false detections.

[0028] Graph 506 shows the output of a DNN classifier trained to recognize sub-portions of a sound event according to certain embodiments of the present disclosure. Sub-portions 520, 530, 540 are labeled "output 1", "output 2", and "output 3", respectively, with scores on the vertical axis. For such a classifier, it is required to detect all three different sub-portions in the correct order to improve the false detection rate.

[0029] The sequence decoder circuit 108 is configured to perform sequence decoding of the marked sound event sub-portions and associated scores generated by the classifier 106 to detect sound events. The sequence decoding is based on the time ordering of the sound event sub-portions and the comparison of the associated scores with the selected threshold score value. The decoder is sensitive to the normal sequence of sub-portion states. Differently, when the correct sequence of sub-portions with sufficiently high scores is detected, it is considered that a sound event has been detected. In some embodiments, the sequence decoder circuit adopts one or more of a weighted finite state transducer, a recurrent neural network (RNN), or a long short-term memory RNN.

[0030] Figure 6 A state diagram representation of a sequence decoder circuit 108 configured according to certain embodiments of the present disclosure is shown. The decoder can be in a non-target event state 602 or a target event state 604. In this example, the decoder remains in state zero (S0) when the classifier generates non-target event outputs 4, 5, 6, 7. When a target event output 1 is generated, the sequence decoder shifts to state 1 (S1) and remains in state 1 before a different event output is generated. In response to the generation of target event output 2, the sequence decoder shifts from state 1 (S1) to state 2 (S2) and remains in state 2 before a different event output is generated. Subsequently, in response to the generation of target event output 3, the sequence decoder shifts from state 2 (S2) to state 3 (S3). When the sequence decoder circuit traverses state 1 to state 3 in the correct order, a sound event with a satisfactory classifier output score can be detected. In some embodiments, event detection can also require the sequence decoder circuit to maintain a selected duration in each of the three states. For example, the duration of each sub-portion of a given sound event can be known or expected to be within a specific time duration value range.

[0031] Now go to Figure 2, a more detailed block diagram of the clustering circuit 102 configured according to certain embodiments of the present disclosure is shown. The clustering circuit 102 is used in a training mode of the sound event detection system 100 and operates on sound features 160 extracted from training data 110 containing sound events. At least some of the training data will contain target data 220 (e.g., sound events of interest to be detected by the classifier). In some embodiments, the training data may also contain non-target data 230, which can improve the training of the classifier to distinguish between target sound events and non-target sound events. If non-target data 230 is provided, a classification process 202 is performed to provide some appropriate type of labeling to distinguish between the target data 220 and the non-target data 230. This process can be performed manually or in an automatic manner, or the classification information can be included as part of a database of sound recordings provided by, for example, a third party.

[0032] The sound features 160 from the target data and optionally from the non-target data are clustered using unsupervised subspace clustering at operations 208 and 210, respectively. In some embodiments, the unsupervised subspace clustering is implemented as k-means clustering, Gaussian mixture model clustering, DenStream clustering, or by other known techniques according to the present disclosure.

[0033] One benefit of using unsupervised subspace clustering technology is that the silent part of data is clustered into a cluster different from the event subpart clustering. This allows the start and end points of the event subparts in the automatic detection training data to be detected, because these event subparts are generally only the non-silent parts of the training data. On the contrary, the detection system based on the mark of the whole event rather than the subpart clustering generally needs the training data through annotation to mark the start and end points of the event, which can be a time-consuming and expensive process. However, in some embodiments, some parts of the training data can include reference annotations indicating the position of the subpart of the target sound event, which can improve the efficiency of the subspace clustering technology and contribute to the initialization of the subspace clustering technology. The generation of these reference annotations is shown in operation 206. In these embodiments, when the reference annotations or other prior knowledge of the relevant training data are available, semi-supervised subspace clustering technology can be adopted. Semi-supervised (or semi-automatic) subspace clustering technology can include semi-supervised variants of constrained k-means clustering, seeding k-means clustering, and Gaussian mixture model clustering and DenStream clustering.

[0034] In some other embodiments, reference annotations for all or most sub-portions of the target sound event are available, thereby eliminating the need to perform a subspace clustering operation.

[0035] At operation 212 of the clustering circuit, the generated clusters undergo post-processing operations to reduce noise (e.g., outliers or ambiguous points that may or may not be associated with a cluster). In some embodiments, this may be achieved by median filtering and / or comparison of clustering results from adjacent frames.

[0036] At operation 214 of the clustering circuit, a label for each cluster (sub-portion) is generated, for example, by associating the sound features with a cluster index. Figure 3 It is shown in Figure 3 A cluster label 302 associated with each sub-portion of the target sound event is shown. In this example, an event 304 labeled "glass breaking" is shown to include three consecutive sub-portions labeled "Cluster 1," "Cluster 2," and "Cluster 3."

[0037] Labeled sound event subportions 302 and associated sound features 160 of a large number of training sound events 110 are collected and provided to the classifier training circuit 104 to train a classifier that distinguishes event subportions using known techniques according to the present disclosure.

[0038] method

[0039] Figure 7 is a flow chart illustrating an example method 700 for sound event detection according to certain embodiments of the present disclosure. It can be seen that the example method includes multiple stages and sub-processes, the order of which may be different in different embodiments. However, when considered together, these stages and sub-processes form a process for sound event detection according to certain embodiments disclosed herein. These embodiments may, for example, use the method described above. Figure 1 and Figure 2 However, other system architectures may be used in other embodiments, as will be apparent from this disclosure. Figure 7 The association of various functions shown in the figures to the specific components shown in other figures is not intended to imply any structural and / or usage limitations. On the contrary, other embodiments may include, for example, the degree of changing the set, wherein multiple functions are effectively performed by one system. For example, in an alternative embodiment, a single module with decoupled submodules can be used to perform all functions of method 700. Therefore, depending on the implementation granularity, other embodiments may have more or fewer modules and / or submodules. In other embodiments, the depicted method may be implemented as a computer program product including one or more non-transitory machine-readable media, which causes the method to be performed when executed by one or more processors. In light of the present disclosure, multiple variations and alternative configurations will be apparent.

[0040] like Figure 7 As shown, in one embodiment, the method 700 for sound event detection begins by extracting sound features from a received audio signal at operation 710. In some embodiments, the extracted sound features may include a short-term Fourier transform of the audio signal. In some embodiments, the audio signal may be provided by a microphone or obtained from a database of pre-recorded audio signals.

[0041] Next, in operation 720, a trained classifier is applied to the extracted sound features to identify and mark the sound event sub-portions of the audio signal and generate scores associated with these sub-portions. These sub-portions can represent, for example, the beginning, middle, or tail end of a sound event such as glass breaking, shooting, or door closing. In some embodiments, the classifier is implemented as a deep neural network, a support vector machine, a decision tree classifier, or a k nearest neighbor classifier.

[0042] At operation 730, based on the temporal ordering of the event sub-portions and the associated scores, sequence decoding is performed on the sound event sub-portions and the associated scores to detect the sound events. In some embodiments, the sequence decoding can be performed by a weighted finite state transducer, a recurrent neural network (RNN), or a long short-term memory RNN.

[0043] Of course, in some embodiments, additional operations may be performed, as described above in conjunction with the system. For example, a classifier is trained on a sub-portion of a sound event generated by an unsupervised subspace clustering technique applied to training data including the target sound event. In some embodiments, the unsupervised subspace clustering may employ k-means clustering, Gaussian mixture model clustering, or DenStream clustering.

[0044] Example System

[0045] Figure 8 An example system 800 for performing sound event detection configured according to certain embodiments of the present disclosure is shown. In some embodiments, the system 800 includes a computing platform 810 that can manage or otherwise be incorporated into a personal computer, a workstation, a server system, a laptop computer, an ultra-laptop computer, a tablet computer, a touchpad, a portable computer, a handheld computer, a palmtop computer, a personal digital assistant (PDA), a cellular phone, a combination cellular phone and PDA, a smart device (e.g., a smart phone or smart tablet), a mobile Internet device (MID), a communication device, a data communication device, an imaging device, a wearable device, an embedded system, etc. Any combination of different devices may be used in certain embodiments.

[0046] In some embodiments, the platform 810 may include any combination of a processor 820, a memory 830, a sound event detection system 100, a network interface 840, an input / output (I / O) system 850, a user interface 860, an audio capture device 862, and a storage system 870. As can be further seen, a bus and / or interconnect 892 is also provided to allow communication between the various components listed above and / or other components not shown. The platform 810 may be coupled to a network 894 via the network interface 840 to allow communication with other computing devices, platforms, or resources. Figure 8 Other components and functionality not reflected in the block diagram will be apparent from this disclosure, and it will be appreciated that other embodiments are not limited to any particular hardware configuration.

[0047] The processor 820 can be any suitable processor and can include one or more coprocessors or controllers (e.g., an audio processor, a graphics processing unit, or a hardware accelerator) to help control and process operations associated with the system 800. In some embodiments, the processor 820 can be implemented as any number of processor cores. The processor (or processor core) can be any type of processor, for example, a microprocessor, an embedded processor, a digital signal processor (DSP), a graphics processor (GPU), a network processor, a field programmable gate array, or other devices configured to execute code. The processor can be a multi-threaded core, wherein they can include more than one hardware thread context (or "logical processor") per core. The processor 820 can be implemented as a complex instruction set computer (CISC) or a reduced instruction set computer (RISC) processor. In some embodiments, the processor 820 can be configured as an x86 instruction set compatible processor.

[0048] The memory 830 may be implemented using any suitable type of digital storage device including, for example, flash memory and / or random access memory (RAM). In some embodiments, the memory 830 may include various layers of a memory hierarchy known to those skilled in the art and / or a memory cache. The memory 830 may be implemented as a volatile memory device, such as, but not limited to, a RAM, a dynamic RAM (DRAM), or a static RAM (SRAM) device. The storage system 870 may be implemented as a non-volatile storage device, such as, but not limited to, a hard disk drive (HDD), a solid state drive (SSD), a universal serial bus (USB) drive, an optical drive, a tape drive, an internal storage device, an attached storage device, a flash memory, a battery-backed synchronous DRAM (SDRAM), and / or a network-accessible storage device. In some embodiments, the storage system 870 may include a technology that increases storage performance protection for valuable digital media when multiple hard drives are included.

[0049] Processor 820 may be configured to execute an operating system (OS) 880, which may include any suitable operating system such as Google Android (Google Inc., Mountain View, Calif.), Microsoft Windows (Microsoft Corporation, Redmond, Wash.), Apple OS X (Apple Inc., Cupertino, Calif.), Linux, or a real-time operating system (RTOS). As will be apparent from this disclosure, the techniques provided herein may be implemented without regard to a particular operating system provided in conjunction with system 800, and thus may also be implemented using any existing or subsequently developed platform.

[0050] The network interface circuit 840 can be any suitable network chip or chipset that allows wired and / or wireless connections between the computer system 800 and / or other components of the network 894, thereby enabling the system 800 to communicate with other local and / or remote computing systems, servers, cloud-based servers, and / or other resources. Wired communications can follow existing (or to be developed) standards for Ethernet, for example. Wireless communications can follow existing (or to be developed) standards for cellular communications such as, including LTE (Long Term Evolution), Wireless Fidelity (Wi-Fi), Bluetooth, and / or Near Field Communications (NFC). Exemplary wireless networks include, but are not limited to, wireless local area networks, wireless personal area networks, wireless metropolitan area networks, cellular networks, and satellite networks.

[0051] The I / O system 850 can be configured to be connected by an interface between various I / O devices and other components of the computer system 800. The I / O device can include, but is not limited to, a user interface 860 and an audio capture device 862 (e.g., a microphone). The user interface 860 can include devices such as, display elements, touch pads, keyboards, mice, and speakers (not shown). The I / O system 850 can include a graphics subsystem configured to perform processing for images rendered on the display element. The graphics subsystem can be, for example, a graphics processing unit or a visual processing unit (VPU). An analog or digital interface can be used to communicatively couple the graphics subsystem and the display element. For example, the interface can be any interface in a high-resolution multimedia interface (HDMI), a display port, a wireless HDMI, and / or any other suitable interface using wireless high-resolution compatible technology. In some embodiments, the graphics subsystem can be integrated into any chipset of the processor 820 or the platform 810.

[0052] It will be appreciated that in some embodiments, various components of system 800 may be combined or integrated in a system-on-chip (SoC) architecture. In some embodiments, a component may be a hardware component, a firmware component, a software component, or any suitable combination of hardware, firmware, or software.

[0053] The sound event detection system 100 is configured to detect an event of interest based on the identification or recognition of sub-portions of the event that occur in an expected time order, as previously described. The sound event detection system 100 may include Figure 1 and Figure 2 810. These components may be implemented or otherwise used in conjunction with various appropriate software and / or hardware coupled to or otherwise forming part of platform 810. These components may also be alternatively or additionally implemented or otherwise used in conjunction with user I / O devices capable of providing information to a user and receiving information and commands from a user.

[0054] In some embodiments, these circuits may be installed locally in the system 800, such as Figure 8 . Alternatively, system 800 may be implemented in a client-server deployment, wherein at least some of the functionality associated with these circuits is provided to system 800 using an applet (e.g., a JavaScript applet) or other downloadable set of modules or sub-modules. Such remotely accessible modules or sub-modules may be provided in real time in response to a request from a client computing system to access a given server having resources of interest to a user of the client computing system. In these embodiments, the server may be local to network 894, or remotely coupled to network 894 via one or more other networks and / or communication channels. In some cases, access to resources on a given network or computing system may require credentials such as a username, password, and / or any other appropriate security mechanism.

[0055] In various embodiments, system 800 may be implemented as a wireless system, a wired system, or a combination of the two. When implemented as a wireless system, system 800 may include components and interfaces suitable for communicating on a wireless shared medium such as, for example, one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. Examples of wireless shared media may include portions of a wireless spectrum, for example, a radio frequency spectrum, etc. When implemented as a wired system, system 800 may include components and interfaces suitable for communicating on a wired communication medium such as, for example, an input / output adapter, a physical connector connecting an input / output adapter to a corresponding wired communication medium, a network interface card (NIC), a disk controller, a video controller, an audio controller, etc. Examples of wired communication media may include wires, cable metal leads, printed circuit boards (PCBs), backplanes, switching matrices, semiconductor materials, twisted pairs, coaxial cables, optical fibers, etc.

[0056] Various embodiments can be realized using hardware elements, software elements or the combination of the two.The example of hardware elements can include processors, microprocessors, circuits, circuit elements (for example, transistors, resistors, capacitors, inductors, etc.), integrated circuits, ASICs, programmable logic devices, digital signal processors, FPGAs, logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc.The example of software can include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, processes, software interfaces, application program interfaces, instruction sets, computing codes, computer codes, code segments, computer code segments, words, values, symbols or any combination thereof.Determining an embodiment is to use hardware elements and / or software elements to realize that it can be changed according to any number of factors, and these factors are, for example, desired computing rate, power level, heat resistance, processing cycle budget, input data rate, output data rate, memory resources, data bus speed and other design or performance constraints.

[0057] Some embodiments may be described using the expressions "coupled" and "connected" and their derivatives. These terms are not intended to be synonymous with each other. For example, some embodiments are described using the terms "connected" and / or "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" may also mean that two or more elements are not in direct contact with each other, but still cooperate or interact with each other.

[0058] Various embodiments disclosed herein can be implemented with various forms of hardware, software, firmware, and / or special processors. For example, in some embodiments, at least one non-transient computer-readable storage medium has instructions encoded thereon, which, when executed by one or more processors, enable one or more sound event detection methods disclosed herein to be implemented. Suitable programming languages ​​such as C, C++, object-oriented C, Java, JavaScript, Visual Basic.NET, beginner general symbolic instruction code (BASIC) or alternatively custom or special instruction sets can be used to encode these instructions. Instructions can be provided in the form of one or more computer software applications and / or small applications that are tangibly embodied on a memory device and can be executed by a computer with any appropriate architecture. In one embodiment, the system can be hosted on a given website and, for example, implemented using JavaScript or other appropriate browser-based technologies. For example, in some embodiments, the system can utilize processing resources provided by a remote computer system accessible via network 894. In other embodiments, the functions disclosed herein can be incorporated into other sound-based software applications, such as security, monitoring, smart home management, robots, and Internet of Things (IoT) applications. The computer software applications disclosed herein may include any number of different modules, submodules, or other components of different functions, and may provide information to or receive information from other components. These modules may be used, for example, to communicate with input and / or output devices such as display screens, touch-sensitive surfaces, printers, and / or any other suitable devices. Other components and functions not reflected in the diagrams will be apparent from this disclosure, and it will be appreciated that other embodiments are not limited to any particular hardware or software configuration, and therefore, in other embodiments, compared to Figure 8 In addition to the components included in the example embodiments of FIG. 8 , system 800 may include additional, fewer, or alternative sub-components.

[0059] The aforementioned non-transient computer-readable medium can be any suitable medium for storing digital information, for example, a combination of a hard disk drive, a server, a flash memory, and / or a random access memory (RAM), or a memory. In an alternative embodiment, the components and / or modules disclosed herein can be implemented using hardware including gate-level logic such as a field programmable gate array (FPGA) or alternatively a dedicated semiconductor such as an application-specific integrated circuit (ASIC). Other embodiments can be implemented using a microcontroller with multiple input / output ports for receiving and outputting data and multiple embedded routines for implementing the various functions disclosed herein. It will be apparent that any suitable combination of hardware, software, and firmware can be used, and other embodiments are not limited to any particular system architecture.

[0060] Some embodiments may, for example, be implemented using a machine-readable medium or article that can store instructions or instruction sets, wherein the instructions or instruction sets, when executed by a machine, can cause the machine to perform methods and / or operations according to an embodiment. Such a machine may include, for example, any suitable processing platform, computing platform, computing device, processing device, computing system, processing system, computer, processing, etc., and may be implemented using any suitable combination of hardware and / or software. The machine-readable medium or article may include, for example, any suitable type of memory unit, memory device, memory article, memory medium, storage device, storage article, storage medium, and / or storage unit, for example, memory, removable or non-removable medium, erasable or non-erasable medium, writable or rewritable medium, digital or analog medium, hard disk, floppy disk, compact disk read-only memory (CD-ROM), compact disk recordable (CD-R) memory, compact disk rewritable (CD-RW) memory, optical disk, magnetic medium, magneto-optical medium, removable memory card or disk, various types of digital versatile disks (DVD), magnetic tape, magnetic cassette, etc. The instructions may include any suitable type of code, e.g., source code, compiled code, executable code, static code, dynamic code, encrypted code, etc., implemented using any suitable high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming language.

[0061] Unless specifically stated to the contrary, it is understood that terms such as "processing", "computing", "calculating", "determining", etc. refer to the action and / or process of a computer or computing system or similar computing device manipulating and / or transforming data represented as physical quantities (e.g., electronics) in registers and / or memory units of a computer system into other data represented as physical quantities in registers, memory units, or other such information storage, transmission, or display of a computer system. The embodiments are not limited to this context.

[0062] The term "circuit" or "circuitry" used in any embodiment herein is functional and may include, for example, any one of a hardware circuit, a programmable circuit (e.g., a computer processor including one or more separate instruction processing cores), a static machine circuit, and / or a firmware storing instructions executed by a programmable circuit, or any combination thereof. The circuit may include a processor and / or a controller configured to execute one or more instructions to perform one or more operations described herein. The instructions may be embodied as, for example, an application, software, firmware, etc. configured to cause the circuit to perform any of the aforementioned operations. The software may be embodied as a software package, code, instructions, instruction set, and / or data recorded on a computer-readable storage device. The software may be embodied or implemented to include any number of processes, and these processes may in turn be embodied or implemented to include any number of threads, etc., in a hierarchical manner. The firmware may be embodied as code, instructions, or instruction sets and / or data hard-coded (e.g., non-volatile) in a memory device. Circuits can be unified or embodied as circuits that form a part of a larger system (e.g., integrated circuits (ICs), application specific integrated circuits (ASICs), systems on chip (SoCs), desktop computers, laptop computers, tablet computers, servers, smart phones, etc.) respectively. Other embodiments can be implemented as software executed by a programmable control device. In these cases, the term "circuit" or "circuitry" is intended to include a combination of software and hardware (e.g., programmable control devices or processors) that can execute software. As described herein, various embodiments can be implemented using hardware elements, software elements, or any combination thereof. Examples of hardware elements may include processors, microprocessors, circuits, circuit elements (e.g., transistors, resistors, capacitors, inductors, etc.), integrated circuits, application specific integrated circuits (ASICs), programmable logic devices (PLDs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), logic gates, registers, semiconductor devices, chips, microchips, chipsets, etc.

[0063] A number of specific details have been provided herein to provide a thorough understanding of the embodiments. It will be appreciated by those of ordinary skill in the art that these embodiments can be implemented without these specific details. In other examples, known operations, components, and circuits are not described to avoid blurring the embodiments. It will be appreciated that the specific structural and functional details disclosed herein can be illustrative and do not necessarily limit the scope of the embodiments. In addition, although the subject matter is described using language specific to structural features and / or method actions, it will be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described herein. On the contrary, the specific features and actions described herein are disclosed as example forms for realizing the claims.

[0064] Further example embodiments

[0065] The following examples relate to further embodiments from which various arrangements and configurations will become apparent.

[0066] Example 1 is a processor-implemented method for detecting sound events, the method comprising: extracting one or more sound features from an audio signal by a processor-based system; applying a trained classifier to the extracted one or more sound features by the processor-based system to identify sound event subportions of the audio signal and generate scores associated with the subportions; and performing sequential decoding of the sound event subportions and associated scores by the processor-based system to detect sound events.

[0067] Example 2 includes the subject matter of Example 1, wherein the sequence decoding is based on a temporal ordering of the sound event sub-portions and a comparison of the associated scores to a threshold score value.

[0068] Example 3 includes the subject matter of Example 1 or 2, further comprising: training a classifier on sound event sub-portions generated by subspace clustering applied to training data, the training data comprising the target sound event.

[0069] Example 4 includes the subject matter of any of Examples 1 to 3, wherein the subspace clustering is unsupervised subspace clustering based on at least one of k-means clustering, Gaussian mixture model clustering, and DenStream clustering.

[0070] Example 5 includes the subject matter of any of Examples 1 to 4, further comprising: generating reference annotations for at least a subset of training data including target sound events, the reference annotations indicating sub-portions of the target sound events, and wherein the subspace clustering is semi-supervised subspace clustering based on constrained k-means clustering, seeded k-means clustering, and at least one of semi-supervised variants of Gaussian mixture model clustering and DenStream clustering.

[0071] Example 6 includes the subject matter of any of Examples 1 to 5, wherein the sequence order decoding is based on at least one of a weighted finite state transducer, a recurrent neural network (RNN), and a long short-term memory RNN.

[0072] Example 7 includes the subject matter of any one of Examples 1 to 6, wherein the extracted one or more sound features include at least one of a short-term Fourier transform representing at least a portion of the audio signal, a filter bank feature representing at least a portion of the audio signal, a Mel-frequency cepstral coefficient representing at least a portion of the audio signal, and a spectral shape descriptor representing at least a portion of the audio signal.

[0073] Example 8 includes the subject matter of any of Examples 1 to 7, wherein the classifier is at least one of a deep neural network, a support vector machine, a decision tree classifier, a k-nearest neighbor classifier, a random forest classifier, a gradient boosting classifier, and a logistic regression-based classifier.

[0074] Example 9 is a system for sound event detection, the system comprising: a feature extraction circuit that extracts sound features from an audio signal; a classifier circuit that identifies sound event subportions of the audio signal based on the extracted sound features and generates scores associated with the subportions; and a sequence decoder circuit that performs sequence decoding of the sound event subportions and associated scores to detect sound events.

[0075] Example 10 includes the subject matter of Example 9, wherein the sequence decoding is based on a temporal ordering of the sound event sub-portions and a comparison of the associated scores to a threshold score value.

[0076] Example 11 includes the subject matter of Example 9 or 10, further comprising: a classifier training circuit that trains a classifier on sound event subportions generated by subspace clustering applied to training data, the training data comprising a target sound event.

[0077] Example 12 includes the subject matter of any of Examples 9 to 11, wherein the subspace clustering is unsupervised subspace clustering based on at least one of k-means clustering, Gaussian mixture model clustering, and DenStream clustering.

[0078] Example 13 includes the subject matter of any of Examples 9 to 12, wherein at least one subset of the training data includes target sound events and associated reference annotations indicating sub-portions of the target sound events, and wherein the subspace clustering is semi-supervised subspace clustering based on constrained k-means clustering, seeded k-means clustering, and at least one of semi-supervised variants of Gaussian mixture model clustering and DenStream clustering.

[0079] Example 14 includes the subject matter of any of Examples 9 to 13, wherein the sequence decoder circuit employs at least one of a weighted finite state transducer, a recurrent neural network (RNN), and a long short-term memory RNN.

[0080] Example 15 includes the subject matter of any one of Examples 9 to 14, wherein the extracted sound features include at least one of a short-term Fourier transform representing at least a portion of the audio signal, a filter bank feature representing at least a portion of the audio signal, a Mel-frequency cepstral coefficient representing at least a portion of the audio signal, and a spectral shape descriptor representing at least a portion of the audio signal.

[0081] Example 16 includes the subject matter of any of Examples 9 to 15, wherein the classifier circuit is at least one of a deep neural network, a support vector machine, a decision tree classifier, a k-nearest neighbor classifier, a random forest classifier, a gradient boosting classifier, and a logistic regression-based classifier.

[0082] Example 17 is at least one non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by one or more processors, result in operations for detecting sound events, the operations comprising: extracting one or more sound features from an audio signal; applying a trained classifier to the extracted one or more sound features to identify sound event subportions of the audio signal and generate scores associated with the subportions; and performing sequential decoding of the sound event subportions and associated scores to detect sound events.

[0083] Example 18 includes the subject matter of Example 17, wherein the sequence decoding is based on temporal ordering of the sound event sub-portions and comparison of the associated scores to a threshold score value.

[0084] Example 19 includes the subject matter of Example 17 or 18, the operations further comprising: training a classifier on sound event sub-portions generated by subspace clustering applied to training data, the training data comprising the target sound event.

[0085] Example 20 includes the subject matter of any of Examples 17 to 19, wherein the subspace clustering is unsupervised subspace clustering based on at least one of k-means clustering, Gaussian mixture model clustering, and DenStream clustering.

[0086] Example 21 includes the subject matter of any one of Examples 17 to 20, wherein the operation further includes: generating reference annotations for at least a subset of training data including the target sound event, the reference annotations indicating a sub-portion of the target sound event, and wherein the subspace clustering is semi-supervised subspace clustering based on constrained k-means clustering, seeded k-means clustering, and at least one of semi-supervised variants of Gaussian mixture model clustering and DenStream clustering.

[0087] Example 22 includes the subject matter of any of Examples 17 to 21, wherein the sequence decoding is based on at least one of a weighted finite state transducer, a recurrent neural network (RNN), and a long short-term memory RNN.

[0088] Example 23 includes the subject matter of any of Examples 17 to 22, wherein the extracted one or more sound features include at least one of a short-term Fourier transform representing at least a portion of the audio signal, a filter bank feature representing at least a portion of the audio signal, a Mel-frequency cepstral coefficient representing at least a portion of the audio signal, and a spectral shape descriptor representing at least a portion of the audio signal.

[0089] Example 24 includes the subject matter of any of Examples 17 to 23, wherein the classifier is at least one of a deep neural network, a support vector machine, a decision tree classifier, a k-nearest neighbor classifier, a random forest classifier, a gradient boosting classifier, and a logistic regression-based classifier.

[0090] Example 25 is a system for detecting sound events, the system comprising: a device for extracting one or more sound features from an audio signal; a device for applying a trained classifier to the extracted one or more sound features to identify sound event subportions of the audio signal and generate scores associated with the subportions; and a device for performing sequential decoding of the sound event subportions and associated scores to detect sound events.

[0091] Example 26 includes the subject matter of Example 25, wherein the sequence decoding is based on temporal ordering of the sound event sub-portions and comparison of the associated scores to a threshold score value.

[0092] Example 27 includes the subject matter of Example 25 or 26, further comprising: means for training a classifier on sound event sub-portions generated by subspace clustering applied to training data, wherein the training data includes target sound events.

[0093] Example 28 includes the subject matter of any of Examples 25 to 27, wherein the subspace clustering is unsupervised subspace clustering based on at least one of k-means clustering, Gaussian mixture model clustering, and DenStream clustering.

[0094] Example 29 includes the subject matter of any of Examples 25 to 28, further comprising: a device for generating reference annotations for at least a subset of training data including a target sound event, the reference annotations indicating a sub-portion of the target sound event, and wherein the subspace clustering is semi-supervised subspace clustering based on constrained k-means clustering, seeded k-means clustering, and at least one of semi-supervised variants of Gaussian mixture model clustering and DenStream clustering.

[0095] Example 30 includes the subject matter of any of Examples 25 to 29, wherein the sequence decoding is based on at least one of a weighted finite state transducer, a recurrent neural network (RNN), and a long short-term memory RNN.

[0096] Example 31 includes the subject matter of any one of Examples 25 to 30, wherein the extracted one or more sound features include at least one of a short-term Fourier transform representing at least a portion of the audio signal, a filter bank feature representing at least a portion of the audio signal, a Mel-frequency cepstral coefficient representing at least a portion of the audio signal, and a spectral shape descriptor representing at least a portion of the audio signal.

[0097] Example 32 includes the subject matter of any of Examples 25 to 31, wherein the classifier is at least one of a deep neural network, a support vector machine, a decision tree classifier, a k-nearest neighbor classifier, a random forest classifier, a gradient boosting classifier, and a logistic regression-based classifier.

[0098] The terms and expressions used herein are used as terms of description rather than limitation, and the use of these terms and expressions does not mean that any equivalents of the features (or parts thereof) shown and described are excluded, and it will be recognized that various modifications are possible within the scope of the claims. Therefore, the claims are intended to cover all equivalents. Various features, aspects, and embodiments are described herein. These features, aspects, and embodiments can be combined with each other and can also be combined with deformation and modification, as will be understood by those skilled in the art. Therefore, the present disclosure should be understood to cover these combinations, deformations, and modifications. It is hoped that the scope of the present disclosure is not limited to this specific description, but is limited by the appended claims. Future applications filed claiming priority to the present application may request protection of the disclosed subject matter in different ways, and generally include any collection of one or more elements disclosed or illustrated differently herein.

Claims

1. A processor-implemented method for detecting a sound event, the method comprising: extracting, by a processor-based system, one or more sound features from the audio signal; applying, by the processor-based system, a trained classifier to the extracted one or more sound features to identify sound event subportions of the audio signal and generate scores associated with the subportions; as well as Sequence decoding of the sound event subportions and associated scores is performed by the processor-based system to detect sound events, wherein the sequence decoding is used to recognize a correct time sequence of the sound event subportions.

2. The method of claim 1, wherein: The sequence decoding is based on a temporal ordering of the sound event sub-portions and a comparison of the associated scores to a threshold score value, and a sound event is detected based on a correct temporal sequence of the sound event sub-portions having a score greater than or equal to the threshold score value.

3. The method of claim 1, further comprising: The classifier is trained on sub-portions of sound events generated by subspace clustering applied to training data, the training data comprising target sound events.

4. The method of claim 3, wherein: The subspace clustering is unsupervised subspace clustering based on at least one of k-means clustering, Gaussian mixture model clustering, and DenStream clustering.

5. The method of claim 3, further comprising: Reference annotations are generated for at least a subset of the training data including target sound events, the reference annotations indicating sub-portions of the target sound events, and wherein the subspace clustering is semi-supervised subspace clustering based on constrained k-means clustering, seeded k-means clustering, and at least one of semi-supervised variants of Gaussian mixture model clustering and DenStream clustering.

6. The method according to any one of claims 1 to 5, wherein: The sequence decoding is based on at least one of a weighted finite state transducer, a recurrent neural network (RNN), and a long short-term memory recurrent neural network.

7. The method of claim 1, wherein: The extracted one or more sound features include at least one of a short-term Fourier transform representing at least a portion of the audio signal, a filter bank feature representing at least a portion of the audio signal, a Mel-frequency cepstral coefficient representing at least a portion of the audio signal, and a spectral shape descriptor representing at least a portion of the audio signal.

8. The method of claim 1, wherein: The classifier is at least one of a deep neural network, a support vector machine, a decision tree classifier, a k-nearest neighbor classifier, a random forest classifier, a gradient boosting classifier, and a logistic regression-based classifier.

9. A system for sound event detection, the system comprising: a feature extraction circuit that extracts sound features from the audio signal; a classifier circuit that identifies a sound event sub-portion of the audio signal based on the extracted sound features and generates a score associated with the sub-portion; as well as A sequence decoder circuit performs sequence decoding of the sound event subportions and associated scores to detect sound events, wherein the sequence decoding is used to recognize the correct time sequence of the sound event subportions.

10. The system of claim 9, wherein: The sequence decoding is based on a temporal ordering of the sound event sub-portions and a comparison of the associated scores to a threshold score value, and a sound event is detected based on a correct temporal sequence of the sound event sub-portions having scores greater than or equal to the threshold score value.

11. The system of claim 9, further comprising: A classifier training circuit trains the classifier on the sound event sub-portions generated by subspace clustering applied to training data, the training data comprising target sound events.

12. The system of claim 11, wherein: The subspace clustering is unsupervised subspace clustering based on at least one of k-means clustering, Gaussian mixture model clustering, and DenStream clustering.

13. The system of claim 11, wherein: At least one subset of the training data includes target sound events and associated reference annotations, wherein the reference annotations indicate sub-portions of the target sound events, and wherein the subspace clustering is semi-supervised subspace clustering based on constrained k-means clustering, seeded k-means clustering, and at least one of semi-supervised variants of Gaussian mixture model clustering and DenStream clustering.

14. The system of any one of claims 9 to 11, wherein: The sequence decoder circuit employs at least one of a weighted finite state transducer, a recurrent neural network (RNN), and a long short-term memory recurrent neural network.

15. The system of claim 9, wherein: The extracted sound features include at least one of a short-term Fourier transform representing at least a portion of the audio signal, a filter bank feature representing at least a portion of the audio signal, a Mel-frequency cepstral coefficient representing at least a portion of the audio signal, and a spectral shape descriptor representing at least a portion of the audio signal.

16. The system of claim 9, wherein: The classifier circuit is at least one of a deep neural network, a support vector machine, a decision tree classifier, a k-nearest neighbor classifier, a random forest classifier, a gradient boosting classifier, and a logistic regression based classifier.

17. A system for sound event detection, the system comprising: means for extracting one or more sound features from an audio signal; means for applying a trained classifier to the extracted one or more sound features to identify sound event subportions of the audio signal and to generate scores associated with the subportions; as well as Means for performing sequence decoding of the sound event sub-portions and associated scores to detect sound events, wherein the sequence decoding is used to recognize the correct time sequence of the sound event sub-portions.

18. The system of claim 17, wherein: The sequence decoding is based on a temporal ordering of the sound event sub-portions and a comparison of the associated scores to a threshold score value, and a sound event is detected based on a correct temporal sequence of the sound event sub-portions having scores greater than or equal to the threshold score value.

19. The system of claim 17, further comprising: Means for training the classifier on sub-portions of sound events generated by subspace clustering applied to training data, the training data comprising target sound events.

20. The system of claim 19, wherein: The subspace clustering is unsupervised subspace clustering based on at least one of k-means clustering, Gaussian mixture model clustering, and DenStream clustering.

21. The system of claim 19, further comprising: A device for generating reference annotations for at least a subset of the training data including a target sound event, the reference annotations indicating a sub-portion of the target sound event, and wherein the subspace clustering is semi-supervised subspace clustering based on constrained k-means clustering, seeded k-means clustering, and at least one of semi-supervised variants of Gaussian mixture model clustering and DenStream clustering.

22. The system of any one of claims 17 to 21, wherein: The sequence decoding is based on at least one of a weighted finite state transducer, a recurrent neural network (RNN), and a long short-term memory recurrent neural network.

23. The system of claim 17, wherein: The extracted one or more sound features include at least one of a short-term Fourier transform representing at least a portion of the audio signal, a filter bank feature representing at least a portion of the audio signal, a Mel-frequency cepstral coefficient representing at least a portion of the audio signal, and a spectral shape descriptor representing at least a portion of the audio signal.

24. The system of claim 17, wherein: The classifier is at least one of a deep neural network, a support vector machine, a decision tree classifier, a k-nearest neighbor classifier, a random forest classifier, a gradient boosting classifier, and a logistic regression-based classifier.

25. At least one non-transitory computer-readable storage medium having encoded thereon instructions that, when executed by one or more processors, implement the method for detecting sound events of any one of claims 1-8.

Images