Systems and methods for CLAP4SED

Abstract

A method for real-time sound event detection on an embedded device includes pre-training a contrast language-audio pre-training model as an audio base model and preparing an offline multi-modal query prototype for a sound event of interest. The pre-trained model and the query prototype are deployed on an embedded device. The device receives an input audio stream and extracts an audio embedding using a pre-trained model. A similarity score between the extracted audio embedding and the prepared query prototype is calculated. The presence of the sound event is determined based on the calculated similarity score, and the real-time sound event detection result is output. The system includes a memory storing a pre-trained model and a query prototype, an audio input interface, and a processor configured to perform extraction, calculation, determination, and output operations. A non-transitory computer readable medium stores instructions that, when executed, cause a processor to implement the method.

Description

Technical Field

[0001] This disclosure generally relates to machine learning systems that use reinforcement learning techniques for audio event detection and classification. Background Technology

[0002] Sound event detection (SED) is complementing vision-based systems by better handling occlusion and supporting omnidirectional signals, and is becoming increasingly important for detecting our environment. Applications range from urban noise monitoring to tracking bird diversity and migration. However, implementing real-time SED on embedded devices presents significant challenges, primarily related to versatility and complexity.

[0003] Existing SED models are primarily suited for closed-form recognition, making it difficult to adapt to new or unseen sound categories. This limitation becomes particularly problematic in real-world applications where the category of interest or acoustic conditions change frequently over time. Continuously retraining the model to adapt to these changes can be excessively expensive and time-consuming.

[0004] Recent advances in Audio Fundamental Models (AFM) offer potential solutions to bridging the generalization gaps encountered with unseen acoustic events or conditions. These models, such as those based on Contrastive Language-Audio Pre-training (CLAP), unlock free-form natural language interaction with audio data and provide new avenues for embedded audio AI solutions. However, AFM typically relies on heavily computational model architectures, especially when accompanied by additional language models. This presents another crucial challenge for using AFM in embedded device settings.

[0005] Furthermore, the increasing prevalence of multimodal foundational models utilizing large language models (LLMs) has led to growing interest in cueing engineering techniques. Exploring how to effectively cue or design queries has become a key topic for extracting desired knowledge from these models. Additionally, the potential challenges of modal gaps for multimodal applications have led to proposed solutions involving the absorption of few-shot prototypes from each modality. Given these challenges, a SED solution is needed that can run in real-time on embedded devices, flexibly adapt to various deployment environments, and leverage the capabilities of audio foundational models without incurring excessive computational costs. Summary of the Invention

[0006] In one or more illustrative examples, a method for real-time sound event detection on an embedded device may include a pre-trained contrastive language-audio pre-trained model as the audio base model. An offline multimodal query prototype can be prepared for sound events of interest. The pre-trained model and the prepared query prototype can be deployed on the embedded device. The method may involve receiving an input audio stream on the embedded device and extracting audio embeddings from the stream using the pre-trained model. A similarity score can be calculated between the extracted audio embeddings and the prepared query prototype. The presence of a sound event can be determined based on these calculated similarity scores. The method outputs real-time sound event detection results. The contrastive language-audio pre-trained model may include an audio encoder and a text encoder trained to contrastively optimize symmetric similarity in a joint multimodal space for audio-text pairs. The audio encoder may be a lightweight, parallel audio neural network architecture. Preparing the query prototype may involve: extracting audio embeddings from few-shot audio samples; generating text prompts using a large language model; extracting text embeddings; and selecting the most relevant text embeddings based on their similarity to the audio embeddings.

[0007] In terms of systems, illustrative examples of real-time sound event detection on embedded devices include memory storing a pre-trained contrastive language-audio pre-trained model and prepared multimodal query prototypes. An audio input interface may be included for receiving an input audio stream. A processor may be configured to extract audio embeddings from the input audio stream using the pre-trained model, compute a similarity score between the extracted audio embeddings and the prepared query prototypes, determine the presence of a sound event based on the computed similarity score, and output real-time sound event detection results. The system can leverage a lightweight, parallel audio neural network architecture in the audio encoder. The prepared multimodal query prototypes may include audio query vectors derived from few-shot audio samples and text query vectors derived from text prompts generated by a large language model. The processor may also be configured to preprocess the input audio stream to generate a spectrogram before extracting the audio embeddings and to determine the presence of a sound event by applying binary thresholding to the computed similarity score.

[0008] In another example, the non-transient computer-readable medium may include instructions that, when executed by a processor on an embedded device, facilitate real-time sound event detection. The method may include loading a pre-trained contrastive language-audio pre-trained model and a prepared multimodal query prototype; receiving an input audio stream; extracting audio embeddings from the input audio stream using the pre-trained model; calculating a similarity score between the extracted audio embeddings and the prepared query prototype; determining the presence of a sound event based on the calculated similarity score; and outputting a real-time sound event detection result. These operations can be performed using a lightweight, parallel audio neural network architecture for an audio encoder, where the similarity score is calculated using dot product computation. The embedded device may include a chip with a quad-core processor, a single-instruction multiple-data accelerator, and a computer vision stream vector processor, enabling efficient real-time processing of the audio stream for sound event detection. Attached Figure Description

[0009] Figure 1 The diagram illustrates the CLAP4SED framework.

[0010] Figure 2 Illustrated example of embedded hardware setup;

[0011] Figure 3 The diagram illustrates an example process for implementing few-shot active learning for audio event detection and classification; and

[0012] Figure 4 The illustration shows an example computing device for implementing active learning for anomaly detection and classification. Detailed Implementation

[0013] This document describes embodiments of the present disclosure. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily drawn to scale, and some features may be exaggerated or minimized to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described in any of the accompanying drawings can be combined with features illustrated in one or more other drawings to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative implementations for typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desirable for a particular application or implementation.

[0014] This disclosure describes, in one or more embodiments, a deep reinforcement-based active machine learning system and method for audio event detection and classification. The system may include three components: an audio event classifier, a reinforcement learning query policy module, and a few-shot adaptation module. These components work together to iteratively improve the audio event detection model while minimizing the number of labeled samples required.

[0015] A contrastive language-audio pre-trained model is used as the audio foundation model for sound event detection. This model, comprising an audio encoder fA(·) and a text encoder fT(·)), is trained to contrastively optimize symmetric similarity in a joint multimodal space for audio-text pairs (Elizalde et al., 2023). The training process involves processing incoming audio sequence pairs Xa and corresponding captions Xt, yielding audio embeddings Ea = fA(Xa) and text embeddings Et = fT(Xt), respectively.

[0016] The kernel for training the CLAP model is encapsulated in a sophisticated loss function, which forms the basis for the CLAP4SED system's ability to implement real-time sound event detection on embedded devices. This loss function is defined as follows:

[0017] L=1 / (2B)∑[log diag(softmax(eta*(Ea·Et^T)))+log diag(softmax(eta*(Et.Ea^T)))]

[0018] Where L represents the total loss to be minimized during training, B represents the mini-batch size used during training, Ea represents the audio embedding generated by the audio encoder fA(·), Et represents the text embedding generated by the text encoder fT(·), η is the temperature parameter for scaling the similarity score, diag(·) refers to the diagonal element of the result matrix, and softmax(·) is the softmax function applied to scale the similarity score.

[0019] The loss function is designed to optimize the symmetric similarity between audio and text embeddings within each batch, prompting the model to learn a joint representation space in which semantically related audio and text pairs are closer to each other. The loss function consists of two symmetric terms, ensuring that the model learns a bidirectional mapping between audio and text modalities. This symmetry is crucial for the CLAP4SED system's ability to efficiently perform cross-modal retrieval and similarity calculation.

[0020] The softmax function is applied to scaled dot product similarities (η*(Ea·Et^T) and η*(Et·Ea^T)). This operation normalizes the similarity scores, effectively creating a probability distribution over possible matches within a batch. The temperature parameter η allows for fine-tuning of this distribution, thus affecting the model's sensitivity to similarity.

[0021] By taking the logarithm of the diagonal elements (which represents the correct audio-text pair), the loss function heavily penalizes incorrect matches while providing a gradually decreasing reward for very confident correct matches. This encourages the model to learn a robust representation that generalizes well. The coefficients 1 / (2B) of the average batch size and the loss over two symmetric terms ensure that the loss is appropriately scaled regardless of the chosen batch size. This normalization is important for stable training across different hardware configurations and dataset sizes.

[0022] The dot product Ea·Et^T calculates the similarity between all audio and text embedding pairs in a batch. This operation is highly efficient and optimized for parallel computation on GPUs or dedicated hardware, such as the Ambarella CV22 chip used in this embedded system. The carefully designed loss function enables the CLAP model to learn a joint embedding space in which semantically related audio and text pairs are located closer to each other. This learned representation is key to the CLAP4SED system's ability to implement efficient and accurate sound event detection on embedded devices with limited computational resources.

[0023] The system's critical component—the audio encoder—is designed as a lightweight, parallel audio neural network architecture optimized for efficient deployment on embedded devices. This design choice reflects the growing trend toward edge computing in audio processing applications for tasks such as noise monitoring in urban areas (Bello et al., 2019), tracking bird diversity (Kahl et al., 2021), and detecting bird migration (Lostanlen et al., 2022). The encoder's lightweight nature is particularly important for real-time processing on resource-constrained devices, enabling applications in diverse environments ranging from smart cities to remote wildlife monitoring stations.

[0024] The architecture of this model was carefully chosen to balance performance and efficiency while considering the computational constraints of the specific audio domain and embedded devices. The use of parallel audio neural networks is inspired by the architecture of PANN (Kong et al., 2020), which addresses the unique challenges of audio data. This approach allows for the efficient processing of both spectral and temporal features simultaneously, which is crucial for accurate sound event detection. The parallel processing nature of this architecture enables the model to capture both short-term acoustic events and long-term temporal patterns, which is necessary for distinguishing between various types of sound events, from transient sounds such as gunshots (Mares and Blackburn, 2021) to more persistent sounds such as birdsong or urban noise.

[0025] The CLAP4SED system introduces an innovative approach to prepare offline multimodal query prototypes for sound events of interest. This process involves a sophisticated combination of audio and text processing techniques. Utilizing transfer learning from a pre-trained model, audio embeddings are extracted from N few-shot audio samples for each sound event of interest. This few-shot approach is particularly valuable in scenarios where collecting large amounts of labeled data for each sound event is impractical or expensive.

[0026] Simultaneously, the system uses large language models, such as GPT-4, to generate text cues describing each sound event (Brown et al., 2020). This approach to cue engineering applies advanced natural language processing techniques to the field of audio event detection, focusing on modal gaps in multimodal applications (Liang et al., 2022). The system employs GPT-4 to rewrite a standard CLAP retrieval template (e.g., "This is the sound of [category label]") into M distinct cues, enriching the textual representation of each sound event. This enrichment process allows for more nuanced and context-aware representations of sound events, potentially capturing subtle semantic differences that may be crucial for accurate detection.

[0027] The text cues are then processed to extract text embeddings, creating a bridge between the auditory and linguistic representations of the sound event. The system employs a clever selection mechanism to choose the most relevant text embeddings based on their similarity to the audio embeddings. Specifically, an audio-informed max-pooling operation is performed on the cues, returning the cue embedding with the maximum dot-product similarity to a given few-shot audio embedding. This results in the final text query q|T∈R1×d, where d represents the dimension of the hidden space in the CLAP model where audio-text contrastive learning is performed.

[0028] This multimodal approach to creating query prototypes surpasses traditional unimodal systems, allowing for a more comprehensive representation of sound events (Kushwaha and Fuentes, 2023). By combining audio and text modalities, the system can capture both the acoustic properties and semantic descriptions of sound events, potentially leading to more robust and adaptive detection capabilities.

[0029] During real-time sound event detection, the system calculates a similarity score between audio embeddings extracted from the input audio stream and a prepared query prototype. This calculation typically involves computing the dot product between these embeddings, a method chosen for its computational efficiency and effectiveness in high-dimensional space. The presence of a sound event is then determined by applying binary thresholding to the calculated similarity scores to form a cross-modal average decision score. This decision-making process can be tuned for different sensitivity requirements, allowing the system to adapt to various deployment scenarios with varying tolerances for false positives and false negatives.

[0030] A notable feature of the CLAP4SED system is its ability to adapt to new types of sound events without large-scale retraining. By leveraging few-shot learning techniques and the flexibility of contrastive language-audio pre-trained models, the system is able to rapidly incorporate new sound events into its detection range. This adaptability is crucial in real-world applications where acoustic environments can change rapidly or unpredictably, such as in the detection of various urban sounds like gunshots (Mares and Blackburn, 2021), glass-breaking sounds (Suliman et al., 2020), or emerging environmental sounds that may be indicators of ecological change.

[0031] The real-time operation of CLAP4SED on embedded devices represents a significant engineering achievement. The system can be designed to run on hardware such as the Ambarella CV22 chip, typically used in IP cameras. This chip can be equipped with a quad-core ARM A-53 processor with Linux support, 1MB of L2 cache, a Neon SIMD accelerator for digital signal processing (DSP), and a computer vision (CV) stream vector processor for deep learning matrix operations. The Neon chip can effectively accelerate the Fast Fourier Transform (FFT) for spectrogram computation, which is crucial for efficient audio processing.

[0032] In the online SED predictor phase, only the lightweight audio encoder and modality-specific query vectors need to be pre-stored in the embedded device. Upon receiving an input audio stream data block (with a window size L), the encoder fA(·) extracts the input audio stream data block to generate a key embedding k∈R1×d. These embeddings, along with the prepared query, are then used to compute a predefined similarity criterion, forming the basis for sound event detection decisions.

[0033] The minimum real-time prediction time grid, denoted as τ, depends on the overall latency of the prediction process. The system design allows for continuous processing of the input audio stream with minimal latency, extraction of audio embeddings, calculation of similarity scores, and output of detection results. This real-time capability, combined with the system's adaptability, makes it suitable for a wide range of applications, from smart home devices to industrial monitoring systems and environmental sensor networks (Mesaros et al., 2021).

[0034] Optimizing the CLAP4SED system involves careful consideration of various components. The architecture of the audio encoder can be fine-tuned to achieve the optimal balance between detection accuracy and computational efficiency on the target embedded device. This may involve techniques such as model compression via distillation and quantization (Polino et al., 2018), which can further reduce computational requirements without significantly impacting detection performance. The process of generating query prototypes can be refined through iterative experimentation, potentially incorporating active learning techniques to identify the most instructive examples for each sound event.

[0035] Evaluations of the CLAP4SED system typically involve comprehensive trials on benchmark sound event detection datasets, such as AudioSet (Gemmeke et al., 2017) or ESC-50. Performance metrics include accuracy and F1 score, as well as real-time performance indicators such as detection latency and computational resource utilization. The system's ability to adapt to new sound events with minimal additional data is a key focus of the evaluation, potentially utilizing few-shot learning benchmarks suitable for the audio domain. These evaluations help quantify the system's effectiveness across a range of sound event types and acoustic environments, providing insights into the system's generalizable capabilities and potential areas for improvement.

[0036] The CLAP4SED system offers several technical advantages over existing methods for sound event detection. Its use of a pre-trained contrastive language-audio model allows for the handling of a wide range of sound events without requiring large-scale, task-specific training. This approach leverages the power of transfer learning, which has significantly impacted many aspects of machine learning in recent years (Lin et al., 2023). The system's ability to efficiently operate with limited labeled data makes it particularly valuable in domains where acquiring large amounts of annotated audio data is challenging or costly.

[0037] This system's use of both audio and textual information in its multimodal query prototype represents an innovative approach to sound event representation. This multimodal approach allows for flexible adaptation to new sound events, potentially capturing semantic information that might be missed by purely audio-based systems (Huang et al., 2024). By bridging the gap between acoustic features and linguistic descriptions, CLAP4SED could potentially achieve more nuanced and context-aware sound event detection, more easily adapting to diverse and changing acoustic environments.

[0038] CLAP4SED's ability to operate in real-time on embedded devices aligns with the growing trend of edge computing in IoT and smart device applications. This capability opens up new possibilities for sound event detection in resource-constrained environments, ranging from wildlife monitoring in remote areas to noise pollution tracking in urban environments (Lostanlen et al., 2022). The system's efficiency and adaptability make it suitable for deployment in a wide range of scenarios, potentially enabling new applications in areas such as environmental monitoring, security, and assistive technologies.

[0039] The CLAP4SED system represents a significant advancement in sound event detection, combining state-of-the-art machine learning techniques with practical considerations for real-world deployments. CLAP4SED's innovative use of contrastive learning, multimodal representation, and efficient embedded processing paves the way for broader and more effective use of audio event detection technology across a diverse range of applications. As the field of audio AI continues to evolve, systems like CLAP4SED are likely to play an increasingly important role in our ability to understand and interact with the acoustic world surrounding us.

[0040] Figure 1 The CLAP4SED system 100 for real-time sound event detection (SED) on embedded devices is shown, featuring two main components: (A) an offline multimodal query preparation phase 102 and (B) an online real-time SED predictor 150. In the offline multimodal query preparation phase 102: the process begins with a sound category 104, represented by a dog icon. This sound category 104 is input to a ChatGPT-like language model 106 for auto-hint generation. The language model 106 generates M distinct hints 108, such as "hint 1," "hint 2," up to "hint M." This auto-hint generation produces rich textual representations of sound categories, enabling more robust query vectors.

[0041] These cues 108 are fed into a text encoder 110, which is part of a pre-trained CLAP (Contrastive Language-Audio Pretraining) model 112. The CLAP model 112 is labeled “Pre-trained CLAP (Frozen)” to indicate that its weights remain fixed during this process. The text encoder 110 outputs a d×M dimension text embedding matrix 114, where d represents the embedding dimension and M is the number of cues. An audio-based max-pooling operation 116 is applied to this dimension of the text embedding matrix 114 to obtain the final text query qT 118—a d-dimensional vector represented by the green rectangle. This max-pooling operation selects the most salient features across all cue embeddings associated with a given few-sample audio, creating a compact and instructive text representation.

[0042] Simultaneously, the audio encoder 122 of the pre-trained CLAP model 112 processes N few-shot audio samples 120 for the sound category. These N few-shot audio samples 120 are visually represented by three waveform icons labeled "N samples". The audio encoder 122 generates a d×N-dimensional audio embedding matrix 124, where N is the number of few-shot audio samples. A mean pooling operation 126 is applied to this matrix 124 to produce the final audio query qA 128—also a d-dimensional vector, represented by a yellow rectangle. This mean pooling operation averages the embeddings across all few-shot samples, creating a representative audio embedding prototype for the sound category.

[0043] In the online real-time SED predictor stage 150 on the embedded device: pre-computed query vectors qT 118 and qA 128 are stored in the device's memories 152 and 154, respectively. This is indicated by the dashed lines connecting them to the embedded device section, emphasizing the migration of queries to be computed offline to the online system. An input audio stream 156, labeled "Input Audio," is continuously fed into the system. This is visually represented by a waveform labeled "Mono, 16kHz," indicating its characteristics—a mono audio signal sampled at 16kHz.

[0044] The input audio stream 156 is also processed by an audio encoder 158 stored on the device to generate a key embedding k 160. This audio encoder 158 is a lightweight version of the CLAP audio encoder, optimized for real-time processing on embedded devices. A similarity score (SimScore) is computed using average pooling on the dot product of the pre-computed query vectors qT 118 and qA 128 stored in the device's memories 152 and 154 with the key embedding 160. This operation is explicitly shown in the formula: “SimScore = MeanPool([qT; qA] * k)”. This computation effectively measures the similarity between the input audio and the pre-computed query vectors in the shared embedding space.

[0045] Finally, thresholding operation 164 is applied to the similarity score to determine the presence or absence of a sound event. This is visually represented by a threshold curve icon, indicating that the system can adjust for different sensitivity levels. The output (SED-out) 166 shows the detection results over time, with an "on" indicator when a sound event (represented by a dog icon) is detected. This is visualized as a timeline with orange blocks indicating the detection period, demonstrating the system's ability to perform continuous real-time sound event detection.

[0046] The CLAP4SED framework enables efficient real-time sound event detection by utilizing pre-computed query vectors 118 and 128 and an audio encoder 158 on an embedded device, making it suitable for resource-constrained environments. The innovative use of both text (qT) and audio (qA) query vectors enhances the system's robustness and adaptability to various acoustic conditions and linguistic descriptions of sound events.

[0047] Figure 2 The diagram was used in Figure 1 The accompanying figure shows the hardware architecture 200 of the embedded device in the CLAP4SED system 100 for real-time sound event detection. The components enable efficient processing of audio signals and execution of sound event detection algorithms. The input audio signal 202 is represented by a microphone icon and an accompanying waveform, indicating the system's ability to capture and process real-time audio streams, corresponding to the ability to receive input audio streams on the embedded device.

[0048] The captured audio signal is then fed into the Ambarella chip 204, which is enclosed within a boundary representing an integrated system-on-a-chip (SoC) architecture. The Ambarella chip 204 contains several components: a digital signal processor (DSP) 206, responsible for the initial processing of the input audio signal. This component can handle tasks such as audio preprocessing and spectrogram generation, preprocessing the input audio stream to generate a spectrogram before extracting the audio embedding.

[0049] Adjacent to the DSP is a multi-core processor 208, which may be a multi-core ARM A53 processor crucial for executing the CLAP4SED algorithm, including extracting audio embeddings from the input audio stream using a pre-trained contrastive language-audio pre-trained model, and calculating similarity scores between the extracted audio embeddings and prepared multimodal query prototypes. An Ambarella memory system 210 provides the necessary storage for the pre-trained model, pre-computed query prototypes, and runtime data. This allows for the deployment of the pre-trained contrastive language-audio pre-trained model and prepared query prototypes on an embedded device.

[0050] At the bottom of the chip diagram is the CVflow Ambarella Vector Processor 212, spanning the full width of the chip representation. This dedicated processor is designed for efficient execution of computer vision and machine learning tasks. In the context of CLAP4SED, it accelerates computations such as the dot product between audio embeddings and query prototypes, and the subsequent average pooling operation described in the claims.

[0051] The overall architecture of the Ambarella chip 204, with its combination of a DSP 206, processor 208, Ambarella memory system 210, and Ambarella vector processor 212, embodies a chip with a quad-core processor, a single-instruction multiple-data accelerator, and a computer vision stream vector processor. This hardware configuration can be specifically designed to meet the computational requirements of real-time sound event detection while maintaining the energy efficiency required for embedded applications. By utilizing this dedicated hardware, the CLAP4SED system 100 can efficiently perform audio embedding extraction, similarity score calculation, and thresholding operations in real time on resource-constrained embedded devices to determine the presence of sound events.

[0052] Figure 3 The illustration depicts process 300 of a CLAP4SED system for real-time sound event detection on an embedded device, according to one or more embodiments. This process includes a comprehensive approach for efficient and adaptive sound event detection utilizing advanced machine learning techniques and optimized hardware implementations.

[0053] The process begins with a pre-trained contrastive language-audio pre-trained model as the audio foundation model, as described in step 302. This crucial step involves training both the audio encoder and the text encoder to contrastively optimize symmetric similarity in a joint multimodal space for audio-text pairs. The audio encoder can be specifically designed as a lightweight, parallel audio neural network architecture, balancing computational efficiency and performance to suit the constraints of embedded devices. This pre-training process can form the basis for the system's ability to generate meaningful audio embeddings for diverse sound events.

[0054] Next, as described in step 304, an offline multimodal query prototype is prepared for the sound events of interest. This sophisticated process involves extracting audio embeddings from a few-shot audio samples for each sound event of interest. Typically, N samples are used per sound event, where N is a small positive integer that allows the system to learn from a finite number of examples. The system then generates text cues describing each sound event using a large language model. This step involves capturing nuanced descriptions of the sound events to rewrite conventional contrastive language-audio pre-trained retrieval templates (e.g., “This is the sound of [category label]”) to enrich the textual expressiveness. Text embeddings are then extracted from these generated text cues. Finally, the system selects the most relevant text embeddings based on their similarity to the audio embeddings, creating a robust multimodal representation for each sound event.

[0055] The pre-trained contrastive language-audio pre-trained model and the prepared query prototype are then deployed on an embedded device, as shown in step 306. Such an embedded device typically includes a dedicated chip with a quad-core processor, a single-instruction multiple-data (SIMD) accelerator, and a computer vision stream vector processor. These hardware specifications are crucial for enabling real-time processing capabilities.

[0056] As indicated in step 308, the process continues to receive the input audio stream on the embedded device. Before further processing, the system can preprocess the input audio stream to generate a spectrogram, enhancing the time-frequency representation of the signal for more effective feature extraction.

[0057] Then, an audio embedding is extracted from the input audio stream (or its spectrogram) using a pre-trained contrastive language-audio pre-trained model, as illustrated in step 310. This step leverages a pre-trained audio encoder to efficiently transform the audio input into a compact, semantically rich representation.

[0058] The system then calculates a similarity score between the extracted audio embeddings and the prepared query prototype, as shown in step 312. This calculation typically involves computing the dot product between the embeddings, a method chosen for its computational efficiency and effectiveness in high-dimensional spaces.

[0059] Based on the calculated similarity score, the presence of the sound event is determined, as described in step 314. This determination typically involves applying a binary thresholding technique to the similarity score. This thresholding can be adjusted to balance detection sensitivity with false alarm rate, adapting to different operational requirements.

[0060] Finally, the process outputs real-time sound event detection results, as shown in step 316. This output indicates whether a specific sound event of interest has been detected in the input audio stream, providing timely information for various applications such as urban noise monitoring, wildlife tracking, or security systems.

[0061] Throughout the process, the CLAP4SED system leverages its unique architecture to implement real-time sound event detection with minimal computational overhead. The use of pre-computed query prototypes and pre-trained models allows for efficient processing on resource-constrained embedded devices. The system's ability to handle few-sample examples and utilize both audio and textual information makes it highly adaptable to novel sound events and diverse acoustic environments. Figure 4 The illustration shows an example 400 of a computing device 402 for implementing a CLAP4SED system for real-time sound event detection on an embedded device. As shown, the computing device 402 includes a processor 404 operatively connected to a memory 406, a network device 408, an output device 410, and an input device 412. It should be noted that this is merely an example, and computing devices 402 with more, fewer, or different components may be used.

[0062] Processor 404 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) and / or a graphics processing unit (GPU). In some examples, processor 404 is a system-on-a-chip (SoC) that integrates the functionality of a CPU and a GPU. The SoC may optionally include other components, such as memory 406 and network device 408, into a single integrated device. In other examples, the CPU and GPU are connected to each other via peripheral connectivity devices such as Fast Peripheral Component Interconnect (PCI) or other suitable peripheral data connections. In one example, the CPU is a commercially available central processing unit that implements an instruction set such as x86, ARM, Power, or the MIPS family of instruction sets.

[0063] Regardless of the specific circumstances, during operation, processor 404 executes stored program instructions retrieved from memory 406. The stored program instructions include software that controls the operation of processor 404 to implement the CLAP4SED process described herein. Processor 404 can execute complex algorithms involved in the contrastive learning process, preparing query prototypes, extracting audio embeddings, calculating similarity scores, and implementing real-time sound event detection.

[0064] Memory 406 may include both non-volatile memory and volatile storage devices. Non-volatile memory includes solid-state memory, such as NOR and NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the system is off or powered off. Volatile memory includes static and dynamic random access memory (RAM) that stores program instructions and data during the operation of the CLAP4SED system. Network device 408 may communicate with the sensor system to receive audio data and store it in memory 406. Alternatively, memory 406 may already contain audio data from the sensor system.

[0065] The GPU may include hardware and software for processing and displaying audio data, intermediate features, and detection results. Output device 410 is configured to present the results of the sound event detection process in a format understandable to a human operator. Output device 410 may include a graphics or visual display device, such as an electronic display screen, projector, printer, or any other suitable device for reproducing a graphic display. As another example, output device 410 may include an audio device, such as a speaker or headphones.

[0066] Input device 412 may include various devices that enable computing device 402 to receive control input from a user. Input device 412 enables the user to interact with the computing device to configure the CLAP4SED process, adjust detection thresholds, and improve operating parameters based on performance evaluation. Suitable examples of input devices that receive human interface input may include keyboards, mice, trackballs, touchscreens, voice input devices, graphics tablets, etc.

[0067] Network device 408 may each include various devices that enable the sending and receiving of data from external devices via a network. Examples of suitable network devices 408 include Ethernet interfaces, Wi-Fi transceivers, cellular transceivers, Bluetooth or BLE transceivers, UWB transceivers, or other network adapters or peripheral interconnection devices that receive data from other computers or external data storage devices, which may be useful for efficiently receiving large audio datasets.

[0068] This hardware configuration, especially when implemented with dedicated components such as the Ambarella CV22 chip, provides the necessary computational power and efficiency for real-time sound event detection. The combination of a powerful processor, efficient memory management, and diverse networking capabilities allows the CLAP4SED system to perform complex operations such as audio embedding extraction, similarity score calculation, and binary thresholding with minimal latency, making it suitable for deployment in resource-constrained embedded environments. The initial definition of abbreviations or other acronyms applies to all subsequent uses of the same acronym herein, and with necessary modifications to the details, applies to the normal grammatical variations of the originally defined acronym. Unless explicitly stated otherwise, the measurement of properties of the same attribute is determined by the same technique referenced before or after.

[0069] The initial definition of acronyms or other abbreviations applies to all subsequent uses of the same abbreviation herein, and, with necessary modifications, to the regular grammatical variations of the originally defined abbreviations. Unless explicitly stated otherwise, the measurement of an attribute is determined by the same technique referenced previously or later for the same attribute.

[0070] It should also be noted that, as used in the specification and appended claims, the singular forms “a,” “an,” and “the” include plural references unless the context clearly indicates otherwise. For example, references to a singular component are intended to include multiple components.

[0071] The term "comprising" is synonymous with "including," "having," "containing," or "characterized by." These terms are inclusive and open-ended, and do not exclude additional, unlisted elements or method steps. The phrase "consisting of" excludes any element, step, or ingredient not specified in the claim. When this phrase appears in a clause of the body of a claim, rather than immediately following the preamble, it limits only the element set forth in that clause; it does not exclude other elements from the claim as a whole. The phrase "essentially constitutes" limits the scope of the claim to the specified material or step, plus those material or steps that do not substantially affect the essential and novel characteristics of the claimed subject matter. The term "one or more" means "at least one," and the term "at least one" means "one or more." The terms "one or more" and "at least one" include "a plurality" as a subset.

[0072] Although exemplary embodiments have been described above, they are not intended to describe all possible forms covered by the claims. The language used in this specification is descriptive rather than restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form other embodiments of the invention that may not be explicitly described or shown. While various embodiments may have been described as offering advantages or being preferred over other embodiments or prior art implementations in one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to obtain desired overall system properties depending on the specific application and implementation. Therefore, if an embodiment is described as less preferred than other embodiments or prior art implementations in terms of one or more characteristics, these embodiments do not fall outside the scope of this disclosure and may still be desirable in a particular application.

Claims

1. A method for real-time sound event detection on an embedded device, comprising: pre-training a contrastive language-audio pre-training model as an audio base model; preparing offline multi-modal query prototypes for sound events of interest; deploying the pre-trained contrastive language-audio pre-training model and the prepared query prototypes on an embedded device; receiving an input audio stream on the embedded device; extracting audio embeddings from the input audio stream using the pre-trained contrastive language-audio pre-training model; computing similarity scores between the extracted audio embeddings and the prepared query prototypes; determining presence of a sound event based on the computed similarity scores; and outputting real-time sound event detection results. The pre-training a contrastive language-audio pre-training model comprises:

2. The method of claim 1, wherein, training an audio encoder and a text encoder to contrastively optimize symmetric similarity in a joint multi-modal space for audio-text pairs. The audio encoder is a lightweight parallel audio neural network architecture.

3. The method of claim 2, wherein, The preparing offline multi-modal query prototypes comprises:

4. The method of claim 1, wherein, extracting audio embeddings from few-shot audio samples for each sound event of interest; generating text prompts describing each sound event using a large language model; extracting text embeddings from the generated text prompts; and selecting the most relevant text embeddings based on similarity to the audio embeddings. The few-shot audio samples comprise N samples per sound event of interest, where N is a small positive integer.

5. The method of claim 4, wherein, The generating text prompts comprises rewriting a regular contrastive language-audio pre-training retrieval template to enrich text expressiveness using a large language model.

6. The method of claim 4, wherein, The computing similarity scores comprises:

7. The method of claim 1, wherein, computing dot products between the extracted audio embeddings and the prepared query prototypes. The determining presence of a sound event comprises:

8. The method of claim 1, wherein, applying binary thresholding to the computed similarity scores. The embedded device comprises a chip having a quad-core processor, a single-instruction multiple-data accelerator, and a computer vision stream vector processor.

9. The method of claim 1, wherein, 10. The method of claim 1, further comprising: preprocessing the input audio stream to generate a spectrogram before extracting audio embeddings.

11. A system for real-time sound event detection on an embedded device, comprising: a memory storing a pre-trained contrastive language-audio pre-training model and prepared multi-modal query prototypes; an audio input interface for receiving an input audio stream; a processor configured for: extracting audio embeddings from the input audio stream using the pre-trained contrastive language-audio pre-training model, computing similarity scores between the extracted audio embeddings and the prepared query prototypes, determining presence of a sound event based on the computed similarity scores, and outputting real-time sound event detection results. The pre-trained contrastive language-audio pre-training model comprises an audio encoder and a text encoder trained for contrastively optimizing symmetric similarity in a joint multi-modal space for audio-text pairs.

12. The system of claim 11, wherein, The audio encoder is a lightweight parallel audio neural network architecture.

13. The system of claim 12, wherein, ​ 14. The system of claim 11, wherein, The prepared multimodal query prototype includes an audio query vector and a text query vector for each sound event of interest.

15. The system of claim 14, wherein, The audio query vector is derived from a few-shot audio sample for each sound event of interest.

16. The system of claim 14, wherein, The text query vector is derived from a text prompt generated by a large language model that describes each sound event of interest.

17. The system of claim 11, wherein, The processor is further configured to: preprocess the input audio stream to generate a spectrogram prior to extracting audio embeddings.

18. The system of claim 11, wherein, The processor is configured to determine the presence of a sound event by applying binary thresholding to the computed similarity scores.

19. The system of claim 11, wherein, The embedded device includes a chip with a quad-core processor, a single instruction multiple data accelerator, and a computer vision stream vector processor.

20. A non-transitory computer-readable medium storing instructions that, when executed by a processor on an embedded device, cause the processor to implement real-time sound event detection by: loading a pre-trained contrastive language-audio pre-training model and a prepared multimodal query prototype; receiving an input audio stream; extracting audio embeddings from the input audio stream using the pre-trained contrastive language-audio pre-training model; computing similarity scores between the extracted audio embeddings and the prepared query prototype; determining the presence of a sound event based on the computed similarity scores; and outputting a result of real-time sound event detection.

Images