Open audio tracking system with audio base model and large language model

Abstract

A method of executing an open audio tracking system is disclosed. A feedback loop between an audio foundation model (AFM) and a large language model (LLM) enables both low-level sound event detection and high-level acoustic scene detection in real-time, which are then used to generate additional text-based event descriptions that are applied to subsequent iteration cycles of the system. The AFM can be similar to a contrastive language-audio pre-training (CLAP) model configured to detect sound events, while the LLM receives the detected given sound events and categorizes these events into acoustic sound classes that can interpret the sound events in the context of the environment.

Description

Technical Field

[0001] This disclosure relates to methods and systems for implementing audio tracking systems using machine learning techniques. Background Technology

[0002] Traditionally, identifying the source of acoustic content from recording devices relies on predefined, closed sets of audio classes. This limitation severely weakens the algorithm's capabilities, as a classifier for a specific acoustic scene can only identify sound events occurring within its training dataset. Given the diversity of sound events in various acoustic scenarios that humans or machines might encounter, such devices quickly become insufficient for practical applications. Summary of the Invention

[0003] In one embodiment, a method for performing an open audio tracking system is provided. The method includes: providing an audio segment and a text-based description to an audio foundation model (AFM), wherein the text-based description corresponds to a description of sound events to be detected by the AFM; performing the AFM to detect a subset of sound events present within the audio segment; providing a corresponding subset of the text-based description to a large language model (LLM); performing the LLM, wherein performing the LLM includes: classifying the audio segment into an acoustic scene category based on the detected subset of sound events; generating an additional text-based description corresponding to descriptions of other sound events involving the acoustic scene category; and providing the additional text-based description for subsequent iterative execution of the open audio tracking system.

[0004] In another embodiment, the system includes a processor and a memory containing instructions that, when executed by the processor, cause the processor to perform these steps.

[0005] In another embodiment, the non-volatile computer-readable medium includes instructions that, when executed by a processor, cause the processor to perform these steps. Attached Figure Description

[0006] Figure 1 The illustration shows a system for training and utilizing machine learning models, such as convolutional neural networks, according to some embodiments.

[0007] Figure 2 The illustration shows a computer-implemented method for training and utilizing machine learning models according to some embodiments.

[0008] Figure 3 The illustration is a schematic overview of an open audio tracking system according to some embodiments.

[0009] Figure 4The illustration is a schematic overview of a contrastive language-audio pre-trained (CLAP) model of an open audio tracking system according to some embodiments.

[0010] Figure 5A The illustration shows an example first iteration of an open audio tracking system according to some embodiments.

[0011] Figure 5B The illustration shows the execution according to some embodiments. Figure 5A The second iteration of the open audio tracking system introduced in [the document / document / etc.] is an example.

[0012] Figure 6 This is a flowchart illustrating the process of performing an open audio tracking system according to some embodiments.

[0013] Figure 7 The illustration is a schematic diagram of the interaction between a computer-controlled machine and a control system according to some embodiments.

[0014] Figure 8 The illustration shows an amplifier and speaker configured to control a hearing aid device according to some embodiments. Figure 7 A schematic diagram of the control system. Detailed Implementation

[0015] Embodiments of this disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be enlarged or reduced to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to use the embodiments in various ways. As will be understood by those skilled in the art, the various features illustrated and described with reference to any figure may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments of typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desired for particular applications or implementations.

[0016] As used in this article, “a,” “one,” and “the” refer to both the singular and plural forms unless the context clearly indicates otherwise. For example, a “processor” programmed to perform various functions refers to one processor programmed to perform each of those functions, or more than one processor programmed together to perform each of those functions.

[0017] There are two main technologies for identifying the source of acoustic content. The first is low-level sound event detection (SED), which aims to track basic sound elements such as alarms, human speech, and dog barking over time. Practical applications of SED include automatically detecting or alarming specific events, such as gunfire or attack detection on surveillance cameras. The second technology is acoustic scene classification (ASC), which focuses on a higher-level understanding of a more comprehensive acoustic environment, which may consist of multiple overlapping sounds. Practical applications of ASC include context awareness in smart devices or scene analysis in smart homes and smart cities.

[0018] However, past implementations of machine learning and modeling schemes for SED and ASC systems relied on predefined, closed sets of audio classes. For example, a classifier built with 10 predefined classes cannot handle any sound events outside that set. This limitation renders the models ineffective for dealing with dynamic and complex audio environments encountered in the real world, such as transitions from indoor to outdoor scenes. A major challenge is that scaling these models requires a considerable amount of labeled data and additional retraining or adaptation processes to achieve effective performance when new sound events need to be introduced. Therefore, this full-loop machine learning iteration is far from being able to respond in real-time to any practical, commercial, or business-oriented needs.

[0019] To overcome these challenges, this disclosure leverages audio and language foundational models to build a real-time, open-source audio tracking system. Such foundational models enable more generalized and widespread performance across a variety of downstream tasks. More specifically, audio foundational models, such as CLAP, enable zero-shot audio classification or retrieval via intuitive, free-form natural language queries, without requiring any predefined closed sets. Furthermore, LLM models, such as GPT-4, enable high-level reasoning, question answering, and knowledge summarization.

[0020] Compared to previous versions of AFM that were limited to handling basic acoustic concepts, such as single sound events, and therefore lacked complex scene reasoning and summarization, this invention applies a cascaded architecture of AFM and LLM. This enables the open audio tracking system to collaboratively implement not only low-level audio signal perception but also high-level acoustic scene reasoning. Therefore, this open audio tracking system is unrestricted in terms of either audio class or acoustic scene, allowing the system to operate within that scope. Furthermore, the model is configured to run in real time, thus enabling the open audio tracking system to be integrated into smart hearing aid devices, etc.

[0021] The following description continues with a general introduction to machine learning techniques related to the methods described herein for utilizing machine learning models. Various embodiments of cascaded AFM and LLM architectures and processes for open audio tracking systems are then discussed. This disclosure will also demonstrate the versatility of the methods and systems described herein for integration into hearing aid devices.

[0022] Figure 1 The illustration shows a system 100 for training and utilizing machine learning models, such as convolutional neural networks, according to some embodiments.

[0023] It should be understood that, although this article refers to... Figure 1 and Figure 2 The example embodiments given involve convolutional neural networks, but Figure 1 and Figure 2 Further embodiments can be applied to any other type of neural network-based or non-neural network-based machine learning model, which can be configured to be developed, trained, fine-tuned, and / or executed for various applications of audio tracking and interpretation further described herein.

[0024] In addition, compared to Figures 3-8 The time points illustrated in the diagrams are, for example, fully trained Open Audio Tracking System 300, AFM 306, Open Audio Tracking System 500, and Open Audio Tracking Subsystem 714. Figure 1 & Figure 2 This involves different, earlier moments in time. The training process of machine learning models, such as AFM and LLM, is described below, thus providing, for example, context for training AFM 306 and LLM 310. In particular, the encoder used within the AFM architecture described herein is flexible and can be configured to utilize different types of neural network architectures, such as Transformers or Convolutional Neural Networks.

[0025] In some embodiments, system 100 may include an input interface for accessing training data 102 for a convolutional neural network. For example, such as Figure 1 As illustrated, the input interface can be comprised of a data storage interface 104, which can access the training data 102 from the data storage device 106. For example, the data storage interface 104 can be a memory interface or a permanent storage interface, such as a hard disk or SSD interface, but it can also be a personal network, local area network, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be an internal data storage device of the system 100, such as a hard disk drive or SSD, but it can also be an external data storage device, such as a network-accessible data storage device.

[0026] In some embodiments, data storage device 106 may further include a data representation 108 of an untrained model version (e.g., a version of a machine learning model that has not yet been trained), which can be accessed by system 100 from data storage device 106. However, it will be appreciated that the training data 102 and data representation 108 of the pre-trained convolutional neural network may also be accessed from different data storage devices, for example, via different subsystems of data storage interface 104. Each subsystem may belong to the type described above for data storage interface 104. In other embodiments, the data representation 108 of the pre-trained convolutional neural network may be generated internally by system 100 based on the design parameters of the neural network, and therefore may be stored non-explicitly on data storage device 106. System 100 may further include a processor subsystem 110, which may be configured to provide an iterative function as a replacement for the layer stack of the convolutional neural network to be trained during operation of system 100. Hereinafter, the individual layers of the replaced layer stack may have mutually shared weights and may receive the output of the previous layer as input, or, for the first layer of the layer stack, receive the initial activation and a portion of the layer stack's input. The processor subsystem 110 may further be configured to iteratively train and / or fine-tune the convolutional neural network using training data 102 (e.g., thus generating an updated version of the machine learning model relative to the first "pre-trained" model version). In this document, the iterations of training performed by the processor subsystem 110 may include a forward propagation portion and a backward, or generative propagation portion.

[0027] System 100 may further include an output interface for outputting a data representation 112 of the trained convolutional neural network, and this data may also be referred to as trained model data 112. For example, as Figure 1 As illustrated in the diagram, the output interface can be comprised of a data storage interface 104, wherein in these embodiments, the interface is an input / output (“IO”) interface through which trained model data 112 can be stored in data storage device 106. For example, the data representation 108 defining a “pre-trained” convolutional neural network can be at least partially replaced by the data representation 112 of the trained neural network during or after training, because the parameters of the convolutional neural network (such as weights, hyperparameters, and other types of convolutional neural network parameters) can be adapted to reflect training on the training data 102. This also... Figure 1The figures are illustrated by reference numerals 108 and 112, which refer to the same data records on data storage device 106. In other embodiments, data representation 112 may be stored separately from data representation 108 defining a "pre-trained" convolutional neural network. In some embodiments, the output interface may be separate from data storage interface 104, but may generally have the type described above for data storage interface 104.

[0028] Figure 1 The system 100 shown is an example of a system that can be used to train and subsequently execute the trained machine learning model described herein.

[0029] Figure 2 The illustration depicts a computer-implemented method for training and utilizing a convolutional neural network according to some embodiments. System 200 may include at least one computing system 202. Computing system 202 may include at least one processor 204 operatively connected to memory unit 208. Processor 204 may include one or more integrated circuits implementing the functionality of a central processing unit (CPU) 206 and—in some embodiments—a graphics processing unit (GPU). CPU 206 may be a commercially available processing unit implementing an instruction set such as x86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. Stored program instructions may include software controlling the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 may be a system-on-a-chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interface, and input / output interface into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of the operation.

[0030] Memory cell 208 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, memory cell 208 may store a machine learning model 210 or algorithm, a training dataset 212 for the machine learning model 210, and the original source dataset 214.

[0031] The computing system 202 may include a network interface device 220 configured to provide communication with external systems and devices. For example, the network interface device 220 may include a wired and / or wireless Ethernet interface defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 220 may include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 220 may further be configured to provide a communication interface to an external network 222 or the cloud.

[0032] External network 222 may be referred to as the World Wide Web or the Internet. External network 222 can establish standard communication protocols between computing devices. External network 222 can allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 can communicate with external network 222.

[0033] The computing system 202 may include an input / output (I / O) interface 216, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 216 may include an additional serial interface (e.g., a Universal Serial Bus (USB) interface) for communicating with external devices.

[0034] The computing system 202 may include a human-machine interface (HMI) device 216, which may include any device enabling the system 200 to receive control input. Examples of input devices may include HMI inputs such as a keyboard, mouse, touchscreen, voice input device, and other similar devices. The computing system 202 may include a display device 226. The computing system 202 may include hardware and software for outputting graphical and textual information to the display device 226. The display device 226 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 202 may further be configured to allow interaction with remote HMIs and remote display devices via a network interface device 220.

[0035] System 200 can be implemented using one or more computing systems. Although the example depicts a single computing system 202 implementing all the described features, it is intended that the various features and functions can be implemented separately by multiple computing units communicating with each other. The specific system architecture chosen can depend on a variety of factors.

[0036] System 200 can implement machine learning algorithm 210 configured to analyze raw source dataset 214. Raw source dataset 214 may include raw or unprocessed sensor data, which may represent the input dataset for the machine learning system. In some examples, machine learning algorithm 210 may be a convolutional neural network algorithm designed to perform a predetermined function. For example, the neural network algorithm may be configured to receive audio segments and text-based event descriptions, as further described below in the case of AFM306.

[0037] Computer system 200 may store a training dataset 212 for machine learning algorithm 210. Training dataset 212 may represent a collection of previously constructed data used to train machine learning algorithm 210. Machine learning algorithm 210 may use training dataset 212 to learn weighting factors associated with a neural network algorithm. Training dataset 212 may include a source dataset having a corresponding outcome or result that machine learning algorithm 210 attempts to replicate through the learning process.

[0038] Machine learning algorithm 210 can run in learning mode using training dataset 212 as input. Machine learning algorithm 210 can be executed in multiple iterations using data from training dataset 212. With each iteration, machine learning algorithm 210 can update its internal weighting factors based on the obtained results. For example, machine learning algorithm 210 can compare its output results (e.g., annotations) with those included in training dataset 212. Since training dataset 212 includes expected results, machine learning algorithm 210 can determine when performance is acceptable. After machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistency with results associated with training dataset 212), machine learning algorithm 210 can be executed using data not in training dataset 212. The trained machine learning algorithm 210 can be applied to new datasets to generate annotated data.

[0039] Machine learning algorithm 210 can be configured to identify specific features in raw source data 214. Raw source data 214 may include multiple instances or input datasets for which desired annotation results are desired. Machine learning algorithm 210 can be programmed to process raw source data 214 to identify the presence of specific features. Machine learning algorithm 210 can be configured to identify features in raw source data 214 as predetermined features. Raw source data 214 can be derived from various sources. For example, raw source data 214 may be actual input data collected by a machine learning system. Raw source data 214 may be machine-generated for testing systems. As an example, raw source data 214 may include audio clips related to the surrounding audio environment and text-based event descriptions.

[0040] In this example, machine learning algorithm 210 can process the raw source data 214 and output an indication of "which of the text-based event descriptions is supported by the audio signal within the audio segment". Machine learning algorithm 210 can generate a confidence level or confidence factor for each generated output. For example, a confidence value exceeding a predetermined high confidence threshold can indicate that machine learning algorithm 210 is confident that the identified feature corresponds to a specific feature; a confidence value below a low confidence threshold can indicate that machine learning algorithm 210 is less confident that a specific feature exists.

[0041] Figure 3 The illustration is a schematic overview of an open audio tracking system according to some embodiments.

[0042] As illustrated in the Open Audio Tracking System 300, the framework comprises two main modules, one of which feeds into the other: (1) an AFM 306, which receives audio segments 304 and text-based event descriptions from a database 316 and outputs detected sound events 308, which are then provided to (2) an LLM 310. The LLM 310 is configured to then process the detected sound events 308 and perform high-level acoustic scene inference to expand the text-based event description database 316 with additional text-based event descriptions for the next iteration cycle of the Open Audio Tracking System 300. As the output of the AFM 306 is fed into the LLM 310, which then provides outputs back to the AFM 306, the Open Audio Tracking System 300 is configured to operate at least in near real-time. Furthermore, even when the audio scene changes over time (e.g., from an indoor kitchen scene to an outdoor baseball game scene), alternatively, at least in part based on the use of the described feedback loop, neither the trained AFM 306 nor the already trained LLM 310 undergoes additional retraining. Figure 3 The architecture shown is untrained, and the open audio tracking system 300 is configured to iteratively and dynamically adapt to the relevant acoustic and sound scenes and events as it receives input audio segments.

[0043] In some embodiments, AFM 306 can be implemented as a CLAP module, and is also referred to herein as a CLAP4SED module or a CLAP module for Sound Event Detection (SED). CLAP can be defined as a contrastive learning model that compares text data samples with audio-based data samples. As a foundational audio-language model, CLAP is pre-trained on audio data and corresponding language descriptions such as audio subtitles, tags, or titles using a contrastive objective, also referred to herein as the objective loss (see above). Figure 1 and Figure 2 (The description provided).

[0044] As described in further detail below, after providing the AFM 306 with an initial seed 302 of text-based event descriptions when first using the open audio tracking system 300, the LLM 310 then expands these samples for each future iteration of the audio tracking system 300. Therefore, the method and system described herein, which simultaneously utilizes the AFM 306 and the LLM 310 for audio tracking, reduces the time and resources otherwise required to train the AFM 306 on specific audio environments, while enabling the realization of the unlimited potential of the LLM without limiting the audio tracking system to a specific language and / or audio environment.

[0045] When AFM 306 is implemented as a CLAP model, it can be further constrained by the following equation, where the target loss... Designed to align audio embeddings With text embedding The latent space is mapped using a modality-specific neural encoder to establish meaningful associations between them. In some embodiments, and ,in Represents the potential spatial dimension.

[0046] CLAP also enables multi-directional interactions within the model. For example, if given an audio signal as input, CLAP can perform audio labeling or subtitle addition, known as "audio input, text output." Conversely, if queries are made with audio and free-form natural language prompts, CLAP can be configured to perform tasks such as audio retrieval and zero-shot audio classification, known as "audio and text input, recognition output."

[0047] Therefore, the CLAP4SED module utilizes the CLAP model for Sound Event Detection (SED) and is configured to track the activity of specific sound events over time. Unlike traditional classification tasks, SED is configured to identify the start and end of active periods of an event. This is achieved by processing the real-time audio stream into small data segments or "chunks," which can be a window size of a few seconds, rather than receiving the entire audio segment at once. The CLAP model is then configured to advance a small incremental buffer or a window jump size of approximately 100-500 ms in time, thus collectively allowing for the streaming of audio in chunks.

[0048] CLAP is also configured to compute on encoded text-based descriptions. (Where N represents the total number of queries in the set) and the encoded segments of the audio clip. The cosine similarity between them. As mentioned above, a text query or prompt can be defined as a natural language description associated with a possible sound event within an audio signal. For example, a text query could include "microwave oven" or "the wind rustling through the leaves".

[0049] Once the cosine similarity between the encoded text-based description and the audio segment is calculated, AFM306 is configured to output a subset of sound events that are missing within a given audio segment or fragment. For example, Figure 3 The diagram illustrates the sound event detection block 308: Event 1 is detected by the CLAP model of AFM 306 for the start and end of a given time interval; while Event 2 is detected for a longer start and end of a given time interval within the total duration defined by a given segment of audio segment 304 analyzed by AFM 306. Furthermore, Event 3 is detected for the entire duration defined by the given segment of audio segment 304. In some embodiments, Events 1 and 2 may also be referred to as “active” events because the calculated cosine similarity is higher than a given threshold. Event 3, on the other hand, may be referred to as an “inactive” event because the calculated cosine similarity is lower than a given threshold.

[0050] The analysis information within sound event detection block 308 is then provided to LLM 310. In some embodiments, a subset of text-based descriptions corresponding to active events present within the audio segment or fragment is provided to LLM 310 in a structured or natural text format. For example, continuing with the specific example shown in sound event detection block 308, the analysis may be provided to LLM 310 in JSON format, such as [{'label':'event1','start':1.2,'end':4.5},{'label':'event2','start':3.0,'end':6.5}]. This structured format indicates that AFM 306 detected the sounds of event 1, which begins at 1.2 seconds and ends at 4.5 seconds, and event 2, which begins at 3.4 seconds and ends at 6.5 seconds. This process enables low-level sound event tracking over time based on expanded, text-based event descriptions, as will be further described below.

[0051] like Figure 3As shown, the large language model 310 is configured to receive a subset of text-based descriptions corresponding to active sound events, output (1) an acoustic scene classification 314, and generate (2) additional text-based event descriptions 312. In some embodiments, the LLM 310 may be implemented as a generative pre-trained transformer (GPT) LLM, such as ChatGPT. However, in other embodiments, the LLM 310 may also be implemented as any other large language model configured to receive a subset of text-based descriptions corresponding to active sound events, output (1) an acoustic scene classification 314, and generate (2) additional text-based event descriptions 312.

[0052] The first of the two outputs provides a summary of the high-level acoustic scene, from which LLM infers a subset of text-based descriptions that may originate from the scene. This classification defines acoustic scene categories such as “kitchen,” “neighborhood park,” or some other descriptive language based on sound event detection block 308.

[0053] Acoustic scene categories can include any high-level description of a local environment. Further examples of acoustic scene categories are provided in... Figure 5A , 5B Provided in the relevant descriptions in this article.

[0054] In some embodiments, one or more prompts may be generated and provided to the LLM 310 to achieve this functionality. For example, a first prompt may be used to determine the possible local environment of an audio segment based on a subset of detected sound events provided to the LLM. A second prompt or instruction may be used to expand the subset of detected sound events, as further described below.

[0055] The second of the two outputs, namely the additional text-based event description 312, refers to an expansion of the list of text-based event descriptions already stored in the database 316. For example, if the database 316 already includes text-based descriptions related to the neighborhood park acoustic scene (e.g., “children laughing”, “ice cream truck”), then the LLM 310 can be configured to expand sound events that might be recognized by the AFM 306, such as “dog barking” and “swing noise”, when provided with audio clips of the neighborhood park acoustic scene. Figure 5A and 5B Further examples of expanding text-based event descriptions are provided in the relevant descriptions of this article.

[0056] These additional text-based event descriptions 312, along with the acoustic scene classification 314, are then stored in a text-based event description database 316 and subsequently provided to the AFM 306 during the next iteration of the Open Audio Tracking System 300. Thus, from one iteration to the next, the Open Audio Tracking System 300 resembles an adaptive method for identifying not only more comprehensive audio tracking frameworks but also more detailed ones.

[0057] In some embodiments, the additional text-based event description 312 may be labeled as corresponding to an existence in the acoustic scene category 314 before being stored in the text-based event description database 316. For example, continuing with the example introduced above, "dog barking" and "swing noise" may be labeled as occurring in the neighborhood park acoustic scene.

[0058] Furthermore, according to some embodiments, the large language model 310 can also be configured to detect false positives that occur during the detection of sound events in an audio segment by the AFM 306. Specifically, a false positive sound event is determined to be unlikely to correspond to an aggregated local context of other sound events within a subset of sound events received by the large language model 310. In such cases, the false positive is removed from the subset of sound events before any additional sound events are stored in the text-based event description database 316. For example, continuing the example introduced above: if text-based descriptions such as "children's laughter," "ice cream truck," and "kitchen mixer" are detected by the AFM 306 and provided to the LLM 310 as part of the sound event detection block 308, the LLM 310 can determine that "kitchen mixer" is a false positive event in the subset because the aggregated local context features of other sound events may indicate that the audio segment 304 represents a neighborhood park.

[0059] Figure 4 The illustration is a schematic overview of a model of the open audio tracking system CLAP according to some embodiments.

[0060] In some embodiments, AFM 306 may be implemented and executed as a CLAP model and by utilizing one or more components of system 200, such as computing system 202. AFM 306 receives both text data samples 400 and specific portions 404 of audio segments as input. Each text input may be a word, phrase, or sentence and is linked or paired with an associated audio signal that is expected to be present in that segment. For example, the text input may be a text-based description of an event such as "wind sound," "microwave oven," or "people shouting," which may be present in the current audio segment 304 or may have already existed in previously received audio segments.

[0061] CLAP's implementation of AFM 306 utilizes contrastive learning to generate a joint multimodal space for audio and text descriptions. CLAP receives audio and text pairs, processes them through independent encoders, and maps their representations to the joint space using linear projection. Specifically, CLAP uses a dual-encoder architecture with a text encoder 402 and an audio encoder 406 to connect the language and audio representations. This approach aims to enable zero-shot predictions without requiring predefined categories during either model training or execution. The two representations are concatenated in the joint multimodal space via linear projection. This space is learned using contrastive learning with the similarity / difference between audio and text pairs in a batch, and is generally shown at 408.

[0062] like Figure 4 As shown, contrastive learning typically proceeds as follows. Initially, text data 400 and audio data 404 are processed separately by dedicated encoders, resulting in text embeddings and audio embeddings, respectively. These embeddings capture the core features or representations of the corresponding data. Simultaneously, several unrelated or significantly different text and audio segments can be fed into the encoder. The embedding vectors are mapped to a joint space using a learnable linear projection, which is used to compare and align the audio and text representations. In the example, the text encoder 402 produces embeddings with features... The text vector is then processed by the audio encoder 406, which generates a text vector with specific features. The audio vector.

[0063] Once the embedding vectors are in the joint space, the model computes the similarity between the embeddings of the audio-text pairs. Similarity can be measured using various metrics, such as cosine similarity or Euclidean distance. For example, the model might assess how close or far the audio representation is from its corresponding text representation in the joint space. Contrastive learning uses a loss function that prompts the model to bring similar pairs closer together while pushing dissimilar pairs apart. It computes the loss based on the similarity between positive sample pairs (audio-text pairs that belong together) and negative sample pairs (audio-text pairs that do not correspond to each other). This prompts the model to learn representations that make similar and dissimilar pairs more distinguishable. The diagonal of the matrix 408 resulting from the dot product operation shows paired audio and text based on their similarity, while the off-diagonal portion presents unpaired audio-text features (e.g., the sound of a person shouting versus the text sample that says "someone is whispering"). Therefore, when AFM 306 is implemented as a CLAP model, its contrastive learning approach aims to minimize the contrastive loss by adjusting model parameters such as the encoder and projection layers. CLAP is thus able to learn to capture the substantial correlation between audio and text representations, effectively learning to associate relevant text-based descriptions with corresponding audio signals.

[0064] Figure 5A and 5BThe illustrations show example first and second iteration loops for implementing an open audio tracking system according to some embodiments.

[0065] As mentioned above Figure 3 Similar to the Open Audio Tracking System 300, the Open Audio Tracking System 500 illustrates another embodiment, wherein the framework includes an AFM 506 and an LLM 510 operating in a feedback loop. In the following description, Figure 5A This can be considered the first iteration cycle of the open audio tracking system 500, at which point the text-based event description database 516... Figure 5A The time points shown are empty. Figure 5B This can be considered a subsequent, or second, iteration of the open audio tracking system 500.

[0066] like Figure 5A As shown, an initial seed 502 of text-based event descriptions is provided to the text-based event description database 516 of the open audio tracking system 500. In some embodiments, the initial seed 502 may include a small number of initial text-based event descriptions for which the open audio tracking system 500 will begin tracking. It should be understood that "wind" and "microwave oven" are merely illustrative examples, and larger or smaller numbers of initial text-based event descriptions may be used. Furthermore, the initial seed 502 may refer to words, phrases, or sentences describing event sounds in various acoustic scenarios. Text-based descriptions may include sound events caused by humans, animals, machines, or natural events (e.g., wind, thunder, rain, etc.).

[0067] Subsequently, the first segment 504 of the given audio clip, along with the text-based event description from database 516, is provided to AFM 506. This module encodes the first segment 504 of the audio clip into an audio embedding and the text-based event description from database 516 into a text embedding. In an embodiment where AFM 506 is implemented as a CLAP model, the cosine similarity between the embedding vectors is calculated to determine whether a sound event corresponding to the text-based event description in database 516 exists within the first segment 504 of the given audio clip.

[0068] like Figure 5A The specific embodiment shown in the sound event detection module 508 indicates the start and end times of the detected "microwave oven" event within the time span of a given audio segment first section 504, while the "wind sound" event is not detected at all.

[0069] The start and end times of "microwave oven" and its text-based event description "microwave oven" are then provided to the LLM 510. The execution process of the LLM 510 includes determining the acoustic scene classification 514 of the given audio segment first section 504 as "kitchen" based on learning from the detected microwave oven sound.

[0070] The execution of LLM 510 also includes generating various other text-based event descriptions 512, which correspond to other sound events that may exist in the "kitchen" acoustic scene classification 514. In the specific embodiment shown, LLM 510 generates "washing dishes," "frying," and "cleaning" in the additional text-based event descriptions 512.

[0071] These additional text-based event descriptions 512 are then stored in a text-based event description database 516 and labeled with the tag "kitchen" acoustic scene category.

[0072] Therefore, the open audio tracking system 500 has completed its first iteration cycle, as... Figure 5B As shown, system 500 continues in the loop to provide a new round of text-based event descriptions and a second segment of a given audio clip to automatic audio matcher 506.

[0073] exist Figure 5B In this implementation, the second segment 550 of a given audio segment, along with a text-based event description, is provided to AFM 506. Database 558 refers to an updated version of database 516, which already stores the additional text-based event description 512. AFM 506 encodes the second segment 550 of the given audio segment into an audio embedding and encodes the text-based event description in database 558 into a text embedding. In an embodiment where AFM 506 is implemented as a CLAP model, the existence of the sound event corresponding to the text-based event description in database 558 in the second segment 550 of the given audio segment is determined by calculating the cosine similarity between the embedding vectors.

[0074] As in Figure 5B The sound event detection module 552, as illustrated in the specific embodiment, detects the "microwave oven" event at the beginning and end of a given time, the "frying" event at the beginning and end of another time, while the "wind sound" is not detected throughout the entire duration of the second segment 550 of the given audio segment, and the "washing dishes" and "washing" events are also not detected.

[0075] The start and end times of "microwave oven" and its text-based event description "microwave oven", as well as the start and end times of "frying" and its text-based event description "frying", are then provided to the LLM 510. The execution of the LLM 510 includes: based on the learning of the detected microwave oven sound and frying sound, determining that the acoustic scene classification 556 of the given audio segment second section 550 is still "kitchen".

[0076] The execution of LLM 510 also includes generating various additional text-based event descriptions 554, which correspond to additional sound events that may exist in the "kitchen" acoustic scene classification 556. In a specific embodiment, as shown by the additional text-based event descriptions 554, LLM 510 generates and outputs "coffee machine" and "eating".

[0077] These additional text-based event descriptions 554 were then stored in a text-based event description database 558 and labeled with the "kitchen" acoustic scene category tag.

[0078] The second iteration of the open audio tracking system 500 is now complete. The system 500 will continue in the loop to provide the AFM 506 with a new round of text-based event descriptions and a third segment of a given audio clip, and so on.

[0079] At a later time point, when the AFM 506 detects a text-based event description of "wind sound," the LLM 510 may adjust the acoustic scene classification, for example, to "city street" or another outdoor scene classification. In this case, the system will generate additional text-based event descriptions related to "city street," such as "dog barking," "vehicle passing sound," or "birdsong."

[0080] Because AFM and LLM are pre-trained, the Open Audio Tracking System 500 can dynamically adapt to newly introduced scenarios even if the acoustic scene classification changes drastically (e.g., from "kitchen" to "city street"). The Open Audio Tracking System 500 is self-contained (e.g., requires no human intervention) without the need for additional retraining.

[0081] Figure 6 This is a flowchart depicting the execution flow of an open audio tracking system according to some embodiments. In some embodiments, flow 600 can be used to describe a specific iteration cycle of the open audio tracking system 300. As indicated by the arrows between flow blocks 650 and 610, flow 600 can be repeated, as described above in iterations #1 and #2 of the open audio tracking system 500.

[0082] In block 610, audio segments (or segments) are provided to the AFM (e.g., CLAP) along with text-based descriptions from a text-based sound event description database. These text-based descriptions correspond to sound events that should be detected (or not detected) by CLAP using cosine similarity calculation.

[0083] In block 620, AFM is performed to detect one or more sound events present within the audio segment, wherein the set of events comes from the set of text-based descriptions described in block 610.

[0084] In block 630, a subset of text-based descriptions is provided to a large language model (LLM). As shown in block 640, the model is configured to classify the audio segments into acoustic scene categories and generate additional text-based descriptions that relate to other potential sound events that may occur within that acoustic scene category.

[0085] In block 650, the additional text-based descriptions are stored in the sound event database and are accessed for future iterations when text-based descriptions and audio clips are provided to the AFM for another iteration of the Open Audio Tracking System.

[0086] Figure 7 The illustration is a schematic diagram of the interaction between a computer-controlled machine and a control system according to some embodiments.

[0087] The methods and systems disclosed in this paper can be used in a variety of scenarios. This section provides some practical applications of the proposed system.

[0088] As a first example, open audio tracking systems can be implemented in context-aware smart devices. Open acoustic scene detection systems can be embedded in existing edge hardware devices to provide additional context-aware capabilities to assist in automated intelligent decision-making. For example, hearing aid devices often require users to manually adjust microphone settings for the best experience [2]. However, such temporary adjustments pose additional challenges for elderly or child users, who may have difficulty remembering and managing different configurations. Integrated open acoustic scene detection systems can automatically adjust predetermined configurations based on the detected scene, thereby providing an optimized user experience.

[0089] As a first example, the open audio tracking system can track low- and high-frequency audio content in near real-time or real-time, providing a comprehensive audio analysis solution. In specific scenarios, the system can be used in conjunction with large language models (LLMs) to query audio tracking results, enabling functions such as audio-based question-and-answer localization of specific events, inference of event sequences, or retrieval of time-series anomalies. This technology can also be applied to surveillance cameras to monitor critical events such as shootings and violence in real time.

[0090] As a second example, open audio tracking systems can be integrated into context-aware smart devices. Open acoustic scene detection systems can be embedded in existing edge hardware devices to provide additional context-aware capabilities, facilitating intelligent automated decision-making. For instance, traditional hearing aids often require users to manually adjust microphone settings for optimal experience when switching between different acoustic scenes. However, this volatile adjustment poses an additional challenge for elderly or child users—they often struggle to remember and manage different configurations, especially in time to capture key cues in different environments. Integrated open acoustic scene detection systems, on the other hand, can automatically adjust predetermined configurations based on the detected acoustic scene, thus providing a more optimized user experience. Hearing aid device 800 and the description below provide further examples of such integration.

[0091] Figure 7 A schematic diagram depicting the interaction between a computer-controlled machine 700 and a control system 702 is shown. The computer-controlled machine 700 includes actuators 704 and sensors 706. Actuators 704 may include one or more actuators, and sensors 706 may include one or more sensors. Sensors 706 may be configured to sense ID and / or OOD data, and a corresponding processor may be configured to determine whether the data is ID or OOD according to the teachings herein. Sensors 706 may be configured to sense the condition of the computer-controlled machine 700. Sensors 706 may be configured to encode the sensed condition into a sensor signal 708 and transmit sensor signal 708 to the control system 702. Non-limiting examples of sensors 706 include microphones, cameras, video sensors, optical sensors, etc. In an embodiment, sensor 706 is configured as a microphone, configured to receive audio signals from the environment near the computer-controlled machine 700.

[0092] The control system 702 is configured to receive sensor signals 708 from a computer-controlled machine 700. As described below, the control system 702 may be further configured to calculate an actuator control command 710 based on the sensor signals and transmit the actuator control command 710 to the actuator 704 of the computer-controlled machine 700.

[0093] like Figure 7As shown, the control system 702 includes a receiving unit 712. The receiving unit 712 can be configured to receive sensor signals 708 from sensor 706 and transform the sensor signals 708 into input signals x. In an alternative embodiment, the sensor signals 708 are received directly as input signals x, without a receiving unit 712. Each input signal x may be a portion of each sensor signal 708. The receiving unit 712 can be configured to process each sensor signal 708 to generate each input signal x. The input signals x may include data corresponding to an image recorded by sensor 706. For example, image-based data samples and text-based data samples may be received by the receiving unit 712.

[0094] The control system 702 includes an open audio tracking subsystem 714. The open audio tracking subsystem 714 can be configured to detect sound events within an audio signal received by the sensor 706. Its parameterization is achieved by parameters, such as those described above (e.g., parameter θ). Parameter θ can be stored in and provided by a non-volatile memory 716. The open audio tracking subsystem 714 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). Parameter θ can be stored in and provided by a non-volatile memory 716. The open audio tracking subsystem 714 is configured to determine an output signal y from an input signal x. Each output signal y includes information assigning one or more tags to each input signal x. The open audio tracking subsystem 714 can transmit the output signal y to a conversion unit 718. The conversion unit 718 is configured to convert the output signal y into an actuator control command 710. The control system 702 is configured to transmit actuator control command 710 to actuator 704, which is configured to actuate computer-controlled machine 700 in response to actuator control command 710. In another embodiment, actuator 704 is configured to actuate computer-controlled machine 700 directly based on output signal y.

[0095] When actuator 704 receives actuator control command 710, actuator 704 is configured to perform an action corresponding to the relevant actuator control command 710. Actuator 704 may include control logic configured to transform actuator control command 710 into a second actuator control command for controlling actuator 704. In one or more embodiments, instead of an actuator or otherwise, actuator control command 710 may be used to control a display.

[0096] In another embodiment, instead of or attached to the computer-controlled machine 700 including sensor 706, control system 702 includes sensor 706. Control system 702 may also include actuator 704 instead of or attached to the computer-controlled machine 700 including actuator 704.

[0097] like Figure 7 As shown, the control system 702 further includes a processor 720 and a memory 722. The processor 720 may include one or more processors. The memory 722 may include one or more memory devices. An open audio tracking subsystem 714 of one or more embodiments may be implemented by the control system 702, which includes a non-volatile storage device 716, a processor 720, and a memory 722.

[0098] Non-volatile storage device 716 may include one or more permanent data storage devices, such as hard disk drives, optical disk drives, magnetic tape drives, non-volatile solid-state devices, cloud storage devices, or any other device capable of permanently storing information. Processor 720 may include one or more devices selected from a high-performance computing (HPC) system, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory 722. Memory 722 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information. Furthermore, processor 720 and memory 722 may be configured to provide acquired data to one or more other computing devices, which also... Figure 8 The domain-specific embodiment shown is configured to execute an open audio tracking subsystem. Such acquired data can be used to generate training and validation datasets for various stages of the preparation and execution of machine learning models in industry-level applications. In the scenario described herein for executing the open audio tracking system, processor 720 and memory 722 may be coupled to or otherwise remotely connected to computing devices that then perform the audio tracking process as described above.

[0099] Processor 720 may be configured to read into memory 722 and execute computer-executable instructions residing in non-volatile storage device 716, and embody one or more machine learning algorithms and / or method techniques of one or more embodiments. Non-volatile storage device 716 may include one or more operating systems and applications. Non-volatile storage device 716 may store compilations and / or interpretations of computer programs created using various programming languages ​​and / or techniques, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, individually or in combination.

[0100] When the program instructions are executed by the processor 720, the computer-executable instructions of the non-volatile storage device 716 can cause the control system 702 to implement one or more machine learning algorithms and / or method techniques as disclosed herein. The non-volatile storage device 716 may also include machine learning data (including data parameters) that support the functionality, features, and processes of one or more embodiments described herein.

[0101] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon, for causing a processor to perform aspects of one or more embodiments. Essentially non-transitory computer-readable storage media can include tangible media, implemented in any method or technology, that are volatile and non-volatile, as well as removable and non-removable, for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, portable compact disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded via a network to an external computer or external storage device.

[0102] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other type of programmable data processing apparatus, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of writing that includes instructions to implement the functions, actions, and / or operations specified in a flowchart or diagram. In alternative embodiments, consistent with one or more embodiments, the functions, actions, and / or operations specified in the flowcharts and diagrams can be reordered, processed sequentially, and / or processed concurrently. Furthermore, any flowchart and / or diagram may include more or fewer nodes or blocks than illustrated consistent with one or more embodiments.

[0103] Processes, methods, or algorithms may be embodied, in whole or in part, using appropriate hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0104] Figure 8 The figure is a schematic diagram of a control system configured to control an amplifier and a speaker in a hearing aid device according to some embodiments.

[0105] In some embodiments, the open audio tracking subsystem 714 may be integrated into the hearing aid device 800. For example... Figure 8 As shown, the hearing aid device 800 may include a sensor (e.g., a microphone 802) configured to detect audio signals from the environment near the device. The detected audio signals are then provided to an open audio tracking subsystem 714 of the control system 702, wherein audio segments of the audio signals, along with various text-based event descriptions, are provided to AFM 812. AFM 812 is then executed to detect a subset of given sound events present within said audio segments.

[0106] The subset of sound events is then fed into a large language model (LLM) 814, which classifies the audio segments into acoustic scene categories based on the detected subset of sound events and generates additional text-based descriptions corresponding to other sound event descriptions associated with that classified scene category. These additional text-based descriptions are ultimately stored in a sound event description database 816.

[0107] In some embodiments, the control system 702 may be configured to provide acoustic scene classification, thereby retrieving predefined parameters from the device memory from acoustic scene-specific parameters 808, which relate to the use of the hearing aid device in an environment matching the acoustic scene category, and providing them to the receiver of the hearing aid device 800, such as amplifier 804, and extendedly, speaker 806.

[0108] In other embodiments, the control system 702 may then be configured to update the signal-to-noise ratio based on a subset of detected sound events and provide the updated signal-to-noise ratio to the amplifier 804 and the speaker 806.

[0109] While exemplary embodiments have been described above, they are not intended to describe all possible forms included in the claims. The language used in this specification is descriptive rather than limiting, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form other embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing advantages over or preferred over other embodiments or prior art implementations in one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Accordingly, any embodiment described as less desirable than other embodiments or prior art implementations in one or more characteristics is not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A hearing aid device, comprising: A microphone configured to detect audio signals; processor; and A memory that stores program instructions, which, when executed by a processor, causes the processor to: Receive audio signals from the microphone; The audio signal is provided to the Audio Foundation Model (AFM) along with audio segments and text-based descriptions, wherein the text-based descriptions are stored in memory and correspond to descriptions of sound events to be detected by the AFM. Perform AFM to detect a subset of sound events present within the audio segment; Provide a corresponding subset of text-based descriptions to large language models (LLMs); Execute the LLM, where the execution of the LLM includes: The audio segments are classified into acoustic scene categories based on the detected subset of sound events; and Generate additional text-based descriptions corresponding to other sound event descriptions involving the aforementioned acoustic scene category; and The additional text-based description is provided for use in another iteration of AFM by means of another audio segment.

2. The hearing aid device of claim 1, wherein the program instructions further cause the processor to: Update the signal-to-noise ratio based on a subset of detected sound events; and An updated signal-to-noise ratio is provided to the speaker of the hearing aid device.

3. The hearing aid device of claim 1, wherein the program instructions further cause the processor to: Retrieve predefined parameters from the memory relating to the use of the hearing aid device within the acoustic scene category; and The predefined parameters are provided to the receiver of the hearing aid device.

4. The hearing aid device of claim 1, wherein the program instructions further cause the processor to: Provides additional text-based descriptions to be stored in memory; and In response to the receipt of another audio segment, the text-based description, the additional text-based description, and the additional audio segment are provided to the AFM for execution.

5. The hearing aid device of claim 4, wherein when an additional text-based description is provided to be stored in memory, the program instructions further cause the processor to label the additional text-based description as corresponding to an existence in an acoustic scene category.

6. The hearing aid device of claim 1, wherein the text-based description corresponding to the description of the sound event includes a description of the sound event caused by a human, animal, or machine.

7. The hearing aid device of claim 1, wherein the acoustic scene category includes a high-level description of the local environment of the hearing aid device during the duration of the audio segment.

8. A computer-implemented method for performing an open audio tracking system, the method comprising: The audio base model (AFM) is provided with audio segments and text-based descriptions, wherein the text-based descriptions correspond to descriptions of sound events to be detected by the AFM; Perform AFM to detect a subset of sound events present within the audio segment; Provide a corresponding subset of text-based descriptions to large language models (LLMs); Executing an LLM, which includes: The audio segments are classified into acoustic scene categories based on the detected subset of sound events; and Generate additional text-based descriptions corresponding to other sound event descriptions involving the aforementioned acoustic scene category; and The additional text-based description is provided for use in another iteration of the open audio tracking system.

9. The computer-implemented method according to claim 8, further comprising: Provide additional text-based descriptions to be stored in the event description database; and In response to the receipt of another audio segment, the text-based description, the additional text-based description, and the additional audio segment are provided to the AFM for execution.

10. The computer-implemented method according to claim 9, further comprising: Before providing additional text-based descriptions to be stored in the event description database, the additional text-based descriptions are labeled as corresponding to those existing in the acoustic scene category.

11. The computer-implemented method of claim 8, wherein performing AFM comprises: Encode text-based descriptions; Encode segments of an audio clip; Calculate the cosine similarity between the encoded text-based description and the encoded audio segment; and When the corresponding cosine similarity is higher than the threshold, it is determined that the given sound event exists within the audio segment.

12. The computer-implemented method of claim 11, wherein performing AFM further comprises: Determine the start and end times of a given sound event; and The start and end times are additionally provided to the LLM for execution.

13. The computer-implemented method of claim 8, wherein AFM is a contrastive language-audio pre-trained (CLAP) model.

14. The computer-implemented method of claim 8, further comprising generating a prompt provided to the LLM for execution, wherein the prompt includes: Based on the provided and detected subset of sound events, determine the possible local environment of the audio segment as a first instruction; and The second instruction expands the subset of detected sound events.

15. The computer-implemented method of claim 8, wherein performing the LLM further comprises: False positives in a subset of the sound events are detected based on determining that the false positive sound event is unlikely to correspond to an aggregated local environment of other sound events within the subset; and Remove the false positive before storing a subset of sound events into the event description database.

16. The computer-implemented method of claim 8, wherein the LLM is a Generative Pre-trained Transformer (GPT) LLM.

17. A non-volatile computer-readable medium storing program instructions, which, when executed on or across a processor, cause the processor to: The audio base model (AFM) is provided with audio segments and text-based descriptions, wherein the text-based descriptions correspond to descriptions of sound events to be detected by the AFM; Perform AFM to detect a subset of sound events present within the audio segment; Provide a corresponding subset of text-based descriptions to large language models (LLMs); Execute the LLM, where the execution of the LLM includes: Based on the detected subset of sound events, the audio segments are classified into acoustic scene categories; and Generate additional text-based descriptions corresponding to other sound event descriptions involving the aforementioned acoustic scene category; and The additional text-based description is provided for use in another iteration performed by means of another audio segment in AFM.

18. The non-volatile computer-readable medium of claim 17, wherein, for the purpose of inducing execution of AFM, the program instructions cause the processor to: Encoding is based on text descriptions; Encoded audio segments; Calculate the cosine similarity between the encoded text-based description and the encoded segments of the audio clip; and When the corresponding cosine similarity is higher than the threshold, it is determined that the given sound event exists within the audio segment.

19. The non-volatile computer-readable medium according to claim 18, wherein, To execute AFM, the program instructions additionally cause the processor to: Determine the start and end times of a given sound event; and The start and end times are additionally provided to the LLM for execution.

20. The non-volatile computer-readable medium according to claim 17, wherein, To execute the LLM, the program instructions also cause the processor to: False positives within a subset of the sound events are detected based on determining that a false positive sound event is unlikely to correspond to an aggregated local environment of other sound events within the subset; and Remove the false positive before storing a subset of sound events in the event description database.

Images