Systems and methods for knowledge-based audio-text modeling via automatic multi-modal graph construction

Abstract

Knowledge-based audio-text modeling via automatic multi-modal graph construction is performed. A set of audio data is received, the set of audio data comprising segments of audio data, where each of the segments of audio data is paired with corresponding metadata describing audio content of the respective segment of audio data. A graph node of interest is identified from a semantic network that describes semantics of a knowledge domain of audio data set content. A large language model (LLM) is used to classify metadata into graph nodes and to infer supplemental data for graph nodes for which metadata is not present, thereby producing an extracted knowledge graph. The extracted knowledge graph is verified using LLM to perform relationship verification of edges between graph nodes of the extracted knowledge graph to mitigate hallucination effects in classification and inference of supplemental data.

Description

Technical Field

[0001] Various aspects of this disclosure generally relate to knowledge-based audio-text modeling constructed via automated multimodal graphs. Background Technology

[0002] A knowledge graph can include nodes, edges, and labels that describe the nodes and edges. Nodes can represent objects, such as locations, people, or things. Edges can define the relationships between nodes. Summary of the Invention

[0003] In one or more illustrative examples, a method for knowledge-based audio-text modeling via an automatically constructed multimodal graph is implemented. An audio dataset is received, comprising segments of audio data, each segment paired with corresponding metadata describing the audio content of the corresponding segment. Graph nodes of interest are identified from a semantic network, each graph node describing the semantics of a knowledge domain of the audio dataset's content. A Large Language Model (LLM) is used to classify metadata into graph nodes and to infer supplementary data for graph nodes lacking metadata, thereby generating an extracted knowledge graph. The extracted knowledge graph is validated using LLM to perform relational verification of edges between graph nodes, mitigating illusions in the classification and inference of supplementary data. The validated knowledge graph is then used for downstream applications.

[0004] In one or more illustrative examples, a system for knowledge-based audio-text modeling via an automatically constructed multimodal graph includes: one or more hardware computing devices configured to receive an audio dataset comprising segments of audio data, wherein each segment of audio data is paired with corresponding metadata describing the audio content of the corresponding segment of audio data; identifying graph nodes of interest from a semantic network, the graph nodes describing the semantics of a knowledge domain of the content of the audio dataset; using a large language model (LLM) to classify metadata into graph nodes and to infer supplementary data for graph nodes for which metadata is absent, thereby generating an extracted knowledge graph; utilizing the LLM to validate the extracted knowledge graph to perform relational verification of edges between graph nodes of the extracted knowledge graph, thereby mitigating the illusion effect in the classification and inference of supplementary data; and using the validated knowledge graph for downstream applications.

[0005] In one or more illustrative examples, a non-transitory computer-readable medium includes instructions for knowledge-based audio-text modeling constructed via an automated multimodal graph. When executed by one or more hardware computing devices, the instructions cause the one or more hardware computing devices to perform operations including: receiving an audio dataset comprising segments of audio data, wherein each segment of audio data is paired with corresponding metadata describing the audio content of the corresponding segment of audio data; identifying graph nodes of interest from a semantic network, the graph nodes describing the semantics of a knowledge domain of the audio dataset content; using a large language model (LLM) to classify metadata into graph nodes and to infer supplementary data for graph nodes for which metadata is absent, thereby generating an extracted knowledge graph; utilizing the LLM to validate the extracted knowledge graph to perform relational verification of edges between graph nodes of the extracted knowledge graph, thereby mitigating illusion effects in the classification and inference of supplementary data; and using the validated knowledge graph for downstream applications. Attached Figure Description

[0006] Figure 1 An example framework for knowledge-based audio-text modeling constructed via an automated multimodal graph is shown.

[0007] Figure 2 An example of captioning audio data using the example framework is shown;

[0008] Figure 3 An example of classifying audio data using the sample framework is shown;

[0009] Figure 4 An example process for performing knowledge-based audio-text modeling via automated multimodal graph construction is shown;

[0010] Figure 5 An example downstream application for knowledge-guided audio generation is shown; and

[0011] Figure 6 An example computing device is shown for performing knowledge-based audio-text modeling via automated multimodal graph construction. Detailed Implementation

[0012] Detailed embodiments of the invention are disclosed herein as needed; however, it is to be understood that the disclosed embodiments are merely examples of the invention that can be embodied in various and alternative forms. The figures are not necessarily to scale; some features may be enlarged or minimized to show details of specific components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to use the invention in different ways.

[0013] Structured knowledge can be represented in the form of a knowledge graph (KG). A knowledge graph can include nodes, edges, and labels describing the nodes and edges. Nodes can represent objects, such as locations, people, or things. Edges can define the relationships between nodes. Knowledge graphs can serve as a robust foundation, providing backend support for a variety of practical applications. These applications can include information retrieval and question-answering systems.

[0014] The goal of knowledge graph construction is to automatically extract specific relations (i.e., edges) and events (i.e., nodes) from provided textual information. This relies on high-quality knowledge annotations to retrieve knowledge graphs in a supervised manner. However, acquiring human labels is expensive and laborious, limiting the scalability of knowledge graphs and hindering their use as general-purpose knowledge extractors. Encouragingly, the recent success of high-level large language models (LLMs) has demonstrated their powerful capabilities in general language understanding and reasoning, opening the door to automatically constructing robust knowledge graphs with minimal human annotations using LLMs (e.g., ChatGPT).

[0015] Generative LLMs can be susceptible to illusions (e.g., content that sounds plausible but contains incorrect factual information) or fabrication of their generated content. These problems can lead to misleading text for users or further error accumulation in machine learning systems. Mitigating the illusion effect of LLMs is an active research topic. Various approaches can be used to address this issue, such as introducing additional knowledge augmentation from external resources (e.g., internet search results) or incorporating validation steps to fact-check LLM responses by self-proposing a series of validation questions.

[0016] Carefully crafted cues are often required to optimally utilize LLM. These customized cues are typically task-dependent, trial-and-error based, and cannot be transferred across scenarios, leading to uncontrollable and unpredictable generated output. Furthermore, knowledge graphs and LLM are often applied in text-only scenarios, and their use in conjunction with audio or other multimedia signals has not been well explored.

[0017] Knowledge graphs can summarize information in a structured format, making them ideal for instructing LLMs to create input cues, thereby generating effective outputs in a controlled manner to reduce illusions. Through a set of structured descriptors, knowledge graphs also have the potential to link different multimodal signals together.

[0018] Various aspects of this disclosure utilize predefined semantic frameworks from linguists (e.g., FrameNet) as initializations for graph nodes in general sound and audio descriptions, thereby forming meaningful audio and language multimodal knowledge graphs. Furthermore, LLM is employed as a general knowledge inference engine to interpret relationships across these structured nodes, resulting in an automated graph construction process. Thus, this approach provides a systematic framework for constructing knowledge graphs specifically for sound / audio events based on existing resources, without requiring additional knowledge-based human annotations. The extracted graphs can be broadly applied to various downstream applications, such as representation learning, real-time engineering, and generative artificial intelligence, serving as a knowledge base for audio-text modeling.

[0019] Figure 1 An example framework 100 for knowledge-based audio-text modeling via automated multimodal graph construction is shown. As illustrated, framework 100 can be divided into two main components: an automated process 102 for knowledge graph construction, and a downstream application 104 utilizing the output of the automated process. Knowledge graph construction 102 can utilize an existing audio dataset 106, which may include audio data 108 and metadata 110 that pairs segments of audio data 108 and describes those segments. Knowledge graph construction 102 can also utilize a semantic framework 112 to define graph nodes of interest 114. The audio dataset 106 and the semantic framework 112 can be provided as input to an LLM 116 to create an extracted knowledge graph 118. The downstream application 104 can then use the generated multimodal knowledge graph from the extracted graph structure 116.

[0020] Audio dataset 106 may include audio data 108. In this context, audio data 108 refers to a digitally encoded representation of sound. Audio data 108 may store configurable parameters such as sample rate, bit depth, byte order, number of channels, and fixed or variable bit rate. The audio data 108 discussed herein may be stored in any of a variety of audio formats, including uncompressed formats (e.g., raw data, pulse code modulation (PCM), waveform audio (WAV), audio exchange file format (AIFF), etc.), lossless compression formats (e.g., Free Lossless Audio Codec (FLAC), Apple Lossless Audio Codec (AIFF), etc.), or lossy compression formats (e.g., MPEG-1 Audio Layer 3 (MP3), Advanced Audio Coding (AAC), Windows Media Audio (WMA), etc.).

[0021] Audio dataset 106 may also include metadata 110 describing the audio data 108. Typically, audio dataset 106 may include pairs of audio data 108 and metadata 110, where the metadata 110 takes the form of a textual description, either as a complete sentence or as a category label. The metadata 110 of audio dataset 106 may include target human labels and / or other available attributes that can be associated with it. These human labels and additional attributes provide rich context about the audio samples.

[0022] The sources of metadata 110 can be considered as two main categories: (i) human-annotated and basic facts, and (ii) metadata inferred from existing models (e.g., labels, attributes, and / or other forms of recognition results inferred from audio and / or speech data using one or more machine learning models). Examples from the first category of (i) human-annotated and basic facts may include speech-related annotations, such as speaker information (e.g., gender and age), sound event labels with additional ontological structure (e.g., music-instrument-guitar); or acoustic descriptors extracted from audio data, such as pitch and intensity.

[0023] Examples of the second category of metadata inferred from existing models (ii) may include external resources, such as pre-trained state-of-the-art automatic recognition models, which can be used to create artificial attributes that may be associated with a given audio sample. This could include, for example, speech recognition of spoken content, emotion recognition of expressive style, audio classification of scene information, or sound event detection of more granular vocal objects and their corresponding temporal information.

[0024] Semantics refers to the study of the meaning of language. In this context, semantic network 112 refers to data describing the linguistic meaning of potential words. Semantic network 112 may include multiple graph nodes 114 representing objects, concepts, etc. Graph nodes 114 can be connected by directed links representing relationships between graph nodes 114, such as events, connections, or other actions associated with graph nodes 114. The set of graph nodes 114 can be implemented as a database, list, set, graph, or other data representation of semantic and phonological relationships between different words or sets of words, including a vocabulary.

[0025] In a non-restrictive example, FrameNet defines a complete set of semantic units, which is useful for covering a comprehensive description of sound. The complete set of defining units can include: source, ground, location, path, manner, cause, degree, component, explanation, position, topic, and noise. Combinations of these units express sound within a structured semantic framework. These components can be used as nodes in a KG (e.g., as event patterns). Selecting only nodes of interest (or task relevance) may limit the size of the KG but increases controllability. Purely data-driven node definitions are also feasible by feeding the entire set of units into an LLM for analysis, based on the acquired metadata.

[0026] LLM 116 can refer to any of a variety of machine learning models trained on large datasets to understand, summarize, generate, and predict new content. As used in this paper, a term generation model can refer to an LLM specifically designed to perform text generation tasks based on initial prompts. In some examples, an LLM is implemented using a decoder-only transformer architecture. LLM 116 can be a model specifically trained for use in Frame 100. Alternatively, in other examples, LLM 116 can be a pre-trained base model that can be used as is by Frame 100, or it can be fine-tuned to improve its performance on the specific tasks performed using Frame 100 as discussed in this paper. Some example LLM 116s may include ChatGPT, Claude, Llama, Gemini, Cohere, etc., which may be available for use in different versions and model sizes.

[0027] Given the metadata 110 of the semantic framework 112 and the defined graph nodes 114, the LLM 116 can be instructed to classify each sample of metadata 110 to its corresponding node 114. Because the available metadata 110 may not contain complete information related to all the defined nodes 114, the LLM 116 can be asked a series of prompting questions, such as “What is the sound source?” and “What might the sound source be based on?” to elicit supplementary information for the node.

[0028] Then, the complete sample-based graph nodes 114 and their respective categories can be injected into the LLM 116 to allow reasoning about relationships across nodes 114. Such associations between nodes 114 can be indicated using hints such as “How could a given sound source possibly be related to or unrelated to [something]?” Thus, a per-sample subgraph is constructed, which can be provided to downstream application 104 for later use. This expanded per-sample subgraph may be referred to herein as the extracted knowledge graph 118.

[0029] Downstream applications 104 can perform a variety of tasks, such as graph-based audio representation learning 104A (e.g., contrastive clustering, graph neural networks (GNNs) etc.), structured cueing engineering 104B (e.g., controlled audio caption generation), and / or knowledge-guided audio generation 104C (e.g., AudioGen, pyroomacoustics), as some non-limiting examples.

[0030] More specifically, one type of downstream application 104 includes graph-based audio representation learning. In such a downstream application 104, automatically generated scene graphs can be used to construct adjacency matrices, which in many cases are prerequisites for training GNNs. In such methods, knowledge relationships between audio nodes can be explicitly incorporated into training, forming structured representation learning methods that reflect common-sense relevance across audio samples (e.g., node embeddings) for better audio scene understanding and reasoning tasks, such as audio question answering (AQA).

[0031] In another example category of downstream application 104, structured cue engineering can be performed. Structured cue engineering methods can leverage audio representation learning, including graph-based methods. Utilizing structured nodes and their representations, text-based queries can be designed via combinations of node embeddings (e.g., mean or weighted pooling) or simple concatenations of text descriptions to cue audio foundational models (FMs) for fine-grained retrieval and / or classification of audio samples.

[0032] In another downstream application (104), knowledge-guided audio generation can be performed. Existing language-based audio generation requires descriptive cues to the model to produce more realistic audio. This manual and often tedious task can be replaced by leveraging extracted knowledge structures. By replacing nodes within the knowledge structure, different cues can be generated while maintaining precise control over other unrelated nodes. This approach provides a system framework for efficiently controlling the output of audio generation.

[0033] Figure 2 Example 200 of captioned audio data 108 is shown. As shown, audio dataset 106 includes samples or segments of audio data 108, where each sample of audio data 108 is paired with corresponding metadata 110. This metadata 110 may include phrases or sentences describing the sound events occurring in the corresponding audio data 108 segment. In some examples, metadata 110 may include human-annotated captions to provide the most accurate description of the events occurring in audio dataset 106. As shown for example segments of audio data 108, the paired metadata 110 may indicate that audio data 108 includes "knocking and barking."

[0034] Additionally, graph nodes of interest 114 are identified from the semantic network 112. These graph nodes of interest 114 can be directly defined by the user based on the user's domain knowledge. In another example, graph nodes of interest 114 can be queried from the semantic network 112 based on domain knowledge encoded in the semantic network 112. In yet another example, graph nodes of interest 114 can be extracted from a separate database of domain knowledge. In yet another example, graph nodes of interest 114 can be posed as questions to the LLM 116, such as, "For the domain of captioned audio, what are the most relevant elements of sound propagation?".

[0035] Regardless of the method, as shown, the four basic elements of how sound propagates are identified as of interest: source, base, location, and path. These can be extracted as definitions for generating an extracted knowledge graph 118 that indicates sound propagation.

[0036] The audio data 108, metadata 110, and graph node 114 can then be fed into the LLM 116. This allows the sound source to be identified based on the metadata 110. Continuing this example, based on the metadata 110, the sound source could be, for example, "knocking" and "dog barking".

[0037] For any missing component defined by graph node 114, a follow-up question can be provided to LLM 116. For example, if no information about the ground graph node 114 is available, the concept can be posed to LLM 116. A question such as “What material or surface might the [knocking] sound be based on?” can be provided to LLM 116 as a prompt. The result can then be used to populate the ground graph node 114. In this example, LLM 116 may have already indicated “wood” for the ground graph node 114, which is incorporated into the extracted knowledge graph 118 as information corresponding to the ground graph node 114.

[0038] Next, these categorized graph nodes 114, including the information specified by metadata 110 and additional information inferred by LLM 116, are reintroduced into LLM 116. At this stage, LLM 116 is required to infer the connections between graph nodes 114 (e.g., edges between graph nodes 114). This operation also serves as a verification measure, thereby reducing the likelihood of hallucination effects.

[0039] As shown in the resulting knowledge graph 118, two types of graph nodes 114 as sources were identified: "barking" and "knocking". For the "barking" source graph node 114, a "home" location graph node 114 was inferred, while for the "knocking" source graph node 114, a "home" location graph node 114 was inferred. The connections between these location graph nodes 114 and their corresponding source graph nodes 114 are along edges labeled "from", indicating the location from which the source originates.

[0040] The base graph node 114 for "Timber" is inferred from the source graph node 114 for "Knock". Furthermore, the path graph node 114 for "Door" is inferred from the source graph node 114 for "Knock". Connections from these base graph nodes 114 to the source graph node 114 for "Knock" follow the edges labeled "on", and connections from the path graph nodes 114 to the source graph node 114 for "Knock" follow the edges labeled "across". This is because LLM 116 infers that the "Knock" action is performed by "Timber", "on", and "across" the door.

[0041] Figure 3 An example of audio data classification is shown, here used for speech emotion classification. As shown, audio dataset 106 again includes samples or segments of audio data 108, where each sample of audio data 108 is paired with corresponding metadata 110. Here, instead of complete sentences, a set of various labels (e.g., anger), attributes (e.g., female), and emotional state scores (e.g., arousal, valence, etc.) is provided as metadata 110.

[0042] Additionally, graph nodes of interest 114 are identified from the semantic network 112. Using prior knowledge from the speech emotion task, the source node can be identified as "human speech." Then, the LLM 116 can be used to infer other graph nodes 114 based on the extracted metadata 110. Alternatively, as described above, the user's domain knowledge, a domain knowledge database, etc., can be used additionally or alternatively.

[0043] As shown, the graph nodes of interest 114 include manner, cause, and degree. For example, acoustic patterns are associated with degree nodes 114, arousal and valence scores can provide information about manner nodes 114, and affective labels can indicate cause nodes 114.

[0044] As previously mentioned, the initially formed knowledge graph 118 can then be re-input into LLM 116 as a relation validation. As described above, the resulting knowledge graph 118 can include source patterns 114 of "human speech". Connections to source nodes 114 can be cause nodes, where "anger" is connected as the "motivation" of the "human speech". Also connected to source nodes 114 can be manner nodes 114, where "shout" is connected as how the "human speech" is "expressed". Furthermore, degree nodes 114 for "intensity" and "pitch" are connected to source nodes 114, with edges indicating both "high" "intensity" and "high" "pitch" of the "human speech" corresponding to "anger" and "shout".

[0045] Figure 4 An example process 400 for performing knowledge-based audio-text modeling via automated multimodal graph construction is shown. In one example, process 400 may be executed as an automated process performed by one or more computing devices.

[0046] At operation 402, an audio dataset 106 is received. The audio dataset 106 is received comprising samples or segments of audio data 108, wherein each sample of audio data 108 is paired with corresponding metadata 110. In some examples, the metadata 110 may include human-annotated captions to provide an accurate description of the events occurring in the audio dataset 106. In some examples, instead of or in addition to human-annotated data, machine learning labels, attributes, and voice emotion state scores, etc., may be inferred and included in the metadata 110 using various machine learning models.

[0047] At operation 404, graph nodes of interest 114 are identified from semantic network 112. These graph nodes of interest 114 can be directly defined by the user based on the user's domain knowledge. In another example, graph nodes of interest 114 can be queried from semantic network 112 based on domain knowledge encoded in semantic network 112. In yet another example, graph nodes of interest 114 can be extracted from a separate database of domain knowledge. In yet another example, graph nodes of interest 114 can be posed as a question to LLM 116, such as, "For the domain of [audio dataset], what are the most relevant elements?".

[0048] At operation 406, LLM 116 is used to classify the metadata 110 of audio data 108 into graph nodes 114. In one example, audio data 108, metadata 110, and graph node 114 can be fed into LLM 116. Hints can be provided to LLM 116, requesting it to classify the given metadata 110 into the name of the indicated graph node 114. This allows metadata 110 to be automatically placed into the identified graph node 114.

[0049] At operation 408, LLM 116 is used to infer supplementary data for graph node 114. For example, there might be a graph node 114 for which metadata 110 is missing. In such an example, one or more hints can be provided to LLM 116 to infer the content of graph node 114 for which metadata 110 is not available. In one example, if no data is available for the source node, a hint can be provided to LLM 116 to ask what the source might be. Or, if no data is available for the location of the sound, a hint can be provided to LLM 116 to ask for the source location. This information can then be added to graph node 114.

[0050] At operation 410, LLM 116 is used to perform relation verification. Operation 410 also acts as a verification measure, thereby mitigating the possibility of illusion effects in operations such as operation 408.

[0051] At operation 412, the knowledge graph 118 is utilized by downstream application 104. The proposed graph construction framework 100 is applicable to any existing resource, such as audio classification datasets (e.g., for sound event detection, development of audio tagging models) and audio captioning datasets (e.g., for audio retrieval tasks). In general, datasets containing paired audio and text language descriptions (whether in the form of complete sentences or just category labels) are well-suited for use by framework 100. After operation 412, process 400 ends.

[0052] The extracted multimodal knowledge graph 118 can be beneficial for use in several audio-text modeling techniques. In one example, the knowledge graph 118 can be represented as an adjacency matrix to perform multimodal graph representation learning. In another example, the knowledge graph 118 can be used to define knowledge-based clustering for advanced contrastive learning frameworks.

[0053] In another example, the event patterns of knowledge graph 118 can be used to generate controllable text output via LLM 116 (or via another LLM 116). For example, parameterized sound descriptions can be achieved by manipulating node 114, but still adhere to the existing graph structure. As a possibility, continue... Figure 2 In Example 200, the base object can change from "wood" to "metal" to indicate different materials for the "door." This strategy can serve as a controlled text data augmentation or on-the-spot curatorial approach, thereby enriching the diversity of the linguistic space in a manageable way for the generated results. Therefore, question / answer-based audio reasoning architectures, or text-based audio generation frameworks (e.g., AudioLDM, AudioGen, etc.), can be improved through exploratory sound graphs.

[0054] Figure 5 An example downstream application 104 for knowledge-guided audio generation 104C is shown. The knowledge-guided audio generation 104C can be provided as part of a cloud-based server 500 configured to communicate with a computing device 502 via a wireless communication link 504. The computing device 502 can include, but is not limited to, laptops, tablets, smartphones, smartwatches, or other wearable devices and / or desktop computers. Among other components, the computing device 502 can include various components such as an audio system 506 with speakers 508 or other audio output devices and / or megaphones 510 or other audio input devices, a monitor 512 or other output devices for displaying information, and / or a keyboard 514 or other input devices for receiving user input.

[0055] The knowledge-guided audio generation 104C may be accessible via a user interface 520. The user interface 520 may include a knowledge graph editor 522 configured to display a knowledge graph 118 for the generated audio. The knowledge graph editor 522 may also support other functionalities, such as providing replacements for nodes 114 in the knowledge graph 118 for different nodes. As a possibility, continuing... Figure 2 In Example 200, the base object can change from "wood" to "metal" to indicate different materials for the "door". This allows for graphical adjustment of the generated sound at the node 114 level.

[0056] The user can then select to generate control 524 to provide the knowledge graph 118 to the audio generation model 526 of the cloud-based server 500 for processing via communication link 504. The audio generation model 526 can generate audio data as a cue based on the knowledge graph 118 and can provide the generated audio data back to the computing device 502 via communication link 504. In one example, the audio generation model 526 can be trained to receive the knowledge graph 118 and generate audio based on it. Alternatively, in another example, the knowledge graph 118 can be converted into a text representation, the audio generation model 526 can be a text-to-audio (TTA) model, and the converted text representation can be provided to the audio generation model 526 as a text cue.

[0057] The cloud-based server 500 can generate audio data and send it to the computing device 502 via the communication link 504. The computing device 502 can then receive and play the desired audio data, for example, using the speaker 508 of the audio system 506. The computing device 502 can also additionally or alternatively store the generated audio data in its own memory device.

[0058] In another example, at least some of the features of the cloud-based server 500 may be stored on the computing device 502. For example, the computing device 502 may locally include an audio generation model 526 stored and executed by the computing device 502 to generate desired audio locally at the computing device 502. While this document provides a specific implementation of the downstream application 104, the downstream application 104 may be implemented in other suitable ways.

[0059] Figure 6 An example 600 of a computing device 602 is shown for use in performing knowledge-based audio-text modeling via automated multimodal graph construction. As shown, the computing device 602 includes a processor 604 operatively connected to a storage device 606, a network device 608, an output device 610, and an input device 612. This is merely an example, and computing devices 602 with more, fewer, or different components may be used.

[0060] Processor 604 may include one or more integrated circuits that implement the functions of a central processing unit (CPU) and / or a graphics processing unit (GPU). In some examples, processor 604 is a system-on-a-chip (SoC) that integrates the functions of both the CPU and GPU. The SoC may optionally include other components, such as storage device 606 and network device 608, into a single integrated device. In other examples, the CPU and GPU are connected to each other via peripheral connectivity devices such as a Peripheral Component Interconnect (PCI) fast channel or another suitable peripheral data connection. In one example, the CPU is a commercially available central processing unit that implements the instruction set of one of the microprocessor families such as x86, ARM, Power, or MIPS (Made in Standard Version) instruction set.

[0061] Regardless of the specific circumstances, during operation, processor 604 executes stored program instructions retrieved from storage device 606. Therefore, the stored program instructions include software that controls the operation of processor 604 to perform the operations described herein. Storage device 606 may include both non-volatile memory and volatile memory devices. Non-volatile memory includes solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the system is disabled or loses power. Volatile memory includes static and dynamic random access memory (RAM), which stores program instructions and data during operation of frame 100.

[0062] The GPU may include hardware and software for displaying at least two-dimensional (2D) and optionally 3D graphics to the output device 610. The output device 610 may include a graphics or visual display device, such as an electronic display screen, projector, printer, or any other suitable device for reproducing a graphical display. As another example, the output device 610 may include an audio device, such as a speaker or headphones. As yet another example, the output device 610 may include a tactile device, such as a mechanically liftable device, which in one example may be configured to display Braille or another physical output that can be touched to provide information to a user.

[0063] Input device 612 may include any of a variety of devices that enable computing device 602 to receive control input from a user. Examples of suitable input devices for receiving human-machine interface input may include a keyboard, mouse, trackball, touchscreen, voice input device, graphics tablet, etc.

[0064] Network device 608 can be any of a variety of devices that enable the device to send data over a network and / or receive data from external devices. Examples of suitable network devices 608 include Ethernet interfaces, Wi-Fi transceivers, cellular transceivers, or Bluetooth or BLE transceivers, UWB transceivers, or other network adapters or peripheral interconnect devices that receive data from another computer or external data storage device, which may be useful for receiving large datasets in an efficient manner.

[0065] The processes, methods, or algorithms disclosed herein may be deliverable to / implemented by a processing device, controller, or computer, which may include any existing programmable electronic control unit or dedicated electronic control unit. Similarly, processes, methods, or algorithms may be stored in many forms as data and instructions executable by a controller or computer, including but not limited to information permanently stored on non-writable storage media such as read-only memory (ROM) devices, and information reproducibly stored on writable storage media such as floppy disks, magnetic tapes, optical discs (CDs), RAM devices, and other magnetic and optical media. Processes, methods, or algorithms may also be implemented in a software executable object. Alternatively, suitable hardware components may be used to embody processes, methods, or algorithms wholly or partially, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0066] While exemplary embodiments have been described above, they are not intended to depict all possible forms covered by the claims. The terms used in this specification are descriptive and not limiting, and it is to be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously described, features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as offering advantages or preferences with respect to one or more desired characteristics compared to other embodiments or prior art implementations, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the particular application and implementation. These properties may include, but are not limited to, strength, durability, lifespan, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Therefore, with respect to one or more characteristics, if any embodiment is described as less desirable than other embodiments or prior art implementations, these embodiments are not outside the scope of this disclosure and may be ideal for a particular application.

Claims

1. A method for knowledge-based audio-text modeling via automatic multimodal graph construction, comprising: Receive an audio dataset, the audio dataset comprising segments of audio data, wherein each segment of audio data is paired with corresponding metadata describing the audio content of the corresponding segment; Identify graph nodes of interest from a semantic network, wherein the graph nodes describe the semantics of the knowledge domain of the audio dataset; Large Language Models (LLMs) are used to classify metadata into graph nodes and to infer supplementary data for graph nodes for which metadata is missing, thereby generating an extracted knowledge graph. LLM is used to validate the extracted knowledge graph in order to perform the verification of the relationship between the edges between the nodes of the extracted knowledge graph, thereby mitigating the illusion effect in the classification and inference of supplementary data; and Use validated knowledge graphs for downstream applications.

2. The method according to claim 1, wherein, The metadata includes human annotations describing the audio content of the corresponding segments of the audio data.

3. The method according to claim 1, wherein, The metadata includes machine learning labels, attributes, and / or other forms of recognition results inferred from audio and / or speech data using one or more machine learning models.

4. The method according to claim 1, wherein, The graph nodes of interest are one or more of the following: user knowledge definitions based on the knowledge domain; graph nodes queried from a semantic network as descriptions of the semantics of the knowledge domain; extracted from a database of domain knowledge; or received from an LLM in response to hints about relevant graph nodes in the knowledge domain.

5. The method of claim 1, wherein inferring supplementary data includes receiving supplementary data from the LLM in response to a prompt requesting the LLM to infer the content for the graph node name for which metadata is not available.

6. The method of claim 1, wherein the downstream application includes an audio classification application that uses the knowledge graph for sound event detection and / or audio tagging.

7. The method according to claim 1, wherein, The downstream applications include audio captioning applications that use knowledge graphs for audio retrieval.

8. The method according to claim 1, wherein, The downstream applications include representing the knowledge graph as an adjacency matrix to perform multimodal graph representation learning.

9. The method of claim 1, wherein the downstream application includes using the knowledge graph to define knowledge-based clustering for contrastive learning.

10. The method according to claim 1, wherein, The downstream applications include using the knowledge graph to manage controllable prompts, captions, and / or descriptive content to build knowledge-guided generative models.

11. A system for knowledge-based audio-text modeling constructed via an automatically constructed multimodal graph, comprising: One or more hardware computing devices are configured to: Receive an audio dataset, the audio dataset comprising segments of audio data, wherein each segment of audio data is paired with corresponding metadata describing the audio content of the corresponding segment; Identify graph nodes of interest from a semantic network, wherein the graph nodes describe the semantics of the knowledge domain of the audio dataset; Large Language Models (LLMs) are used to classify metadata into graph nodes and to infer supplementary data for graph nodes for which metadata is missing, thereby generating an extracted knowledge graph. LLM is used to validate the extracted knowledge graph in order to perform the verification of the relationship between the edges between the nodes of the extracted knowledge graph, thereby mitigating the illusion effect in the classification and inference of supplementary data; and Use validated knowledge graphs for downstream applications.

12. The system of claim 11, wherein the metadata includes human annotations describing the audio content of a corresponding segment of the audio data.

13. The system according to claim 11, wherein, The metadata includes machine learning labels, attributes, and / or other forms of recognition results inferred from audio and / or speech data using one or more machine learning models.

14. The system according to claim 11, wherein, The graph nodes of interest are one or more of the following: user knowledge definitions based on the knowledge domain; graph nodes queried from a semantic network as descriptions of the semantics of the knowledge domain; extracted from a database of domain knowledge; or received from an LLM in response to hints about relevant graph nodes in the knowledge domain.

15. The system according to claim 11, wherein, The inference of supplementary data includes receiving supplementary data from the LLM in response to a request for the LLM to infer the content for the names of graph nodes for which there is no available metadata.

16. The system according to claim 11, wherein, The downstream applications include audio classification applications that use the knowledge graph for sound event detection and / or audio tagging.

17. The system according to claim 11, wherein, The downstream applications include audio captioning applications that use knowledge graphs for audio retrieval.

18. The system according to claim 11, wherein, The downstream applications include representing the knowledge graph as an adjacency matrix to perform multimodal graph representation learning.

19. The system according to claim 11, wherein, The downstream applications include using the knowledge graph to define knowledge-based clustering for contrastive learning.

20. The system according to claim 11, wherein, The downstream applications include using the knowledge graph to manage controllable prompts / captions / descriptive content to build knowledge-guided generative models.

21. A non-transitory computer-readable medium comprising instructions for knowledge-based audio-text modeling constructed via an automated multimodal graph, wherein, when executed by one or more hardware computing devices, the instructions cause the one or more hardware computing devices to perform operations, including: Receive an audio dataset, the audio dataset comprising segments of audio data, wherein each segment of audio data is paired with corresponding metadata describing the audio content of the corresponding segment; Identify graph nodes of interest from a semantic network, wherein the graph nodes describe the semantics of the knowledge domain of the audio dataset; Large Language Models (LLMs) are used to classify metadata into graph nodes and to infer supplementary data for graph nodes for which metadata is missing, thereby generating an extracted knowledge graph. LLM is used to validate the extracted knowledge graph in order to perform the verification of the relationship between the edges between the nodes of the extracted knowledge graph, thereby mitigating the illusion effect in the classification and inference of supplementary data; and Use validated knowledge graphs for downstream applications.

Images