Systems and methods for a collaborative audio agent using large language model and retrieval augmented generation

Abstract

Systems and methods for diagnosing device problems through chat sessions address the challenge of diagnosing complex audio-related problems. The system includes a large language model (LLM) that serves as an interface between a human user and a plurality of expert audio models. The system receives an audio recording from the user, executes the LLM to gather additional information, identifies a sound domain, and selects an expert audio agent specializing in that domain. Retrieval-augmented generation (RAG) is used to process the audio to determine a likely cause of the problem, which is then communicated to the user. The system enhances the ability to solve problems by enabling collaboration between specialized agents, thereby improving the efficiency and accuracy of audio diagnosis.

Description

Technical Field

[0001] This disclosure generally relates to artificial intelligence technologies. In specific embodiments, this disclosure relates to a multi-agent collaborative framework for audio applications that utilize large language models and retrieval enhancements. Background Technology

[0002] Significant progress has been made across various industry categories, including manufacturing, healthcare, and logistics, in the ever-evolving field of artificial intelligence (AI). These advancements have enabled the creation of systems adept at managing complex and multifaceted interactions. Despite these progresses, the integration of AI into audio applications continues to present challenges. Traditional audio systems typically rely on single agents or models tailored for specific tasks, such as voice recognition or audio classification. These systems are limited in their ability to manage complex interactions, often requiring strict syntax or predefined rules, which hinders their adaptability and responsiveness in dynamic environments.

[0003] Current solutions in the audio application field primarily emphasize individual agent systems or machine translation-based multilingual dialogue systems. While these approaches may be effective in certain situations, they also come with significant limitations. Systems utilizing automatic machine translation for multilingual interaction are often limited by language-specific capabilities and cannot meet the broader need for task coordination and collaboration among multiple specialized agents. Furthermore, methods relying on selecting a single agent based on past performance do not fully leverage the potential for collaborative problem-solving among specialized agents, thus limiting the system's ability to effectively handle complex tasks. Summary of the Invention

[0004] In one embodiment, a method for diagnosing device problems via a chat session utilizes a large language model (LLM) as an interface between a human user and multiple expert audio models. The method includes receiving an audio recording from a human user, wherein the audio recording is associated with a potential error or problem in the device; performing LLM to obtain additional information about the audio recording from the user; identifying a sound domain based on the audio recording and additional input from the user; selecting an expert audio agent specializing in the identified sound domain, wherein the expert audio agent is one of multiple expert audio agents configured to analyze specific types of audio characteristics in a corresponding domain of the sound domain; processing the audio recording via the selected expert audio agent using retrieval enhancement generation (RAG) to generate data related to possible causes of errors or problems associated with the audio recording; and performing LLM to communicate the possible causes of the errors or problems output by the selected expert audio agent to the user.

[0005] In one embodiment, a computational system for diagnosing device problems via a chat session utilizes a large language model (LLM) as an interface between a human user and multiple expert audio models. The computational system includes one or more processors programmed to receive audio recordings from a human user, wherein the audio recordings are associated with potential errors or problems in the device; performing an LLM to obtain additional information about the audio recordings from the user; identifying sound domains based on the audio recordings and additional input from the user; selecting an expert audio agent specializing in the identified sound domain, wherein the expert audio agent is one of multiple expert audio agents configured to analyze specific types of audio characteristics in a corresponding domain of the sound domain; processing the audio recordings via the selected expert audio agent using retrieval enhancement generation (RAG) to generate data related to possible causes of errors or problems associated with the audio recordings; and performing an LLM to communicate the possible causes of the errors or problems output by the selected expert audio agent to the user.

[0006] In one embodiment, non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform a method comprising: receiving an audio recording from a human user, wherein the audio recording is associated with a potential error or problem of a device; performing LLM to obtain additional information about the audio recording from the user; identifying a sound domain based on the audio recording and additional input from the user; selecting an expert audio agent specializing in the identified sound domain, wherein the expert audio agent is one of a plurality of expert audio agents configured to analyze specific types of audio characteristics in a corresponding domain of the sound domain; processing the audio recording via the selected expert audio agent using retrieval enhancement generation (RAG) to generate data related to possible causes of errors or problems associated with the audio recording; and performing LLM to communicate the possible causes of errors or problems output by the selected expert audio agent to the user. Attached Figure Description

[0007] Figure 1 A system for training a neural network according to one embodiment is shown in general.

[0008] Figure 2 The present invention generally illustrates a computer-implemented method for training and utilizing a neural network according to one embodiment.

[0009] Figure 3A This is a block diagram of the interaction framework of a multi-agent audio system with an LLM regulator according to one embodiment.

[0010] Figure 3B It is a sequence diagram of interactions between a human user and the system according to one embodiment.

[0011] Figure 4 This is a flowchart of a method for diagnosing audio-related problems using an expert agent and LLM, according to an embodiment.

[0012] Figure 5 A schematic diagram depicts the interaction between a computer-controlled machine and a control system according to one embodiment.

[0013] Figure 6 A vehicle configured to control a vehicle according to one embodiment is depicted. Figure 5 The diagram shows the control system of the vehicle, which can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0014] Figure 7 Depicting Figure 5 A schematic diagram of a control system configured to control manufacturing machines, such as stamping and cutting machines, cutting machines, or gun drills, within a manufacturing system (e.g., a part of a production line).

[0015] Figure 8 Depicting Figure 5 A schematic diagram of a control system configured to control power tools, such as drills or drives, having at least a partially autonomous mode.

[0016] Figure 9 Depicting a system configured to control an automated personal assistant Figure 5 A schematic diagram of the control system.

[0017] Figure 10 Depicting Figure 5 A schematic diagram of a control system configured as a control and monitoring system, such as a control access system or a monitoring system.

[0018] Figure 11 Depicting Figure 5 A schematic diagram of the control system configured to control an imaging system, such as an MRI device, an X-ray imaging device, or an ultrasound device. Detailed Implementation

[0019] Embodiments of this disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be enlarged or reduced to show details of particular components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to use the embodiments in various ways. As will be understood by those skilled in the art, the various features illustrated and described with reference to any figure may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments of typical applications. However, various combinations and modifications of features consistent with the teachings of this disclosure may be desired for particular applications or implementations.

[0020] As used herein, “a,” “one,” and “the” refer to both the singular and plural forms unless the context clearly indicates otherwise. For example, a “processor” programmed to perform various functions refers to one processor programmed to perform each of those functions, or more than one processor programmed together to perform each of those functions.

[0021] In recent years, advancements in artificial intelligence (AI) have dramatically transformed various industries, including manufacturing, healthcare, and logistics. However, despite these progresses, integrating AI systems into audio applications remains challenging. Traditional audio systems typically rely on single agents or models specializing in specific tasks, such as voice recognition or audio classification. These systems are limited in their ability to handle complex, multifaceted interactions and often require strict grammars or predefined rules, which restricts their adaptability and responsiveness to dynamic environments.

[0022] In the field of audio applications, previous developments have primarily focused on single-agent systems or machine translation-based multilingual dialogue systems. While these approaches are effective in certain situations, they also exhibit some drawbacks. For example, systems employing automatic machine translation for multilingual interaction are limited by language-specific capabilities and cannot meet the broader need for task coordination and collaboration among multiple expert agents. Similarly, methods that select a single agent based on previous performance fail to leverage the potential for problem-solving through collaboration among cross-specialty agents, limiting the system's ability to effectively handle complex tasks.

[0023] This system addresses these challenges by introducing a novel multi-agent collaboration framework specifically designed for audio applications. This framework facilitates interaction and collaboration among independent audio expert agents, each specializing in different tasks such as audio understanding, manipulation, and generation. By leveraging the capabilities of a large language model (LLM), the system enables these agents to operate autonomously, communicating, exchanging crucial information, and sharing knowledge using audio and other modalities. This collaborative approach enhances efficiency and problem-solving capabilities, positioning agents as experts capable of participating in discussions, debates, and providing guidance to users. Multi-agent systems excel at handling complex real-time challenges, optimizing resource management, and improving decision-making processes, generating deeper insights and more innovative solutions than a single entity could achieve alone.

[0024] According to embodiments disclosed herein, this disclosure provides the creation of a multi-agent collaboration framework specifically designed for audio applications, leveraging the capabilities of LLM to enhance communication, decision-making, and knowledge sharing among expert audio agents. The framework introduces an integration of Retrieval Augmented Generation (RAG) to provide domain-specific knowledge, enabling agents to perform tasks autonomously and effectively. Unlike previous systems that typically focused on reinforcement learning in single-agent systems or multi-agent environments, this framework emphasizes a collaborative approach where human users and multiple expert audio agents work together to tackle complex tasks. Using LLM as a central coordinator facilitates seamless interaction and task coordination, and differentiates the framework by providing a scalable, distributed solution optimized for resource management and problem-solving in real-time audio processing scenarios.

[0025] The following description continues with a general introduction to machine learning techniques related to methods for leveraging machine learning models, such as those described herein. Various embodiments of architectures and processing flows for diagnosing device problems using large language models (LLMs) via chat sessions are discussed next. This disclosure will also demonstrate the versatility of the methods and systems described herein in integration with devices.

[0026] Figure 1 A system 100 for training and utilizing machine learning models, such as convolutional neural networks, according to some embodiments is shown. It should be understood that, although referred to herein in the following text... Figure 1 and Figure 2 The example embodiments given involve convolutional neural networks, but Figure 1 and Figure 2 Further embodiments can be applied to any other type of neural network-based or non-neural network-based machine learning model, which can be configured to be developed, trained, fine-tuned, and / or executed for various applications of audio tracking and interpretation further described herein.

[0027] In some embodiments, system 100 may include an input interface for accessing training data 102 for a convolutional neural network. For example, such as Figure 1 As illustrated in the diagram, the input interface can be composed of a data storage interface 104, which can access the training data 102 from the data storage device 106. For example, the data storage interface 104 can be a memory interface or a permanent storage interface, such as a hard disk or SSD interface, but it can also be a personal network, local area network, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be an internal data storage device of the system 100, such as a hard disk drive or SSD, but it can also be an external data storage device, such as a network-accessible data storage device.

[0028] In some embodiments, data storage device 106 may further include a data representation 108 of an untrained model version (e.g., a version of a machine learning model that has not yet been trained), which can be accessed by system 100 from data storage device 106. However, it will be appreciated that the training data 102 and data representation 108 of the pre-trained convolutional neural network may also be accessed from different data storage devices, for example, via different subsystems of data storage interface 104. Each subsystem may belong to the type described above for data storage interface 104. In other embodiments, the data representation 108 of the pre-trained convolutional neural network may be generated internally by system 100 based on the design parameters of the neural network, and therefore may be stored non-explicitly on data storage device 106. System 100 may further include a processor subsystem 110, which may be configured to provide an iterative function as a replacement for the layer stack of the convolutional neural network to be trained during operation of system 100. Hereinafter, the individual layers of the replaced layer stack may have mutually shared weights and may receive the output of the previous layer as input, or, for the first layer of the layer stack, receive the initial activation and a portion of the layer stack's input. The processor subsystem 110 may further be configured to iteratively train and / or fine-tune the convolutional neural network using training data 102 (e.g., thus generating an updated version of the machine learning model relative to the first "pre-trained" model version). In this document, the iterations of training performed by the processor subsystem 110 may include a forward propagation portion and a backward, or generative propagation portion.

[0029] System 100 may further include an output interface for outputting a data representation 112 of the trained convolutional neural network, and this data may also be referred to as trained model data 112. For example, as Figure 1As illustrated in the diagram, the output interface can be comprised of a data storage interface 104, wherein in these embodiments, the interface is an input / output (“IO”) interface through which trained model data 112 can be stored in a data storage device 106. For example, the data representation 108 defining a “pre-trained” convolutional neural network can be at least partially replaced by the data representation 112 of the trained neural network during or after training, because the parameters of the convolutional neural network (e.g., weights, hyperparameters, and other types of convolutional neural network parameters) can be adapted to reflect training on the training data 102. This also... Figure 1 The figures are illustrated by reference numerals 108 and 112, which refer to the same data records on data storage device 106. In other embodiments, data representation 112 may be stored separately from data representation 108 defining a "pre-trained" convolutional neural network. In some embodiments, the output interface may be separate from data storage interface 104, but may generally have the type described above for data storage interface 104.

[0030] Figure 1 The system 100 shown is an example of a system that can be used to train and subsequently execute the trained machine learning model described herein.

[0031] Figure 2 A computer-implemented system 200 for training and utilizing convolutional neural networks, according to some embodiments, is illustrated. System 200 may include at least one computing system 202. Computing system 202 may include at least one processor 204 operatively connected to memory unit 208. Processor 204 may include one or more integrated circuits implementing the functionality of a central processing unit (CPU) 206 and—in some embodiments—a graphics processing unit (GPU). CPU 206 may be a commercially available processing unit implementing an instruction set such as x86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. Stored program instructions may include software controlling the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 may be a system-on-a-chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interface, and input / output interface into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation.

[0032] Memory cell 208 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, memory cell 208 may store machine learning model 210 or algorithm, training dataset 212 for machine learning model 210, original source dataset 216, etc.

[0033] The computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 222 may include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 222 may further be configured to provide a communication interface to an external network 224 or the cloud.

[0034] External network 224 may be referred to as the World Wide Web or the Internet. External network 224 can establish standard communication protocols between computing devices. External network 224 can allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 can communicate with external network 224.

[0035] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 220 may include an additional serial interface (e.g., a Universal Serial Bus (USB) interface) for communicating with external devices.

[0036] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device enabling the system 200 to receive control input. Examples of input devices may include HMI inputs such as a keyboard, mouse, touchscreen, voice input device, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphical and textual information to the display device 232. The display device 232 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 202 may also be configured to allow interaction with remote HMIs and remote display devices via a network interface device 222.

[0037] System 200 can be implemented using one or more computing systems. Although the example depicts a single computing system 202 implementing all the described features, it is intended that the various features and functions can be implemented separately by multiple computing units communicating with each other. The specific system architecture chosen can depend on a variety of factors.

[0038] System 200 can implement machine learning algorithm 210 configured to analyze raw source dataset 216. Raw source dataset 216 may include raw or unprocessed sensor data, which may represent the input dataset for the machine learning system. In some examples, machine learning algorithm 210 may be a convolutional neural network algorithm designed to perform a predetermined function. For example, the neural network algorithm may be configured to receive audio segments and text-based event descriptions, and operate as an agent 306, LLM 304, and / or additionally as a sub-model 320 as described below.

[0039] Computer system 202 may store a training dataset 212 for machine learning algorithm 210. Training dataset 212 may represent a collection of previously constructed data used to train machine learning algorithm 210. Machine learning algorithm 210 can use training dataset 212 to learn weighting factors associated with a neural network algorithm. Training dataset 212 may include a source dataset having a corresponding outcome or result that machine learning algorithm 210 attempts to replicate through the learning process.

[0040] Machine learning algorithm 210 can run in learning mode using training dataset 212 as input. Machine learning algorithm 210 can be executed in multiple iterations using data from training dataset 212. With each iteration, machine learning algorithm 210 can update its internal weighting factors based on the obtained results. For example, machine learning algorithm 210 can compare its output results (e.g., annotations) with those included in training dataset 212. Since training dataset 212 includes expected results, machine learning algorithm 210 can determine when performance is acceptable. After machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistency with results associated with training dataset 212), machine learning algorithm 210 can be executed using data not in training dataset 212. The trained machine learning algorithm 210 can be applied to new datasets to generate annotated data.

[0041] Machine learning algorithm 210 can be configured to identify specific features in raw source data 216. Raw source data 216 may include multiple instances or input datasets for which desired annotation results are desired. Machine learning algorithm 210 can be programmed to process raw source data 216 to identify the presence of specific features. Machine learning algorithm 210 can be configured to identify features in raw source data 216 as predetermined features. Raw source data 216 can be derived from various sources. For example, raw source data 216 may be actual input data collected by a machine learning system. Raw source data 216 may be machine-generated for testing systems. As an example, raw source data 216 may include audio recordings and / or text-based event descriptions related to the surrounding audio environment.

[0042] In this example, machine learning algorithm 210 can process the raw source data 216 and output an indication of "which of the text-based event descriptions is supported by the audio signal within the audio segment". Machine learning algorithm 210 can generate a confidence level or confidence factor for each generated output. For example, a confidence value exceeding a predetermined high confidence threshold can indicate that machine learning algorithm 210 is confident that the identified source or cause of the sound corresponds to the actual source or cause of the sound. A confidence value below a low confidence threshold can indicate that machine learning algorithm 210 is less confident that the identified cause or source of the sound is actually the correct one.

[0043] As AI systems evolve, the nature of human-agent interaction is changing to accommodate more complex and context-aware commands. Unlike earlier systems that specialized for single tasks, modern AI-powered systems can handle multifaceted contextual interactions and understand nuanced queries without strict grammar. This shift has driven the development of multi-agent collaboration frameworks in audio applications, where agents can interact and respond intelligently in real time. Similar to collaborative robots, which work alongside humans to enhance safety, creativity, and productivity, multi-agent audio systems integrate AI to provide more flexible and personalized support. These systems are also capable of AI-enhanced decision-making, providing real-time insights that allow for dynamic, context-sensitive responses. In an autonomous ecosystem, agents autonomously self-regulate and learn, while human operators supervise high-level strategies. The increasing complexity of AI-driven collaboration is driving innovation in industries such as manufacturing, healthcare, and logistics, where complex tasks require seamless communication and decision-making among multiple agents.

[0044] AI agents are playing an increasingly important role in audio applications, autonomously processing and analyzing sound to perform tasks such as audio classification, sound recognition, and real-time sound mixing. In environments ranging from factories to nature reserves, these agents can monitor noise, such as the hum of machinery or the calls of wildlife, and trigger appropriate actions, such as alerting maintenance teams or registering animal sounds. By collecting and interpreting audio data, these agents provide valuable insights and can act independently, making them highly effective in sound-based tasks. Different types of AI agents are tailored to handle a variety of audio challenges. Simple reflection agents respond to specific audio inputs based on predefined rules, such as automatically reducing noise when decibel levels rise. Model-based reflection agents go further, predicting potential problems by evaluating sound patterns, such as detecting subtle changes in factory machinery noise to prevent equipment malfunction. Goal-based agents are ideal for tasks such as noise cancellation, selecting the most effective strategy to achieve the desired result. Utility-based agents, common in music production, help balance sound quality and artistic preference, while learning agents continuously improve by analyzing previous sound data, enhancing their performance in dynamic environments. Layered, structured agents manage complex audio tasks by coordinating different agents, ensuring seamless operation of systems like smart homes, where sound monitoring and playback are handled simultaneously.

[0045] Multi-agent collaboration involves the interaction and cooperation of autonomous entities, such as AI agents or collaborative robots, to achieve a common goal. In industries such as manufacturing, healthcare, and logistics, these agents work together to streamline complex tasks, automate workflows, and effectively manage resources. Each agent operates independently but shares relevant data to coordinate actions. This workload allocation improves productivity because agents can focus on specialized tasks while contributing to the overall goal. In audio applications, multi-agent collaboration improves processes such as sound processing, speech recognition, and speech synthesis. Agents in intelligent environments can collectively optimize microphone placement, reduce background noise, and improve sound quality, resulting in clearer communication and more adaptive and responsive audio systems. Despite their potential, multi-agent collaboration frameworks face challenges, including scalability, communication overhead, and coordination issues. As system complexity increases, real-time coordination of actions—especially in non-speech audio applications—becomes problematic. Limited computational resources, difficulties adapting to unfamiliar environments, and security concerns further hinder the efficiency of these systems. Future advancements should focus on developing decentralized, scalable frameworks, improving adaptive learning mechanisms, and enhancing multimodal collaboration by combining audio with other sensory inputs.

[0046] Large Language Models (LLMs) like ChatGPT play a crucial role in advancing multi-agent collaboration by enhancing communication, decision-making, and knowledge sharing among agents. LLMs can act as central coordinators or mediators, interpreting and generating natural language instructions, enabling agents to better understand complex tasks, collaborate more effectively, and adapt to changing environments. By facilitating seamless interaction between agents and human operators, LLMs streamline multi-agent workflows, reduce misunderstandings, and improve overall coordination. Their ability to rapidly process large amounts of data also supports real-time decision-making, which is crucial in dynamic settings such as autonomous systems or collaborative robots. Furthermore, LLMs enable agents to exchange context and high-level information using natural language, which can be applied to audio applications such as audio signal processing or sound classification. By interpreting commands and coordinating agent actions in a more intuitive way, LLMs contribute to improving the overall efficiency of collaborative multi-agent systems, allowing them to handle more complex and nuanced tasks. Their adaptability and learning capabilities also make them valuable in the future development of multi-agent collaboration, helping agents navigate new challenges and environments more effectively.

[0047] Retrieval-Enhanced Generation (RAG) is an advanced machine learning approach that combines the strengths of retrieval-based and generative models to enhance information generation and accuracy. While RAG is commonly used in natural language processing, its application to audio signals opens up new possibilities for tasks such as audio classification, sound event detection, and audio generation. By integrating retrieval mechanisms, RAG can access a vast repository of reference sounds, enabling it to generate more accurate and context-sensitive output when processing audio. For example, in sound event detection, RAG can retrieve similar past audio events from a database, improving the system's ability to classify or understand the current signal more effectively. This combination of retrieval and generation can lead to better performance in applications such as ambient sound monitoring, audio creation, or even medical signal analysis, where accurate and context-aware audio processing is crucial. This disclosure proposes incorporating LLM into a powerful module, rather than relying solely on model-based or data-driven approaches to solve complex user queries. LLM provides broad knowledge and general solutions, working alongside specialized generative AI agents focused on specific tasks. This combination enhances the flexibility and adaptability of the solution, improves overall efficiency, and delivers more insightful and comprehensive responses.

[0048] refer to Figure 3A This illustrates a system 300 for diagnosing device problems via chat sessions using a large language model (LLM). System 300 can utilize the above reference... Figure 1-2 The described structure and system are used to operate.

[0049] System 300 may include one or more different types of inputs 302. Inputs serve as the initial point of interaction between the user and the system, including various data types that the user can provide to facilitate the diagnosis of device problems. These inputs may include audio recordings, textual descriptions of audio events, and images corresponding to the audio events, each providing unique insights into the nature of the problem at hand. Audio recordings may include direct captures of sounds that may indicate malfunctions or irregularities in the device, such as unusual noises from the machine or distorted audio output from a speaker. The user can record these sounds using a microphone or recording equipment and submit them to the system for analysis.

[0050] Text descriptions of audio events provide a narrative or contextual explanation of audio-related issues. Users can describe characteristics of the sound, such as its frequency, duration, or any patterns they observe. This text input can be particularly useful when users cannot directly capture the audio, or when additional context is needed to fully understand the problem. By providing detailed descriptions, users help the system better interpret audio data and identify potential causes of the problem.

[0051] Images corresponding to audio events can also be submitted as input, providing a visual representation of the environment or equipment associated with the audio problem. For example, a user might provide a photograph of a machine emitting strange noises, highlighting specific components that may be malfunctioning. These images help the system correlate visual cues with audio data, thereby enhancing the overall analysis. Users can upload these images through an interface, where they are processed along with audio and text input. Once the input is received, the regulatory agent 304 utilizes LLM to integrate and analyze the data to determine the source or cause of the error or problem illustrated by the audio. This comprehensive approach ensures that the system considers all available information, resulting in more accurate and effective diagnostics.

[0052] To this end, System 300 has three core modules (or machine learning models): supervisory agent 304, expert agent 306, and retrieval enhancement generation (RAG).

[0053] The supervisory agent 304, or orchestrator, acts as the high-level brain and coordinator of system 300, leveraging LLM to facilitate seamless communication and coordination among various expert agents and users. Supervisory agent 304 can also function as an ML model 210. As an intelligent mediator, supervisory agent 304 interprets user input, determines the context and nature of the problem, and orchestrates workflows by delegating tasks to appropriate expert agents. It acts as a central hub for knowledge sharing, ensuring that each agent receives the necessary information to perform its specialized function. By utilizing the capabilities of LLM, the supervisory agent can decompose complex tasks into manageable subtasks, distribute these tasks among expert agents, and synthesize their outputs into a coherent response to the user. This dynamic role allows the supervisory agent to adapt to evolving conditions and unforeseen challenges, thereby improving the overall efficiency and effectiveness of the system in diagnosing and resolving equipment problems.

[0054] Expert agents 306 are specialized components within the system, each designed to handle specific tasks related to audio processing and analysis. These agents 306 can operate as ML models 210. These agents possess domain-specific expertise, enabling them to perform functions such as audio understanding, manipulation, generation, simulation, and recommendation. For example, an audio understanding agent might focus on interpreting the semantic content of audio input, while a manipulation agent might modify audio characteristics to meet certain criteria. By operating autonomously within their respective domains, expert agents contribute to collaborative problem-solving environments, where their combined efforts lead to more accurate and insightful diagnostics. The system's architecture allows these agents to work in parallel, sharing relevant data and insights through a supervisory agent, which optimizes resource management and enhances the system's ability to handle complex, real-time audio challenges.

[0055] RAGs enhance the ability of expert agents to access and utilize domain-specific knowledge. They combine the strengths of retrieval-based and generative models to provide more accurate and context-relevant output. In the context of audio processing, RAGs enable expert agents to retrieve similar past audio events or reference sounds from a vast repository (e.g., 208), improving their ability to classify or understand current signals. This retrieval mechanism ensures that agents can access the most relevant information and then use it to generate more accurate analyses and recommendations. By integrating RAGs into workflows, systems can deliver more comprehensive and insightful responses to user queries, ultimately leading to more efficient problem-solving and enhanced user satisfaction.

[0056] In this embodiment, the supervisory agent 304 acts as an intelligent mediator, facilitating communication, coordination, and decision-making among multiple agents. Utilizing an LLM (Limited Language Management), this module enhances the collective capabilities of both humans and human agents, serving as a central hub for knowledge sharing, interaction, and task execution. The LLM enables natural language communication, allowing seamless information exchange between agents and humans, making interactions more intuitive. It also acts as a dynamic knowledge repository, providing agents with access to vast amounts of data and expertise to solve problems and make decisions. Furthermore, the LLM supports task coordination by interpreting complex requirements, breaking them down into manageable subtasks, and assigning these tasks among expert agents 306 for optimized collaboration. Their understanding of context ensures consistency in multi-agent interactions, while role assignment capabilities allow the LLM to delegate tasks to the most qualified agent, thereby improving system performance. Additionally, the LLM supports adaptive decision-making, allowing agents to adjust their strategies in response to evolving conditions or unforeseen challenges. By interpreting human queries and translating them into agent actions, the LLM guides agents to complete tasks, using prompt templates to extract knowledge from past interactions to inform each agent of the task. This creates a more flexible, robust, and efficient collaborative environment where the oversight agent not only orchestrates workflows but also continuously learns and adapts to enhance the system's overall effectiveness in addressing complex audio-related challenges.

[0057] In operation, the supervisory agent 304 can communicate with the user about general aspects of the sound the user wishes to know more about. For example, the user can submit input, and the supervisory agent 304 can process that input and request further feedback about the sound. For instance, the supervisory agent 304 can ask the user to explain what type of device the sound is coming from, the frequency of the sound, whether there are any known causes for the sound, etc. The supervisory agent 304 can then enlist the assistance of one or more expert agents 306 for subprocessing that is not suitable for the LLM itself.

[0058] Expert agents are specialized components within a multi-agent framework, each equipped with the necessary tools and knowledge to excel in its respective domain. These agents are designed to handle specific tasks related to audio processing and analysis, facilitating a collaborative environment where their combined efforts lead to more accurate and insightful diagnostics. Figure 3A As shown, the following features highlight the capabilities of expert agents: understanding, reasoning, generation, manipulation, simulation, and recommendation.

[0059] Understanding 308 refers to the ability of expert agents to interpret audio signals and extract their semantic meaning, thereby bridging the gap between sound and understanding. These models analyze auditory features (e.g., pitch, timbre, and rhythm) and the semantic content embedded in the audio. For example, in a real-world scenario, an understanding agent can analyze recordings of malfunctioning machines to identify specific sound patterns that indicate mechanical problems, such as a high-pitched scream suggesting bearing wear.

[0060] Inference 310 goes beyond basic audio recognition or classification, extending to tasks such as understanding context, recognizing speaker intent, and making logical inferences from audio signatures. This involves analyzing, interpreting, and drawing inferences from audio signals, requiring the model to understand the context and meaning of the sounds. For example, an inference agent might assess the audio environment of a factory floor to infer that a sudden increase in noise levels is due to a conveyor belt malfunction rather than normal operating noise.

[0061] Generative 312 involves creating or synthesizing new sounds from scratch, such as generating ambient sounds or music, using techniques like Generative Adversarial Networks (GANs) or diffusion-based models. In practice, generative agents can be used to simulate the sound of a machine operating under optimal conditions, providing a reference for comparison with real-world recordings to identify differences and potential problems.

[0062] Manipulation 314 refers to the process of modifying existing audio samples or audio received by a user, including tasks such as sound editing, audio enhancement, and style transfer. This involves altering audio characteristics—such as pitch, timbre, or rhythm—while preserving the essence or signature of the original sound. For example, manipulation agents can be used to filter background noise from a machine's recording, allowing for clearer analysis of the machine's operating sound.

[0063] Simulation 316 refers to the creation of a realistic auditory experience that closely resembles real-world sounds and environments. This includes simulating different sound scenarios, ambient noise, and complex acoustic interactions such as echoes, reverberations, or directional sound effects. Simulation agents can be used to recreate the acoustic environment of a manufacturing plant, enabling engineers to test how changes in machine placement affect sound propagation and noise levels.

[0064] Recommendation 318 involves analyzing user queries, audio characteristics, and contextual cues to deliver personalized and relevant audio content. This process evaluates factors such as user preferences, interaction history, and the acoustic properties of the audio—e.g., genre, tempo, and mood. For example, a recommendation agent might suggest maintenance actions based on analysis of audio recordings from machines, suggesting components that may need replacement or tweaking to improve performance.

[0065] In embodiments, audio expert agents can leverage foundational audio models, which serve as a robust starting point for a variety of audio-related tasks. These foundational models can be pre-trained on extensive datasets, encompassing a wide range of audio types and characteristics, enabling the expert agent to possess a comprehensive understanding of audio signals. For tasks such as understanding, these models can be fine-tuned to identify and interpret specific audio patterns, allowing the agent to accurately identify and classify sounds based on their semantic content. For example, an audio understanding agent can utilize foundational models trained on a variety of environmental sounds to detect anomalies in industrial noise, such as identifying high-pitched whirring sounds indicating potential mechanical failure. In the realm of audio generation, foundational models can be used to synthesize context-sensitive and auditoryally rich new sounds. By using techniques like generative adversarial networks (GANs) or diffusion models, these agents can create realistic audio simulations that mimic real-world environments or generate entirely new soundscapes for applications such as virtual reality or gaming. The adaptability of foundational models allows expert agents to be fine-tuned using domain-specific datasets, ensuring they are not only proficient in general audio tasks but also optimized for specific applications, thereby improving accuracy and effectiveness in real-world scenarios. Integrating the base model into the workflow of the audio expert agent helps to achieve more efficient and accurate audio processing methods, ultimately enhancing the system's overall ability to cope with complex audio challenges.

[0066] Each of these expert agent features allows the system to address complex audio-related challenges with precision and efficiency. By leveraging these capabilities, the system can provide deeper insights and more innovative solutions than a single entity could achieve on its own.

[0067] As explained above, RAG can enhance the capabilities of audio expert agents across a variety of tasks, including understanding, reasoning, generation, manipulation, simulation, and recommendation. By integrating retrieval mechanisms with generative models, RAG provides agents with access to a vast repository of contextual information and reference sounds, enabling more accurate and context-sensitive output.

[0068] For audio understanding, RAG can retrieve similar past audio events from a database, helping the agent better analyze new audio inputs. For example, in a factory setting, the understanding agent can use RAG to compare current recordings of mechanical noise with historical data, thereby identifying patterns that suggest specific types of mechanical wear or failure.

[0069] In audio inference, RAG supports the agent's logical reasoning by providing access to relevant audio samples or contextual data. For example, an inference agent responsible for diagnosing complex audio problems in a concert hall might retrieve past recordings of similar acoustic environments to infer the cause of unexpected echoes or feedback loops.

[0070] For audio generation, RAG can enhance the creative process by retrieving sound samples that match desired characteristics, ensuring that the generated audio meets specific requirements. In real-world scenarios, generation agents can use RAG to create soundscapes for virtual reality games, leveraging natural sound libraries to create immersive forest environments.

[0071] In audio manipulation, RAG can provide reference sound clips or effects that serve as guidelines for precise changes. For example, a manipulation agent processing podcasts might retrieve high-quality audio samples to guide enhancements in speech clarity and reductions in background noise, thereby ensuring professional sound quality.

[0072] For audio simulation, RAG can retrieve real-world audio data or environmental recordings as templates for creating highly realistic and immersive sound scenes. Simulation agents can use RAG to model the acoustic effects of new architectural designs and retrieve data from similar structures to predict how sound propagates in space.

[0073] In audio recommendation, RAG analyzes user preferences and past interactions to retrieve similar or complementary audio content and tailor suggestions to improve user engagement. For example, a recommendation agent can use RAG to suggest vehicle maintenance actions based on audio recordings of engine sounds, and to suggest potential problems and preventative measures based on historical data from similar vehicles.

[0074] By leveraging RAG, audio expert agents can perform their tasks with greater accuracy and context awareness, ultimately leading to more effective problem-solving and user satisfaction.

[0075] In an embodiment, such as Figure 3A As shown, a regulatory agent powered by an LLM can combine various sub-models320 to enhance its functionality and improve the overall efficiency of the system. These sub-models include feedback integration, logging and monitoring, data versioning, response generation, context enrichment, and query planning. Each of these sub-models can play a crucial role in optimizing system performance and ensuring accurate and effective diagnostics.

[0076] Feedback integration is a sub-model that allows the supervisory agent to continuously learn and adapt based on user interactions and outcomes. By collecting feedback from users regarding the accuracy and usefulness of the system's responses, the supervisory agent can refine its algorithms and improve future interactions. This iterative learning process helps the system become better at diagnosing problems and providing relevant solutions over time.

[0077] Logging and monitoring are crucial for maintaining system reliability and performance. Supervisory agents can track and log all interactions, including user input, system responses, and actions by expert agents. This data is invaluable for identifying patterns, diagnosing system errors, and ensuring compliance with operational standards. Monitoring also enables real-time oversight of system activity, allowing for timely intervention when any anomalies or inefficiencies are detected.

[0078] Data version control involves the evolution of the system's knowledge base. As new data is integrated and models are updated, the oversight agent can maintain different versions of datasets and algorithms. This ensures that the system can be reverted to a previous version if necessary and provides a clear change history for auditing and improvement purposes.

[0079] Response generation is a sub-model that focuses on providing consistent and context-appropriate responses to user queries. The supervisory agent leverages the natural language processing capabilities of LLM to synthesize information from expert agents and present it in a user-friendly manner. This involves selecting the most relevant data, logically organizing the data, and ensuring the language is clear and concise.

[0080] Enriched context enhances a system's ability to understand and interpret user input by incorporating additional contextual information. Supervisory agents can leverage historical data, user profiles, and environmental factors to provide a more comprehensive problem analysis. This rich context allows the system to tailor its responses more precisely to the specific circumstances of each user.

[0081] The query plan involves developing a strategy for the sequence and nature of interactions with users and expert agents. The oversight agent can determine the most efficient ways to gather necessary information, contact the appropriate expert agent, and deliver solutions. This planning ensures smooth system operation and minimizes the time and effort required to resolve issues.

[0082] By integrating these sub-models, management agents can perform their tasks more efficiently, providing a robust and adaptive framework for diagnosing and resolving device issues through a collaborative multi-agent system.

[0083] refer to Figure 3AIn System 300, expert agents are designed not only to operate autonomously within their areas of expertise but also to collaborate with other agents and share information to enhance their performance. This exchange of knowledge and insights allows agents to leverage the system's collective expertise, enabling them to perform tasks more accurately and effectively. For example, an audio understanding agent can receive contextual data from an inference agent that has analyzed the broader environment in which sound anomalies occur, providing additional clues to the refinement agent's analysis. Similarly, a generative agent responsible for creating a sound scene can incorporate feedback from a manipulation agent that has tuned the clarity and quality of audio samples to ensure the generated audio meets specific standards. This collaborative approach allows agents to cross-reference findings, validate hypotheses, and integrate different perspectives, ultimately leading to more comprehensive solutions. By leveraging shared knowledge, expert agents can adapt to complex scenarios, address multifaceted challenges, and optimize resource management, thereby enhancing the system's overall ability to respond accurately and insightfully to real-world audio problems. This interconnectedness fosters a dynamic and responsive environment in which agents continuously learn from each other over time, improving their individual and collective performance.

[0084] Furthermore, as explained above, in a multi-agent framework, multiple audio experts or agents can collaborate to accomplish complex tasks by leveraging the capabilities of LLMs. These LLMs act as central coordinators, facilitating seamless communication and coordination among agents, enabling them to work together more effectively. By interpreting and generating natural language instructions, LLMs allow agents to better understand complex tasks and collaborate effectively, adapting to changing environments and task requirements. This collaborative approach is particularly beneficial when tasks are too complex for a single agent to handle alone, such as diagnosing multifaceted audio problems or generating complex soundscapes. For example, in a scenario where the sound of a machine in operation needs to be analyzed, an understanding agent might identify sound patterns, while a reasoning agent infers underlying causes, and a manipulation agent enhances the audio to make it clearer. LLMs facilitate this collaboration by ensuring that each agent receives the information and context necessary to effectively perform its role, synthesizing their outputs into a consistent solution. This synergy not only enhances the system's problem-solving capabilities but also optimizes resource management, allowing agents to deliver more innovative and accurate solutions than they could provide individually. Through the capabilities of LLMs, a multi-agent system becomes a dynamic and adaptive entity capable of accurately and insightfully addressing complex audio challenges.

[0085] To better illustrate the use of System 300, Figure 3BThe diagram illustrates a sequence of interactions between a human user and a system according to one embodiment. Here, an understanding agent 302, an inference agent 310, and a manipulation agent 314 are utilized; however, it should be understood that any or all of the aforementioned agents may be used. The diagram illustrates how these components collaborate to diagnose and resolve audio-related problems presented by the user.

[0086] First, the user uploads an audio file and envisions the LLM with the following input: "A machine is generating a strange sound like this. Do you know what's wrong with it?" The LLM receives this input and begins its work. It first recruits Understanding 302. This component analyzes the user-uploaded audio file, identifying specific characteristics, such as periodic high-frequency squeaking. Understanding 302 interprets these auditory features to provide an initial understanding of the sound's nature. Then, via the LLM, Understanding asks whether the sound corresponds to a known type, such as "xxx".

[0087] Once the user confirms that the sound is "xxx", the LLM recruits the inference model 306. Inference 306 follows understanding 308 by evaluating the context and potential causes of the identified sound characteristics. This process considers the high-frequency components detected by understanding 308 and suggests possible sources or conditions that could lead to this audio characteristic. Inference 306 extends the analysis by deriving logical reasoning from the audio data, helping the user understand the underlying issues. The LLM may then respond to the user again, for example, "If this is the sound of xxx, then these high-frequency components are likely caused by…".

[0088] The user sees this output from the LLM and responds with their own noise theory. For example, the user might say, "I see. I thought it was because…". The LLM can view this as an opportunity to recruit the manipulation model 306. The task of manipulation 306 is to generate audio based on the analysis provided by understanding 308 and reasoning 310. If the identified cause is accurate, this component provides a simulation of the possible sound, allowing the user to compare the generated audio with the original sound. Manipulation 306 enhances the diagnostic process by providing the user with a tangible reference, facilitating a deeper understanding of the audio problem. The generated audio can be played back to the user via the user interface.

[0089] In another example of a real-world scenario, suppose a user encounters unusual noise emanating from their coffee machine and seeks assistance from system 300. The user records the noise and submits it to the supervisory agent 304. Upon receiving the audio input, the supervisory agent first uses an understanding expert agent 308 to analyze the characteristics of the sound, such as frequency and rhythm, to identify any patterns indicative of common problems in the coffee machine. The understanding agent might detect repetitive grinding noises, suggesting an underlying mechanical problem. Next, considering factors such as the coffee machine's operating history and typical failure modes, the supervisory agent recruits an inference expert agent 310 to assess the context and possible causes of the noise. The inference agent might infer that the noise is due to an misaligned grinder or a clogged water pipe. To provide a more comprehensive analysis, the supervisory agent then invokes a manipulation expert agent 314 to enhance the audio recording, filtering out background noise to more clearly isolate the sound from the coffee machine. This refined audio allows for a more accurate assessment of the problem. Finally, the supervisory agent synthesizes the insights from these expert agents and communicates a possible diagnosis to the user, suggesting specific components to check or actions to take, such as cleaning the grinder or checking the water pipe, thereby helping the user effectively resolve the problem.

[0090] Figure 4 This is a method 400 for diagnosing device problems via a chat session, according to one embodiment. The method utilizes an LLM (Local Level Model) as an interface between a human user and multiple expert audio models. The method can use... Figure 1-2 The structure shown, such as computing system 202, is implemented and executed. At 402, computing system 202 (or one or more of its processors) receives audio recordings from a human user. The audio recordings are associated with potential errors or problems with the device. For example, it could be a scream, grinding sound, etc., from an appliance, workpiece, robot, or the like.

[0091] In a 404 error, an LLM (e.g., 304) is executed to obtain additional information from the user about the audio recording. For example, the LLM might ask the user what device the sound originated from, when the sound was generated, what noticeable conditions existed during the sound's generation, and so on. This additional input can be general and can help the LLM determine which expert agent to consult.

[0092] At 406, the LLM identifies the domain of the sound based on the audio recording and additional input from the user. The domain of the sound can be referred to as a general classification of the sound, such as the type of device from which the sound is emitted. At 408, the LLM selects an expert audio agent specializing in the identified domain of sound. For example, in an example where the domain of sound relates to a coffee machine, the selected expert audio agent is an expert audio agent specializing in coffee machine sounds. This expert audio agent is one of several expert audio agents configured to analyze specific types of audio characteristics within a corresponding domain of sound.

[0093] At 408, the expert audio agent uses Retrieval Enhancement Generation (RAG) to process the audio recording to generate data related to possible causes of errors or problems associated with the audio recording. This can be communicated to the LLM. Then, at 410, the LLM communicates the possible causes of errors or problems output by the selected expert audio agent to the user. This form of communication can be performed via Chabot or other connected HMIs.

[0094] The methods and systems disclosed herein can be used in many different applications. This section provides some practical applications of the proposed systems. Figure 5 A schematic diagram illustrating the interaction between a computer-controlled machine 500 and a control system 502 according to one embodiment is shown. Figure 6-11 Various exemplary configurations in which the control system 502 can be implemented are shown.

[0095] refer to Figure 5 The computer-controlled machine 500 includes actuators 504 and sensors 506. Actuators 504 may include one or more actuators, and sensors 506 may include one or more sensors. Sensors 506 are configured to sense the condition of the computer-controlled machine 500. Sensors 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Sensors 506 may be one or more of the sensors described above; non-limiting examples of sensors 506 include microphones, cameras, video sensors, radar, lidar, ultrasonic sensors, and motion sensors. In one embodiment, sensor 506 is a microphone configured to generate audio signals of the environment within or near the computer-controlled machine 500.

[0096] The control system 502 is configured to receive sensor signals 508 from the computer-controlled machine 500. As described below, the control system 502 may also be configured to calculate actuator control commands 510 based on the sensor signals and transmit the actuator control commands 510 to the actuator 504 of the computer-controlled machine 500.

[0097] like Figure 5 As shown, the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive sensor signals 508 from sensor 506 and convert the sensor signals 508 into input signals x. In an alternative embodiment, the sensor signals 508 are received directly as input signals x, without a receiving unit 512. Each input signal x can be a part of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signals x can include data corresponding to the image recorded by sensor 506.

[0098] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine learning algorithm, such as the neural network described above. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by a non-volatile storage device 516. The classifier 514 is configured to determine an output signal y from the input signal x. Each output signal y includes information assigning one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to an actuator 504, which is configured to actuate a computer-controlled machine 500 in response to the actuator control command 510. In another embodiment, actuator 504 is configured to actuate computer-controlled machine 500 directly based on output signal y.

[0099] When actuator 504 receives actuator control command 510, actuator 504 is configured to perform an action corresponding to the relevant actuator control command 510. Actuator 504 may include control logic configured to translate actuator control command 510 into a second actuator control command for controlling actuator 504. In one or more embodiments, actuator control command 510 may be used to control a display instead of an actuator or in addition to an actuator.

[0100] In another embodiment, instead of the computer-controlled machine 500 including sensor 506, or in addition to said computer-controlled machine 500, the control system 502 includes sensor 506. Instead of the computer-controlled machine 500 including actuator 504, or in addition to said computer-controlled machine 500, the control system 502 may also include actuator 504.

[0101] like Figure 5As shown, the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. A classifier 514 of one or more embodiments (e.g., a machine learning algorithm, such as those described above) may be implemented by the control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522.

[0102] Non-volatile storage device 516 may include one or more permanent data storage devices, such as hard disk drives, optical disk drives, magnetic tape drives, non-volatile solid-state devices, cloud storage devices, or any other device capable of permanently storing information. Processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory 522. Memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0103] Processor 520 may be configured to read into memory 522 and execute computer-executable instructions residing in non-volatile storage device 516, and embody one or more machine learning algorithms and / or method techniques of one or more embodiments. Non-volatile storage device 516 may include one or more operating systems and applications. Non-volatile storage device 516 may store compilations and / or interpretations of computer programs created using various programming languages ​​and / or techniques, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, individually or in combination.

[0104] When the program instructions are executed by processor 520, the computer-executable instructions of non-volatile storage device 516 can cause control system 502 to implement one or more machine learning algorithms and / or method techniques as disclosed herein. Non-volatile storage device 516 may also include machine learning data (including data parameters) supporting the functionality, features, and processes of one or more embodiments described herein.

[0105] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon, for causing a processor to perform aspects of one or more embodiments. Essentially non-transitory computer-readable storage media can include tangible media, implemented in any method or technology, that are volatile and non-volatile, as well as removable and non-removable, for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, portable compact disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded via a network to an external computer or external storage device.

[0106] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other type of programmable data processing apparatus, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of writing that includes instructions to implement the functions, actions, and / or operations specified in a flowchart or diagram. In alternative embodiments, consistent with one or more embodiments, the functions, actions, and / or operations specified in the flowcharts and diagrams can be reordered, processed sequentially, and / or processed concurrently. Furthermore, any flowchart and / or diagram may include more or fewer nodes or blocks than illustrated consistent with one or more embodiments.

[0107] Processes, methods, or algorithms may be embodied, in whole or in part, using appropriate hardware components such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0108] The models and teachings provided here can be deployed and utilized in a variety of settings. Figure 6A schematic diagram of a control system 502 configured to control a vehicle 600, which may be at least partially autonomous or at least partially autonomous robot, is depicted. The vehicle 600 includes actuators 504 and sensors 506. Sensors 506 may include one or more microphones, video sensors, cameras, radar sensors, ultrasonic sensors, lidar sensors, and / or position sensors (e.g., GPS). One or more of these specific sensors may be integrated into the vehicle 600. In the context of sign recognition and processing as described herein, sensor 506 is a camera mounted to or integrated into the vehicle 600. Alternatively, or in addition to one or more specific sensors identified above, sensor 506 may include a software module configured to determine the state of actuator 504 upon execution. A non-limiting example of the software module includes a weather information software module configured to determine the current or future state of weather near or at locations other than the vehicle 600.

[0109] The classifier 514 of the control system 502 of vehicle 600 can be configured to detect errors or problems in the sound emitted from vehicle 600 based on the input signal x. In such an embodiment, the output signal y may include information characterizing the vicinity of an object on vehicle 600. An actuator control command 510 can be determined based on this information. For example, the actuator control command 510 can be used to avoid collisions with detected objects.

[0110] In embodiments where vehicle 600 is at least partially autonomous, actuator 504 may be embodied in the vehicle 600's brakes, propulsion system, engine, transmission system, or steering mechanism. Actuator control command 510 can be determined to control actuator 504, causing vehicle 600 to avoid collisions with detected objects. Detected objects can also be classified according to what classifier 514 deems them most likely to be, such as pedestrians or trees. Actuator control command 510 can be determined based on the classification. In cases where hostile attack may occur, the system can be further trained to better detect objects or recognize changes in lighting conditions or angles of sensors or cameras on vehicle 600.

[0111] In other embodiments where the vehicle 600 is at least partially autonomous, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be at least partially autonomous lawnmower or at least partially autonomous cleaning robot. In such embodiments, actuator control commands 510 may be determined such that the propulsion unit, steering unit, and / or braking unit of the mobile robot can be controlled, enabling the mobile robot to avoid collisions with identified objects.

[0112] In another embodiment, the vehicle 600 is at least partially autonomous in the form of a gardening robot. In such an embodiment, the vehicle 600 may use an optical sensor, as sensor 506, to determine the state of plants in the environment near the vehicle 600. The actuator 504 may be a nozzle configured to spray chemicals. Based on the species and / or state of the identified plants, an actuator control command 510 may be determined to cause the actuator 504 to spray an appropriate amount of suitable chemical onto the plants.

[0113] The vehicle 600 may be a partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include washing machines, stoves, ovens, microwave ovens, or dishwashers. In such a vehicle 600, sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case of a washing machine, sensor 506 may detect the state of the clothes inside the washing machine. Actuator control commands 510 may be determined based on the detected state of the clothes.

[0114] Figure 7 A schematic diagram of a control system 502 is depicted, which is configured to control a system 700 (e.g., a manufacturing machine), such as a stamping and cutting machine, a cutting machine, or a gun drill, of a manufacturing system 702, such as part of a production line. The control system 502 may also be configured to control an actuator 504, which is configured to control the control system 700 (e.g., the manufacturing machine).

[0115] Sensor 506 of system 700 (e.g., manufacturing machine) may be an optical sensor configured to capture one or more attributes of manufactured product 704. Classifier 514 may be configured to determine the state of manufactured product 704 based on one or more of the captured attributes. Actuator 504 may be configured to control system 700 (e.g., manufacturing machine) based on the determined state of manufactured product 704 for subsequent manufacturing steps of manufactured product 704. Actuator 504 may be configured to control system 700 (e.g., manufacturing machine) on subsequent manufactured product 706 of system 700 (e.g., manufacturing machine) based on the determined state of manufactured product 704.

[0116] Figure 8 A schematic diagram of a control system 502 is depicted, which is configured to control a power tool 800, such as a drill or actuator, having at least a partially autonomous mode. The control system 502 may be configured to control an actuator 504, which is configured to control the power tool 800.

[0117] The sensor 506 of the power tool 800 may be an optical sensor configured to capture one or more properties of the working surface 802 and / or the fastener 804 driven into the working surface 802. The classifier 514 may be configured to determine the state of the working surface 802 and / or the fastener 804 relative to the working surface 802 based on one or more of the captured properties. This state may be that the fastener 804 is flush with the working surface 802. Alternatively, the state may be the hardness of the working surface 802. The actuator 504 may be configured to control the power tool 800 such that the drive function of the power tool 800 is adjusted based on the determined state of the fastener 804 relative to the working surface 802 or one or more captured properties of the working surface 802. For example, if the state of the fastener 804 is flush with the working surface 802, the actuator 504 may interrupt the drive function. As another non-limiting example, the actuator 504 may apply additional or less torque depending on the hardness of the working surface 802.

[0118] In another embodiment, sensor 506 is a microphone configured to generate one-dimensional sound data associated with power tool 800 when power tool 800 drives fastener 804 into work surface 802. According to the disclosure provided above, classifier 514 can be configured to determine the torque of the motor of tool 800 by converting the sound emitted by the tool into a torque signal.

[0119] Figure 9 A schematic diagram is depicted of a control system 502 configured to control an automated personal assistant 900. The control system 502 can be configured to control an actuator 504, which in turn is configured to control the automated personal assistant 900. The automated personal assistant 900 can be configured to control household appliances such as a washing machine, stove, oven, microwave oven, or dishwasher.

[0120] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive video images of the user 902's posture 904. The audio sensor may be configured to receive voice commands from the user 902.

[0121] The control system 502 of the automated personal assistant 900 can be configured to determine an actuator control command 510 configured to be controlled by the control system 502. The control system 502 can be configured to determine the actuator control command 510 based on a sensor signal 508 from a sensor 506. The automated personal assistant 900 is configured to send the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 can be configured to execute a gesture recognition algorithm to recognize a gesture 904 made by the user 902, determine the actuator control command 510, and transmit the actuator control command 510 to the actuator 504. The classifier 514 can be configured to retrieve information from a non-volatile storage device in response to the gesture 904 and output the retrieved information in a form suitable for the user 902 to receive.

[0122] Figure 10 A schematic diagram of a control system 502 configured to control a monitoring system 1000 is depicted. The monitoring system 1000 can be configured to physically control entry through a door 1002. A sensor 506 can be configured to detect and determine whether entry is permitted in a relevant scenario. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. The control system 502 can use such data to detect faces.

[0123] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to determine a person's identity by interpreting image and / or video data by matching the identities of known persons stored in non-volatile storage device 516. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock door 1002 in response to the actuator control command 510. In other embodiments, non-physical logical access control is also possible.

[0124] The monitoring system 1000 can also be a surveillance system. In such an embodiment, sensor 506 may be an optical sensor configured to detect a scene under surveillance, and control system 502 is configured to control display 1004. Classifier 514 is configured to determine the classification of the scene, such as whether the scene detected by sensor 506 is suspicious. Control system 502 is configured to send actuator control command 510 to display 1004 in response to the classification. Display 1004 may be configured to adjust the displayed content in response to actuator control command 510. For example, display 1004 may highlight objects deemed suspicious by classifier 514. Using embodiments of the disclosed system, the surveillance system can predict the appearance of objects at specific times in the future.

[0125] Figure 11 A schematic diagram of a control system 502 is depicted, configured to control an imaging system 1100, such as an MRI apparatus, an X-ray imaging apparatus, or an ultrasound apparatus. A sensor 506 may be, for example, an imaging sensor. A classifier 514 may be configured to determine a classification of all or part of the sensed image. The classifier 514 may be configured to determine or select an actuator control command 510 in response to a classification obtained by a trained neural network. For example, the classifier 514 may interpret a region of the sensed image as a potential anomaly. In this case, an actuator control command 510 may be determined or selected to cause the display 1102 to display the image and highlight the region of the potential anomaly.

[0126] This specification uses the term "configuration" in conjunction with system and computer program components. For a system of one or more computers configured to perform specific operations or actions, this means that software, firmware, hardware, or a combination thereof are installed on the system, which, in operation, causes the system to perform those operations or actions. For one or more computer programs configured to perform specific operations or actions, this means that one or more programs include instructions that, when executed by a data processing device, cause the device to perform those operations or actions.

[0127] In this specification, the term "mechanism" is used broadly to refer to a software-based system, subsystem, or process programmed to perform one or more specific functions. Typically, a mechanism (e.g., a self-focusing mechanism) will be implemented as one or more software modules or components installed on one or more computers at one or more locations. In some cases, one or more computers will be dedicated to a particular mechanism; in other cases, multiple mechanisms may be installed and run on the same one or more computers. The reference to a multi-head self-focusing mechanism may also be referred to as a multi-head self-focusing machine learning model.

[0128] While exemplary embodiments have been described above, they are not intended to describe all possible forms included in the claims. The language used in this specification is descriptive rather than limiting, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As previously stated, features of various embodiments may be combined to form other embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing advantages over or preferred over other embodiments or prior art implementations in one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Accordingly, any embodiment described as less desirable than other embodiments or prior art implementations in one or more characteristics is not outside the scope of this disclosure and may be desirable for a particular application.

Claims

1. A method for diagnosing device problems via chat sessions, utilizing a large language model (LLM) as an interface between a human user and multiple expert audio models, the method comprising: Audio recordings are received from human users; Perform an LLM to obtain additional information from the user about the audio recording; The field of sound recognition based on audio recordings and additional input from the user; Select an expert audio agent that specializes in the domain of the identified sound, wherein the expert audio agent is one of a plurality of expert audio agents configured to analyze a specific type of audio characteristics in a corresponding domain of the sound domain; The audio recordings are processed via a selected expert audio agent using Retrieval Enhancement Generation (RAG) to generate data related to the possible sources of the audio in the recordings. and Perform an LLM to convey the possible sources of audio output by the selected expert audio agent to the user.

2. The method according to claim 1, wherein, The audio recording is associated with potential errors or problems with the device, and the method further includes: Execute expert audio agents to provide users with advice or solutions to resolve possible causes of device errors or problems; and The recommendations or solutions are communicated to the user via LLM.

3. The method according to claim 1, further comprising: The task is accomplished through collaboration among multiple expert audio agents via LLM.

4. The method according to claim 1, wherein, The multiple expert audio agents each specialize in different functions, including but not limited to reasoning, manipulation, and analysis.

5. The method according to claim 1, wherein, The selected expert audio agent includes multiple selected expert audio agents, each of which specializes in its own different functions.

6. The method according to claim 5, wherein, The process of processing audio recordings involves exchanging information or knowledge among several selected professional audio agents.

7. The method according to claim 6, wherein, The LLM facilitates the exchange of information or knowledge among multiple selected expert audio agents.

8. The method according to claim 1, wherein, Processing audio recordings via an expert audio agent involves using RAG to access a repository of stored reference sounds and contextual information, thereby enhancing the agent's ability to generate more accurate and context-sensitive output when diagnosing possible causes of errors or problems associated with the audio recordings.

9. A computational system for diagnosing device problems via chat sessions, utilizing a large language model (LLM) as an interface between a human user and multiple expert audio models, the computational system comprising: One or more processors are programmed to: Receive audio recordings from human users, where the audio recordings are associated with potential errors or problems with the device; Perform an LLM to obtain additional information from the user about the audio recording; Identify sound domains based on audio recordings and additional input from users; Select an expert audio agent that specializes in the identified sound domain, wherein the expert audio agent is one of a plurality of expert audio agents configured to analyze a specific type of audio characteristics in a corresponding domain of the sound domain; The audio recording is processed via a selected expert audio agent using Retrieval Enhancement Generation (RAG) to generate data related to the possible causes of errors or problems associated with the audio recording. and Execute LLM to communicate the possible causes of errors or problems output by the selected expert audio agent to the user.

10. The computing system according to claim 9, wherein, The processor is also programmed to: Perform expert audio agent to provide users with advice or solutions to resolve possible causes of device errors or problems; and The recommendations or solutions are communicated to the user via LLM.

11. The computing system according to claim 9, wherein, The processor is also programmed to: Through LLM, multiple expert audio agents collaborate on tasks to complete them.

12. The computing system according to claim 9, wherein, The multiple expert audio agents each specialize in different functions, including but not limited to reasoning, manipulation, and analysis.

13. The computing system according to claim 9, wherein, The selected expert audio agent includes multiple selected expert audio agents, each of which specializes in its own different functions.

14. The computing system according to claim 13, wherein, The processing of audio recordings involves exchanging information or knowledge among multiple selected expert audio agents.

15. The computing system according to claim 14, wherein, The LLM facilitates the exchange of information or knowledge among the multiple selected expert audio agents.

16. The computing system according to claim 9, wherein, Processing audio recordings via an expert audio agent involves using RAG to access a repository of stored reference sounds and contextual information, thereby enhancing the agent's ability to generate more accurate and context-sensitive output when diagnosing possible causes of errors or problems associated with the audio recordings.

17. A non-transitory computer-readable medium storing instructions, which, when executed by a processor, cause the processor to perform a method comprising: Receive audio recordings from human users, where the audio recordings are associated with potential errors or problems with the device; Perform an LLM to obtain additional information from the user about the audio recording; The field of sound recognition based on audio recordings and additional input from the user; Select an expert audio agent that specializes in the domain of the identified sound, wherein the expert audio agent is one of a plurality of expert audio agents configured to analyze a specific type of audio characteristics in a corresponding domain of the sound domain; The audio recordings are processed via a selected expert audio agent using Retrieval Enhancement Generation (RAG) to generate data related to the possible causes of errors or problems associated with the audio recordings. and Execute LLM to communicate the possible causes of errors or problems output by the selected expert audio agent to the user.

18. The non-transitory computer-readable medium according to claim 17, wherein, The selected expert audio agent includes multiple selected expert audio agents, each of which specializes in its own different functions.

19. The non-transitory computer-readable medium according to claim 18, wherein, The steps involved in processing audio recordings include exchanging information or knowledge among the multiple selected expert audio agents.

20. The non-transitory computer-readable medium according to claim 19, wherein, The LLM facilitates the exchange of information or knowledge among the multiple selected expert audio agents.

Images