Multi-agent task cooperative processing system and processing method for voice interaction

Abstract

The application discloses a multi-agent task cooperative processing system and method for voice interaction. The system comprises: a voice input module for receiving and preprocessing original audio signals from a user; a voice processing module for generating a structured task representation based on the audio signals using a deep neural network model; a task coordination module for assigning tasks to agents in a task processing module based on the structured task representation; the task processing module contains multiple agents, all connected to the task coordination module, each configured to process tasks in a specific field; a response generation module for integrating the processing results from the agents into a coherent response and converting it into a voice signal for output. The application integrates voice understanding, task scheduling and cooperative execution to provide a robust, efficient and intelligent voice-driven solution that can complete complex voice interaction tasks.

Description

Multi-agent task collaborative processing system and processing method for voice interaction Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a multi-agent task collaborative processing system and method for voice interaction. Background Technology

[0002] With the continuous upgrading of intelligent services, voice interaction has become an important way of human-machine collaboration, especially in professional fields such as hospital logistics, industrial operation and maintenance, and intelligent customer service. Users expect to complete complex tasks such as work order reporting, data querying, equipment control, and maintenance guidance through natural language voice. However, existing technologies have the following problems in meeting these advanced needs:

[0003] First, traditional voice interaction systems generally employ a cascaded processing pipeline architecture: "automatic speech recognition → natural language understanding → dialogue manager → natural language generation → text-to-speech synthesis." This architecture suffers from error propagation problems. Any recognition error generated by the automatic speech recognition module, such as misidentifying a keyword as another word, will be passed to the downstream natural language understanding and dialogue manager modules and may be further amplified, ultimately leading to a complete misunderstanding of the user's intent and task failure.

[0004] Secondly, existing voice assistants have limited capabilities in performing complex, multi-step tasks, especially when deep interaction with external systems is required. Most voice agents excel at answering factual questions or executing simple pre-set commands, but struggle with tasks that require accessing databases, creating records, searching internal documents, or executing a series of API calls. Summary of the Invention

[0005] To address the problems in the prior art, this invention provides a multi-agent task collaborative processing system and method for voice interaction.

[0006] This solution provides a multi-agent task collaborative processing system for voice interaction, including a voice input module, a voice processing module, a task coordination module, a task processing module, and a response generation module.

[0007] The voice input module is used to receive and preprocess the raw audio signal from the user;

[0008] The speech processing module is connected to the speech input module and is used to generate a structured task representation based on the audio signal using a deep neural network model. The structured task representation includes an intent identifier, a function identifier, and a set of function parameters.

[0009] The task coordination module is connected to the speech processing module and is used to receive the structured task representation and assign tasks to the intelligent agents in the task processing module based on the structured task representation.

[0010] The task processing module includes multiple intelligent agents, all connected to the task coordination module. Each agent is configured to handle tasks within a specific domain. Each intelligent agent includes:

[0011] The tool invocation unit is used to execute external functions or call application programming interfaces based on the received function identifier and function parameters.

[0012] The knowledge retrieval unit is used to query the knowledge base to retrieve relevant information to assist in generating a response.

[0013] The response generation module is connected to the task processing module and is used to integrate the processing results from the agent into a coherent response and convert it into a speech signal for output.

[0014] Preferably, the voice processing module includes:

[0015] An acoustic encoder unit is used to encode the received audio signal into a high-dimensional hidden representation sequence;

[0016] The task encoder unit is used to decode the hidden representation sequence into a structured task representation containing an intent identifier, a function identifier, and a set of function parameters.

[0017] Preferably, the acoustic encoder unit includes:

[0018] A convolutional neural network layer, used to extract local acoustic time-frequency features of the audio signal, specifically includes:

[0019] Apply a short-time Fourier transform to the overlapping frames of the audio signal to convert the audio signal into a two-dimensional time-frequency representation;

[0020] Multiple two-dimensional convolutional filters are applied to the two-dimensional time-frequency representation to detect local acoustic time-frequency features;

[0021] A recurrent neural network layer, used to capture the long-range dependencies of the local acoustic video features to generate a high-dimensional hidden representation sequence, specifically includes:

[0022] At each time step, a time-frequency feature from the convolutional neural network layer is received as the current input;

[0023] The hidden state of the current time step is updated by combining the current input and the hidden state of the previous time step;

[0024] The sequence of hidden states updated at all time steps is output as the high-dimensional hidden representation sequence.

[0025] Preferably, the model in the speech processing module is jointly optimized using a multi-task loss function, which is expressed as:

[0026] ;

[0027] In the formula, For multi-tasking loss, Cross-entropy loss is used to optimize the accuracy of intent classification. This is a temporal classification loss used to optimize the generation quality of the parameter sequence. For speech recognition loss, , and These are the hyperparameters determined during model training.

[0028] Preferably, the speech processing module also generates a confidence score corresponding to the intent category.

[0029] Preferably, the step of assigning tasks to agents in the task processing module based on the structured task representation specifically includes:

[0030] Obtain the intent category and the confidence score corresponding to the intent category from the structured task representation;

[0031] Based on the intent category and confidence score, a routing function is used to calculate the probability of assigning the intent category to each agent in the task processing module;

[0032] Assign tasks to the agents with the highest probability;

[0033] The mathematical expression for the routing function is:

[0034] ;

[0035] In the formula, The task is represented by the intent category and the confidence score corresponding to that intent category. The agent that corresponds to this task.

[0036] Preferably, the knowledge retrieval unit generates an auxiliary response based on retrieval enhancement, specifically including:

[0037] The received structured task representation is transformed into a query vector using a query encoder;

[0038] In a pre-built knowledge base that stores domain document vectors, the cosine similarity between the query vector and all document vectors is calculated.

[0039] Select the original text content of the k documents with the highest similarity and use this content as background knowledge or context to combine it with the original user question;

[0040] The concatenated user question is input into a generative language model, which then generates a natural language answer.

[0041] Another aspect of this solution provides a multi-agent task collaborative processing method for voice interaction, specifically including:

[0042] Receive a raw audio signal using the voice input module;

[0043] The original audio signal is processed by a neural network model in the speech understanding module to generate a structured task representation that includes intent identifiers, function identifiers, and function parameters;

[0044] Based on the intent identifier, the task coordination module routes the structured task representation to one of the multiple agents in the task processing module.

[0045] A task is performed by invoking an external tool based on the function identifier and parameters through a selected agent;

[0046] Based on the result of the task being performed, a voice response is generated.

[0047] Compared with the prior art, the advantages of this invention are:

[0048] By eliminating traditional cascading pipelines, error propagation problems are fundamentally avoided, significantly improving the system's robustness and efficiency in understanding user intent. An end-to-end neural network model directly maps speech to executable task instructions, enabling direct voice control of complex tasks. Deep integration of tool invocation and knowledge retrieval allows the system to not only complete tasks but also generate corresponding explanations and descriptions, ensuring accurate responses and rich information, providing users with a more intelligent, reliable, and efficient voice interaction experience. Attached Figure Description

[0049] Figure 1 is a system architecture diagram of the multi-agent task collaborative processing system for voice interaction proposed in this invention.

[0050] Figure 2 is a flowchart of the multi-agent task collaborative processing method for voice interaction proposed in this invention. Detailed Implementation

[0051] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0052] Example 1:

[0053] Referring to Figure 1, this embodiment provides a detailed description of the multi-agent task collaborative processing system for voice interaction included in the scheme. The system includes a voice input module, a voice processing module, a task coordination module, a task processing module, and a response generation module.

[0054] The voice input module receives and preprocesses raw audio signals from the user. These raw audio signals can originate from a telephone line, a local microphone, or a recorded audio file. The module also integrates signal preprocessing algorithms for noise suppression, echo cancellation, and voice activity detection to improve the quality of the input audio and provide a clearer signal for subsequent processing.

[0055] Speech activity detection is used to accurately distinguish between segments containing human speech and non-speech segments (such as silence or background noise) in a continuous audio stream. The speech activity detection algorithm in this embodiment employs a machine learning model to analyze the spectral characteristics of the audio and output a speech presence probability for each audio frame, thereby achieving accurate detection of speech activity.

[0056] After speech activity detection identifies speech and non-speech segments, the system initiates a noise suppression process to improve the signal-to-noise ratio. This embodiment employs spectral subtraction. The specific steps are as follows: First, the average spectral characteristics of the background noise are estimated using the non-speech segments detected by speech activity detection, forming a noise profile. Then, the entire audio signal (including speech segments) is transformed to the frequency domain using a Fast Fourier Transform. Finally, the previously estimated noise spectrum is subtracted from the spectrum of the speech segments, thereby suppressing the background noise.

[0057] Echo cancellation is used to eliminate the echo generated when sound played from a speaker (far-end signal) is re-picked up by a microphone. In this embodiment, echo cancellation works through an adaptive filter. This filter continuously models the acoustic path from the speaker to the microphone. It uses the far-end signal as a reference to generate a predicted signal that is highly similar to the actual echo, and then subtracts this predicted signal from the mixed signal captured by the microphone, thereby effectively eliminating the echo and preserving only the clean speech of the near-end user.

[0058] The speech processing module is connected to the speech input module and is used to generate a structured task representation based on the audio signal using a deep neural network model. The structured task representation includes an intent identifier, a function identifier, and a set of function parameters. This module includes an acoustic encoder unit and a task decoder unit.

[0059] The input audio sequence is first processed by an acoustic encoder unit. The acoustic encoder consists of multi-layer convolutional neural networks and recurrent neural networks or their variants (such as Long Short-Term Memory (LSTM) or gated recurrent units (GRUs). The convolutional neural network layers are used to extract local acoustic time-frequency features, specifically:

[0060] Apply a short-time Fourier transform to the overlapping frames of the audio signal to convert the audio signal into a two-dimensional time-frequency representation.

[0061] Multiple two-dimensional convolutional filters are applied to the two-dimensional time-frequency representation to detect local acoustic time-frequency features.

[0062] The recurrent neural network layer is used to capture the long-range dependencies of these features over time to generate a high-dimensional hidden representation sequence, specifically:

[0063] At each time step, a time-frequency feature from the convolutional neural network layer is received as the current input.

[0064] The hidden state of the current time step is updated by combining the current input and the hidden state of the previous time step.

[0065] The sequence of hidden states updated at all time steps is output as the high-dimensional hidden representation sequence.

[0066] The hidden representation sequence is then fed into the task decoder unit. The decoder can employ an attention-based sequence-to-sequence architecture. It learns to directly decode the encoded acoustic context into a structured task representation:

[0067] ;

[0068] in, To hide the representation sequence, For task decoder functions, This indicates the user's intent category, such as "report repair," "inquiry," or "guidance." This indicates the identifier of the specific tool or API that needs to be invoked to complete the intent. Parameters represent the key-value pairs of parameters required to execute the tool or API, such as the location related to the work order, the device ID, or the date range related to the query.

[0069] To jointly optimize all the above tasks, the model training uses a weighted multi-task loss function:

[0070] ;

[0071] in, For multi-tasking loss, Cross-entropy loss is used to optimize the accuracy of intent classification. This is a temporal classification loss used to optimize the generation quality of the parameter sequence. For speech recognition loss, , and These are the hyperparameters determined during model training.

[0072] In this scheme, by introducing an auxiliary ASR task and adding its loss to the total loss with a small weight λ3, a powerful regularization effect is achieved. This forces the acoustic encoder to learn hidden representations that are not only relevant to the semantic task but also retain sufficiently rich underlying acoustic and phoneme information to reconstruct the original text transcription. This design makes the features learned by the encoder more robust to various acoustic variations (such as background noise, different accents, and speech rate changes), thereby indirectly improving the performance and generalization ability of the main tasks (intent and parameter recognition) and effectively addressing the speech recognition challenges in applications.

[0073] The task coordination module is connected to the speech processing module and is used to receive the structured task representation and assign tasks to agents in the task processing module based on the structured task representation. Specifically, it includes:

[0074] Obtain the intent category and the confidence score corresponding to the intent category from the structured task representation.

[0075] Based on the intent category and confidence score, a routing function is used to calculate the probability of assigning the intent category to each agent in the task processing module.

[0076] The task is assigned to the agent with the highest probability.

[0077] The mathematical expression for the routing function is:

[0078] ;

[0079] The goal of this function is to, given a task representation Under the given conditions, find the sub-agent most likely to successfully handle the task. Among them, the probability distribution This can be achieved in the following ways:

[0080] Rule-based mapping: One or more corresponding agents are hard-coded for each intent. This method is suitable for scenarios where intents are strongly correlated with domain functions.

[0081] Training the classifier: Train a lightweight classification model (such as logistic regression, support vector machine, or small neural network) to represent the task. The features (such as intent and the occurrence of parameters) are taken as input, and the most suitable agent category is output.

[0082] Understandably, in real-world interactions, user expressions may be ambiguous or lack information. The task coordination module presets a confidence threshold. When the highest probability value calculated by the routing function is lower than the preset confidence threshold, or when the identified parameters are insufficient to invoke any agent's tools, the task coordination module initiates a clarification sub-dialogue. It generates an interrogative question, such as "Do you want to create a new repair order or query an existing one?", or "Please provide the device number you are querying." The user's next round of voice input will be used to supplement the missing information, and then the task coordination module will re-make routing decisions. This proactive clarification mechanism greatly improves the success rate of interactions and the user experience.

[0083] The task processing module contains multiple agents, all connected to the task coordination module, and each configured to handle tasks within a specific domain. For example, the work order agent is specifically responsible for interacting with the work order management system, while the energy management agent is used to query and analyze energy consumption data. This specialization allows each agent to have a highly focused toolset and knowledge base, thereby improving processing efficiency and accuracy.

[0084] Each agent includes: a tool invocation unit, used to execute external functions or call application programming interfaces based on the received function identifier and function parameters; and a knowledge retrieval unit, used to query the knowledge base to retrieve relevant information to assist in generating a response.

[0085] When the task is execution-oriented, the sub-agent activates its tool invocation unit. This is based on the information passed from the task coordination module. and Construct and execute a specific API call: Result of the call (For example, a newly created work order ID, a set of data returned from a database query, or a status code indicating successful device control) serve as important inputs for the final response.

[0086] When the task is question-and-answer or instruction-based, the agent activates its knowledge retrieval unit, which operates based on the principle of retrieval-enhanced generation. Specifically, this includes:

[0087] Query vectorization: Vectorizing user requests (based on...) The information in the query vector (or its textual form) is converted into a query vector by a query encoder.

[0088] Similarity retrieval: Calculate the cosine similarity between the query vector and all document vectors in a pre-built knowledge base that stores a large number of domain document vectors (such as user manuals and best practice guidelines).

[0089] Context enhancement: Select the original content of the k documents with the highest similarity and use this content as background knowledge or context, then combine it with the original user question.

[0090] Answer generation: The enhanced user question is input into a generative language model (such as the GPT series models), which generates a natural language answer.

[0091] The response generation module is connected to the task processing module and is used to integrate the processing results from the agent into a coherent response and convert it into a speech signal for output.

[0092] The response can be a structured response object:

[0093] ;

[0094] in, This indicates the execution status of the task (e.g., success, failure, in progress). For knowledge-based answers generated by the knowledge retrieval unit, This is confirmation information for operations that have been performed.

[0095] Finally, this structured object is fed into the TTS engine, synthesized into natural and fluent speech, and output to the user through devices such as speakers, completing a full and intelligent voice interaction.

[0096] In a preferred embodiment, the system employs a dialogue state machine to track and maintain the context state of the dialogue. After each round of interaction, the state is updated based on the current user input, the system's understanding, and the execution result.

[0097] ;

[0098] In the formula, for The context state at any given moment. for The context state at any given moment. for The task representation at a given moment for The call result generated by the time tool calling unit.

[0099] The information stored in the system can include: currently active sub-agents, recently mentioned entities (such as device IDs and work order numbers), and the status of completed tasks.

[0100] This context management capability allows users to engage in multi-turn conversations. For example, after creating a work order, a user can immediately ask, "What is the status of the work order?" The system can automatically understand what the user is referring to by using the work order ID stored in the context, without requiring the user to provide information repeatedly.

[0101] Example 2:

[0102] This embodiment illustrates the multi-agent task collaborative processing method for voice interaction included in the scheme, comprising:

[0103] Receive a raw audio signal using the voice input module;

[0104] The original audio signal is processed by a neural network model in the speech understanding module to generate a structured task representation that includes intent identifiers, function identifiers, and function parameters;

[0105] Based on the intent identifier, the task coordination module routes the structured task representation to one of the multiple agents in the task processing module.

[0106] A task is performed by invoking an external tool based on the function identifier and parameters through a selected agent;

[0107] Based on the result of the task being performed, a voice response is generated.

[0108] Example 3:

[0109] This embodiment illustrates the coordination mechanism between the tool invocation unit and the knowledge retrieval unit in the intelligent agent of this scheme through an example.

[0110] For example, when a user says, "The air conditioner on the third floor is leaking water again, what should be done according to regulations?"

[0111] The task coordination module assigns this request to the "facility operation and maintenance agent".

[0112] The agent first executes a tool call: it resolves "third-floor air conditioner" as the device identifier, queries the maintenance database, and finds that the device has two records of water leakage in the past six months.

[0113] Simultaneously or subsequently, it performs a knowledge retrieval: using "Regulations for Handling Repeated Air Conditioner Leaks" as the query, it retrieves relevant standard operating procedures from the knowledge base through a retrieval enhancement generation mechanism. The content may include "For equipment that experiences the same fault three times in a row, the work order should be upgraded to P1 priority, and an engineer should be assigned to perform a systematic diagnosis."

[0114] Finally, the response generation module intelligently integrates the information from these two sources to generate the following comprehensive voice response: "This is the third leak record for this device in the past six months. According to operating procedures, such recurring issues require escalation. I have created a P1 priority work order for you and assigned an engineer to conduct a thorough inspection. In the meantime, it is recommended that you shut off the water supply valve next to the device."

[0115] In summary, by combining real-time data (tool invocation) and static knowledge (knowledge retrieval), the system provides a solution with diagnostic and decision support capabilities that far surpasses simple question-and-answer or command execution.

[0116] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0117] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A multi-agent task collaborative processing system for voice interaction, characterized in that, The system includes a voice input module, a voice processing module, a task coordination module, a task processing module, and a response generation module. The voice input module receives and preprocesses raw audio signals from the user. The voice processing module, connected to the voice input module, generates a structured task representation based on the raw audio signals using a deep neural network model. The structured task representation includes an intent identifier, a function identifier, and a set of function parameters. The task coordination module, connected to the voice processing module, receives the structured task representation and assigns tasks to agents in the task processing module based on the structured task representation. The task processing module contains multiple agents, all connected to the task coordination module. Each agent is configured to handle tasks within a specific domain. Each agent includes: a tool invocation unit, used to execute external functions or call application programming interfaces based on received function identifiers and function parameters; and a knowledge retrieval unit, used to query a knowledge base to retrieve relevant information to assist in generating a response. The response generation module is connected to the task processing module and is used to integrate the processing results from the agents into a coherent response and convert it into a speech signal for output. The speech processing module includes: an acoustic encoder unit, used to encode the received audio signal into a high-dimensional hidden representation sequence; and a task encoder unit, used to decode the hidden representation sequence into a structured task representation containing an intent identifier, a function identifier, and a set of function parameters. The acoustic encoder unit includes: a convolutional... The network layer is used to extract local acoustic time-frequency features of the audio signal, specifically including: applying a short-time Fourier transform to the overlapping frames of the audio signal to convert the audio signal into a two-dimensional time-frequency representation; applying multiple two-dimensional convolutional filters to the two-dimensional time-frequency representation to detect local acoustic time-frequency features; and a recurrent neural network layer is used to capture the long-range dependencies of the local acoustic time-frequency features to generate a high-dimensional hidden representation sequence, specifically including: receiving a time-frequency feature from the convolutional neural network layer as the current input at each time step; updating the hidden state of the current time step by combining the current input and the hidden state of the previous time step; and outputting the sequence of hidden states updated at all time steps as the high-dimensional hidden representation sequence. The model in the speech processing module uses a multi-task loss function for joint optimization, which is expressed as: L = λ1L intent +λ2L param +λ3L ASR In the formula, L represents the multi-task loss, L intent Cross-entropy loss is used to optimize the accuracy of intent classification. param L is the temporal classification loss, used to optimize the generation quality of the parameter sequence. ASR λ1, λ2, and λ3 are the speech recognition loss parameters, determined during model training.

2. The multi-agent task collaborative processing system for voice interaction according to claim 1, characterized in that, The speech processing module also generates a confidence score corresponding to the intent category.

3. The multi-agent task collaborative processing system for voice interaction according to claim 2, characterized in that, The process of assigning tasks to agents in the task processing module based on the structured task representation specifically includes: obtaining the intent category and the confidence score corresponding to the intent category in the structured task representation; calculating the probability of assigning the intent category to each agent in the task processing module using a routing function based on the intent category and the confidence score; and assigning the task to the agent with the highest probability. The mathematical expression of the routing function is: A route (y)=maxP((Agent) i |y); where y represents the task, i.e., the intent category and the confidence score corresponding to that intent category, (Agent) i The agent that corresponds to this task.

4. The multi-agent task collaborative processing system for voice interaction according to claim 3, characterized in that, The knowledge retrieval unit generates an assisted response based on retrieval enhancement, specifically including: converting the received structured task representation into a query vector through a query encoder; calculating the cosine similarity between the query vector and all document vectors in a pre-built knowledge base storing domain document vectors; selecting the original content of the k documents with the highest similarity, using this content as background knowledge or context, and concatenating it with the original user question; and inputting the concatenated user question into a generative language model, which generates a natural language answer.

5. A processing method for a multi-agent task collaborative processing system for voice interaction as described in claim 4, characterized in that, Specifically, it includes: A raw audio signal is received using a voice input module; the raw audio signal is processed by a neural network model in a voice understanding module to generate a structured task representation containing an intent identifier, a function identifier, and function parameters; based on the intent identifier, the structured task representation is routed to one of multiple agents in a task processing module using a task coordination module; and through the selected agent, an external tool based on the function identifier and parameters is invoked to perform a task. Based on the result of the task being performed, a voice response is generated.

Images