A multi-agent based voice interaction method

Abstract

The application provides a Multi-Agent-based voice interaction method, comprising the following steps: S1, waking up a voice assistant, collecting voice data input by a user, converting the collected voice data into a text input through an STT voice-to-text module, and storing the text input into a historical dialogue database; through a weighted intent classification algorithm using an intent classification large model and a preset vector library, the accurate distribution of user tasks is significantly improved. At the same time, smart home and device operation are set up as independent agents, reducing the burden of the intent classification large model and improving the intelligence and accuracy of the system. The design ensures that the system link is simple and efficient, while providing highly personalized services, adapting to various devices and application scenarios, and improving the user experience.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of voice interaction, more particularly, to a voice interaction method based on Multi-Agent.

[0002] With the increasing popularity of intelligent devices and artificial intelligence technology, voice interaction assistants have gradually become an indispensable tool in our daily life. These assistants not only can accurately recognize and respond to user needs, but also provide a variety of services such as smart home control, music playback, knowledge question and answer, real-time weather query and news information, greatly improving the convenience and comfort of life. They enrich the user's information acquisition method and enhance the overall experience of daily life.

[0003] However, the voice assistants on the market currently generally rely on a single intelligent processing unit to execute tasks and parse instructions, which has obvious limitations. First of all, when dealing with complex or multi-level conversations, these systems can usually only perform simple rule matching and command recognition, and it is difficult to deeply analyze the user's deep intention. Especially when dealing with multiple tasks at the same time, the performance of existing systems is often not satisfactory. In addition, traditional rule-based voice assistants perform poorly in understanding semantics and user emotions, making users often feel that they are only interacting with mechanical programs, and lacking the necessary personification experience.

[0004] Therefore, it has become an important technical requirement today to develop a voice interaction method based on Multi-Agent. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a voice interaction method based on Multi-Agent that not only improves the user's interaction experience, but also expands the application scenarios of voice assistants, in view of the deficiencies of the above technical solutions.

[0006] The present application provides a voice interaction method based on Multi-Agent, the method comprising the following steps:

[0007] S1, wake up the voice assistant, collect the user's input voice data, and convert the collected voice data into text input through the STT voice-to-text module, and store the text input to the historical dialogue database;

[0008] S2, extract the historical dialogue data corresponding to the user and the dialogue from the historical dialogue database, and transmit the text in the historical dialogue data and the historical dialogue data to the intent classification large model Agent_Classifer(), the intent classification large model Agent_Classifer() classifies the text and the historical dialogue data to obtain the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene, calculates the possibility of each scene, and judges which scene has the highest possibility;

[0009] S3, according to the possibility of the scene, select appropriate knowledge supplement mode, external tool calling, smart home task dispatching verification and processing and device manipulation task dispatching verification, input the supplemented knowledge or rejected text into the question and answer large model, the question and answer large model generates a final text output in real time, and sends the text output, the specified home device instruction and the device manipulation completion specified operation instruction to the TTS text to speech module to generate a voice reply.

[0010] The Multi-Agent-based voice interaction method provided by the application comprises the following steps:

[0011] S21, reconstruct the text input into a multi-round text input text, extract the historical dialogue data history corresponding to the user and the dialogue, and store the text input text into the historical dialogue database History(text); wherein the historical data corresponding to the user and the dialogue is expressed as History(text) = history + text, that is, the historical dialogue data history plus the text input text in this round;

[0012] S22, fine-tune the intent classification large model Agent_Classifer(), and call the fine-tuned intent classification large model Agent_Classifer() to classify the reconstructed multi-round text input History(text) to obtain the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene, and calculate the possibility of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene; wherein the calculation formula of the possibility vector of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene is as follows:

[0013]

[0014] wherein R1 represents the possibility vector of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene, wherein SH represents the smart home scene, DC represents the device manipulation scene, Tools represents the tool calling scene, and QA represents the question and answer scene; wherein respectively represent the possibility ranking of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene.

[0015] S23, the text input text is converted into a vector Input_Vector, and is stored to a preset vector library Categpry_Vector, and the similarity of the text input text and the preset vector library Categpry_Vectors of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene is calculated, and the possibility ranking R2 is obtained, and the calculation formula of the possibility ranking is as follows:

[0016]

[0017] Wherein, respectively represent the possibility ranking of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene.

[0018] In the Multi-Agent-based voice interaction method provided by the application, the step S2 further comprises the following steps:

[0019] S24, the possibility vector R1 of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene and the possibility ranking R2 of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene are weightedly averaged to obtain the final classification scene R final , and the possibility of that scene is highest according to the final classification scene R final ; wherein the final classification scene R final is calculated by the formula

[0020]

[0021] Wherein, alpha and beta represent weights.

[0022] In the Multi-Agent-based voice interaction method provided by the application, the step S3 comprises the following steps:

[0023] S31, if the final classification scene R final is the question and answer scene, that is, the weighted ranking value of the question and answer scene is less than that of all other scenes, so that the possibility of the question and answer scene is highest; if the final classification scene R final is the question and answer scene and If Rag ==1, the RAG mode is enabled for knowledge supplement; if the final classification scene R final is the question and answer scene and If RagIf R is equal to 0, then the knowledge is supplemented by a multi-engine crawler search mode, and when the knowledge supplementing is completed, a final input set Final_Input containing a multi-round text input History(text) and supplemented knowledge Knowledge is output, wherein Final_Input=(History(text), Knowledge). Rag The mark bit is represented.

[0024] The Multi-Agent-based voice interaction method provided by the application further comprises the following steps:

[0025] S32, if the final classification scene R final is a tool calling scene, that is, The weighted ranking value of the tool calling scene is less than that of all other scenes, so that the tool calling scene has the highest possibility, and a specified tool is called to obtain supplemented information, and a final input set Final_Input containing a multi-round text input History(text) and supplemented information Supplement_Info is generated.

[0026] The Multi-Agent-based voice interaction method provided by the application further comprises the following steps:

[0027] S33, if the final classification scene R final is a smart home scene, that is, The weighted ranking value of the tool calling scene is less than that of all other scenes, so that the tool calling scene has the highest possibility, and a specified tool is called to obtain supplemented information, and a final input set Final_Input containing a multi-round text input History(text) and supplemented information Supplement_Info is generated.

[0028] The Multi-Agent-based voice interaction method provided by the application further comprises the following steps:

[0029] S34, if the final classification scene R final is a device manipulation scene, that is, wherein, If the weighted ranking value of the device manipulation scene represents the device manipulation scene is less than all other scenes, then the possibility of the device manipulation scene is the highest; it is determined whether it belongs to the device manipulation category, if it belongs to the device manipulation category, the device to be called and the function to be executed by the user are analyzed, and the user demand is converted into a device calling statement and executed, if it does not belong to the device manipulation category, it is rejected.

[0030] In the Multi-Agent-based voice interaction method of the present application, the input of the knowledge supplemented in steps S31 to S32 or the multi-round text input rejected in step S32 is input into the question and answer large model in step S3, the question and answer large model generates a final text output in a streaming manner, and the text output is sent to a TTS text-to-speech module in real time to generate a voice reply.

[0031] The Multi-Agent-based voice interaction method of the present application significantly improves the accurate distribution of user tasks by using the weighted intent classification algorithm of the intent classification large model and the preset vector library. At the same time, the smart home and device manipulation are set up as independent agents, reducing the burden of the intent classification large model and improving the intelligence and accuracy of the system. This design ensures that the system link is simple and efficient, while providing highly personalized services, adapting to various devices and application scenarios, and improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flowchart of the Multi-Agent-based voice interaction method of the present application;

[0033] Figure 2 is a flowchart of an embodiment of the Multi-Agent-based voice interaction method of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0035] It should be noted that the terms "first", "second", and the like in the description and in the claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0036] As Figures 1-2 shown, Figure 1 is a flowchart of an embodiment of a Multi-Agent-based voice interaction method of the present application. A Multi-Agent-based voice interaction method is provided, comprising the following steps:

[0037] In step S1, the voice assistant is woken up, voice data input by the user is collected, and the collected voice data is converted into text input through an STT voice-to-text module, and the text input is stored in a historical dialogue database;

[0038] In step S2, historical dialogue data corresponding to the user and the dialogue in the historical dialogue database is extracted, and the text in the historical dialogue data and the historical dialogue data are transmitted to an intent classification large model Agent_Classifer(), the intent classification large model Agent_Classifer() classifies the text and historical dialogue data to obtain an intelligent home scene, a device manipulation scene, a tool calling scene and a question and answer scene, calculates the possibility of each scene, and determines which scene has the highest possibility;

[0039] In step S3, according to the possibility of the scene, appropriate knowledge supplementing methods, external tool calling, intelligent home task dispatching verification and processing, and device manipulation task dispatching verification are selected, the supplemented knowledge or rejected text input is input into a question and answer large model, the question and answer large model generates a final text output in real time, and the text output, the called specified home device instruction and the device manipulation completed specified operation instruction are sent to a TTS text-to-speech module to generate a voice reply.

[0040] In this embodiment, the step S2 comprises the following steps:

[0041] In step S21, the text input is reconstructed into a multi-turn text input text, the history dialogue data history corresponding to the user and the conversation is extracted, and the text input text is stored in the history dialogue database History(text); wherein the history data corresponding to the user and the conversation is expressed as History(text) = history + text, that is, the history dialogue data history plus the text input text of this round;

[0042] In step S22, the intent classification large model Agent_Classifer() is fine-tuned, and the fine-tuned intent classification large model Agent_Classifer() is called to classify the reconstructed multi-turn text input History(text) to obtain the smart home scene, device manipulation scene, tool calling scene and question and answer scene, and calculate the possibility of the smart home scene, device manipulation scene, tool calling scene and question and answer scene; wherein the calculation formula of the possibility vector of the smart home scene, device manipulation scene, tool calling scene and question and answer scene is as follows:

[0043]

[0044] wherein R1 represents the possibility vector of the smart home scene, device manipulation scene, tool calling scene and question and answer scene, wherein SH represents the smart home scene, DC represents the device manipulation scene, Tools represents the tool calling scene, and QA represents the question and answer scene; wherein respectively represent the possibility ranking of the smart home scene, device manipulation scene, tool calling scene and question and answer scene;

[0045] Specifically, the value of each dimension represents the possibility ranking of the scene, and if that is, the large model determines that the subsequent task is most likely to require the smart home module to process. For example, if the possibility ranking of the current input belongs to the smart home is second

[0046] In step S23, the text input text is converted into a vector Input_Vector and stored in a preset vector library Categpry_Vector, and the similarity between the text input text and the preset vector library Categpry_Vectors of the smart home scene, device manipulation scene, tool calling scene and question and answer scene is calculated to obtain the possibility ranking R2, and the calculation formula of the possibility ranking is as follows:

[0047]

[0048] wherein, The possibility ranking of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene respectively.

[0049] In the embodiment, the step S2 further includes the following steps:

[0050] In step S24, the possibility vectors R1 of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene and the possibility ranking R2 of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene are weighted and averaged to obtain a final classification scene R final , and the possibility of the scene is determined according to the final classification scene R final ; wherein the final classification scene R final is the weighted average of the possibility vectors R1 and the possibility ranking R2.

[0051]

[0052] wherein α and β represent weights, and α and β can be preset according to actual experience or dynamically adjusted according to classification performance. For example, α and β are preset as 0.8 and 0.2 based on experience, and R final = 0.8·R1+0.2·R2= [0.8·1+0.2·1, 0.8·4+0.2·4, 0.8·2+0.2·3, 0.8·3+0.2·

[0053] 2] = [1, 4, 2.2, 2.8].

[0054] In the embodiment, the step S3 includes the following steps:

[0055] In step S31, if the final classification scene R final is the question and answer scene, i.e. , the weighted ranking value of the question and answer scene is less than that of all other scenes, and the possibility of the question and answer scene is the highest; if the final classification scene R final is the question and answer scene and If Rag == 1, the RAG mode is enabled to supplement knowledge; if the final classification scene R final is the question and answer scene and If Rag == 0, the knowledge is supplemented through a multi-engine crawler search mode, and when the knowledge supplement is completed, a set Final_Input of a multi-round text input Histoty(text) and supplemented knowledge Knowledge is output, wherein Final_Input = (History(text), Knowledge), and If Rag represents a flag bit.

[0056] In the embodiment, the step S3 further includes the following steps:

[0057] In step S32, if the final classification scene R final is a tool invocation scene, i.e. The weighted ranking value of the tool invocation scene is less than that of all other scenes, indicating that the tool invocation scene has the highest possibility; a designated tool is invoked to obtain supplementary information, and a supplementary information Supplement_Info is returned after the tool invocation ends, and a final set Final_Input containing multiple rounds of text input History(text) and supplementary information Supplement_Info is generated.

[0058] In this embodiment, the step S3 further includes the following steps:

[0059] In step S33, if the final classification scene R final is a smart home scene, i.e. The weighted ranking value of the tool invocation scene is less than that of all other scenes, indicating that the tool invocation scene has the highest possibility; it is determined whether it belongs to the smart home category; if it belongs to the smart home category, the home device to be invoked and the function to be executed by the user are analyzed, and the user's demand is converted into a smart home invocation statement and executed, and is encapsulated as SHAgent(), and if it does not belong to the smart home category, it is rejected.

[0060] In this embodiment, the step S3 further includes the following steps:

[0061] In step S34, if the final classification scene R final is a device manipulation scene, i.e. wherein, The weighted ranking value of the tool invocation scene is less than that of all other scenes, indicating that the tool invocation scene has the highest possibility; it is determined whether it belongs to the smart home category; if it belongs to the smart home category, the home device to be invoked and the function to be executed by the user are analyzed, and the user's demand is converted into a smart home invocation statement and executed, and is encapsulated as SHAgent(), and if it does not belong to the smart home category, it is rejected.

[0062] In this embodiment, in the step S3, the multiple rounds of text input with supplemented knowledge or rejected in the steps S31 to S32 are input into a large question-answering model, the large question-answering model generates a final text output in real time, and the text output is sent to a TTS text-to-speech module to generate a voice reply.

[0063] The application greatly improves the task distribution accuracy in complex applications of multiple scenes and multiple devices by integrating multiple intelligent processing agents (Agents) and using an optimized intent classification algorithm. The multi-agent design not only effectively reduces the burden of the intent recognition module, but also enhances the intelligence of the system and the accuracy of understanding user intent. Through an efficient collaboration mechanism, the system achieves faster and more accurate speech understanding and response. In addition, combined with RAG knowledge base retrieval, multi-engine crawler and tool calling module, the system makes full use of rich historical data and real-time information to enhance the relevance and accuracy of interaction. The system is based on a powerful and flexible multi-agent model platform, with excellent adaptability and scalability, suitable for various hardware and software environments. Although the number of agents is increased, the reasonable architecture design avoids significant increase in link complexity, ensuring that users have an efficient and natural interaction experience. The system can quickly reorganize and adjust according to specific application requirements, making the product highly portable and market adaptable, widely applicable to smart interaction scenarios such as in-vehicle assistants, smart home control, personal life assistants, etc., meeting users' diverse needs.

[0064] Specifically, the STT speech-to-text module is used to convert the user's voice input into text for subsequent processing, and the intent classification large model is used to analyze the user's text input and classify it into different intent categories. The large language model (LLM) fine-tuned, i.e. the intent recognition large model, combines with the preset vector library to output the final intent classification scenario through weighted average. According to the intent recognition result, the supplementary knowledge mode is selected and obtained. The supplementary knowledge mode includes RAG (Retrieval Augmented Generation) and multi-engine crawler search to ensure efficient knowledge expansion. The tool calling is used to parse and execute the tool calling statement generated in the intent classification large model, complete the calling of specific tools and obtain the corresponding supplementary information. Smart home is used to determine whether the task distributed by the intent classification large model belongs to the smart home control category, and execute the corresponding home device operation. Device manipulation is used to analyze the user's instructions and operate the specified device, such as a mobile phone or a computer, to complete the calling of specific functions. The question and answer large model is used to receive and process all multi-turn text inputs and supplementary information, generate the final text output, and deliver it to the TTS text-to-speech module for speech generation in real time. And the generated voice replies to the user. The RAG knowledge base storage is used to retrieve augmented knowledge data, while supporting the knowledge expansion function of the system. The historical dialogue database is used to store the user's historical dialogue records to support multi-turn dialogue and context understanding.

[0065] When the system carrier continuously circulates to listen to the environment, and transmits the content listened to to the wake-up recognition module for analysis. Once the user-set wake-up word is recognized, the system will issue a voice interaction prompt to guide the user to input, and start the Multi-Agent core.

[0066] Collect the user's speech input, and filter the blank part in the input according to the preset blank threshold. Then, call the STT (Speech-to-Text) speech-to-text module to convert the user's speech input into text input. text = STT (Filter (Speech_Input, Blank_Threshold)

[0067] Where Speech_Input is the collected user's single round speech input, and Blank_Threshold is the set blank threshold. According to the threshold, the blank part such as pause in the input is filtered.

[0068] In this embodiment, when the user's single round speech input Speech_Input is 5 seconds long, and the set Blank_Threshold is 200 milliseconds. According to the threshold, the blank part of 200 milliseconds and above in the user's input speech is filtered. After recognition, a total of 3 segments of speech blank with length of 200 milliseconds or above are detected, which are 300 milliseconds, 250 milliseconds and 450 milliseconds. After blank filtering, the effective length of the user's input speech is shortened to 4s = 4000ms = 5s-250ms-300ms-450ms seconds. The filtered speech is converted using the STT speech-to-text module, and the final text input by the user is: "Yesterday it rained all day, it's really annoying, the weather is good today, help me open the curtains and let the wind blow."

[0069] That is

[0070] text =

[0071] {user: "Yesterday it rained all day, it's really annoying, the weather is good today, help me open the curtains and let the wind blow."}.

[0072] The foregoing text input is stored in the historical dialogue database and submitted to the intent classification large model together with the historical dialogue data to obtain the smart home scene, device manipulation scene, tool calling scene and question and answer scene.

[0073] In this embodiment, the user is the first round of dialogue, and there is no historical dialogue, that is, history = [], so

[0074] History (text) = [] +

[0075] {user: "Yesterday it rained all day, it's really annoying, the weather is good today, help me open the curtains and let the wind blow."} = [{user: "Yesterday it rained all day, it's really annoying, the weather is good today, help me open the curtains and let the wind blow."}].

[0076] In step S31, when the intent classification large model calculates the category, it will determine in advance which knowledge supplement method to choose, and through If Rag The flag indicates whether to use the RAG method. If the final classification scenario is question and answer (QA) and If Rag == 1, the RAG method is enabled for knowledge supplement. Otherwise, the multi-engine crawler search will be used to supplement knowledge. In addition, when the RAG method cannot be recalled, the multi-engine crawler search method will also be switched to. After the knowledge supplement is completed, a multi-round text input History(text) and a set of supplemented knowledge Knowledge Final_Input: Final_Input = (History(text), Knowledge) will be output.

[0077] In this embodiment, since The knowledge supplement module does not meet the enabling conditions, so the current round of knowledge supplement module is dormant.

[0078] In step S32, when the intent classification large model calculates the category, a tool call statement tool_call_message will be generated in advance. In the tool call, this statement will be parsed, i.e., through Tool_Fetcher() for tool call. The tool call is responsible for parsing the tool specified by the tool call statement and the parameters passed, and returns a supplement information Supplement_Info after the tool call is completed. Finally, a set Final_Input containing multi-round text input History(text) and supplement information Supplement_Info is generated.

[0079] In this embodiment, since The tool call module does not meet the enabling conditions, so the current round of tool call module is dormant.

[0080] In step S33, the smart home will determine whether the task assignment of the intent classification large model is reasonable, and will reject unreasonable assignments. For reasonable task assignments, further analysis will be performed and the specified home device will be called. The smart home module itself is an Agent intelligent agent centered on the large model, so it does not depend on the analysis of the intent classification large model and can analyze the multi-round text input History(text) by itself to determine whether it belongs to the smart home category SmartHome.

[0081] If it belongs to the smart home category, the home device called by the user and the function executed will be parsed, and the user's demand will be converted into a smart home call statement and executed. Its process is similar to the Tools module, which is encapsulated as SHAgent(). If it does not belong to the smart home category, it will be rejected Reject.

[0082] There are two ways to reject: for time-sensitive scenarios, the question and answer large model can be rejected directly to step S3, so as to quickly reply to the user; for relatively time-insensitive scenarios, the intent classification large model is redirected. The system can only be redirected at most once, and if the redirection fails, the question and answer large model is rejected.

[0083] In this embodiment, since The smart home module is enabled. After the smart home large model verification, the current user intent belongs to the smart home processing category History(text)∈SmartHome, the large model analyzes the multi-round text input History(text), and the structured feedback of the smart home operation required by the user and the operation required to be performed: furniture_call_message={"device":"curtain","action":"open",}

[0084] Subsequently, the script analyzes the above large model structured feedback, and performs the corresponding operation on the specified home.

[0085] The device manipulation judges whether the task dispatch of the intent classification large model is reasonable, rejects the unreasonable dispatch, further analyzes the reasonable task dispatch, and manipulates the device to complete the specified operation.

[0086] Specifically, the device manipulation is also an Agent intelligent body with the large model as the core, and therefore does not depend on the analysis of the intent classification large model, can analyze the multi-round text input History(text) by itself, and judge whether it belongs to the device manipulation category DeviceControl.

[0087] If it belongs to the device manipulation category, the device (such as a mobile phone, a computer) called by the user and the function executed are analyzed, the user demand is converted into a device call statement and executed, the flow is similar to the Tools module, and is encapsulated as DCAgent(). The rejection mechanism is the same as that of the smart home.

[0088] In this embodiment, since The device manipulation does not meet the device manipulation enabling condition, and therefore this round of device manipulation is dormant.

[0089] The Multi-Agent-based voice interaction method provided by the embodiment of the application has at least the following beneficial effects:

[0090] 1、The Multi-Agent-based voice assistant general framework of the application can adapt to various carrier platforms. The flexibility of the framework makes it easy to deploy on different devices. In addition, after removing the first and last voice processing links, the framework can still be used as a powerful text processing system, providing more extensive support for various applications.

[0091] 2The application significantly improves the accuracy of intent recognition by introducing an innovative intent classification algorithm that weights and averages the classification results of large models with the similarity of a preset vector library. The application of this algorithm ensures that user tasks can be more accurately distributed and processed, thereby improving the overall efficiency and user experience of the system.

[0092] 3The application sets up the smart home module as an independent agent, enabling the system to autonomously determine and execute tasks related to smart home. The smart home module, combined with the intent classification large model, can independently analyze multi-round dialogue input from users, accurately identify user needs, and intelligently schedule home devices to provide highly personalized services.

[0093] 4The application also integrates a device manipulation module and sets it up as an independent agent, enhancing the system's device control capabilities. This module can quickly identify and execute manipulation tasks involving multiple devices (such as mobile phones and computers) through independent analysis of user input. This function not only enriches the system's interaction methods but also significantly improves the convenience and efficiency of user operations.

[0094] 5The application sets up the smart home and device manipulation modules as independent agents, reducing the pressure on the intent classification large model and improving the accuracy and intelligence of understanding and processing user intent. At the same time, this architecture can verify whether the intent classification large model's division is correct, allowing the intent classification large model to only handle large domain divisions, while the execution of tasks within each small domain (such as the smart home module) is analyzed and executed by the corresponding module, enhancing the system's intelligence and accuracy.

[0095] 6The application improves the accuracy of user intent recognition without increasing the complexity of the system's chain links. Each chain link is almost the same length and relatively short, ensuring timely response to users.

[0096] It should be noted that for the above-mentioned method embodiments, in order to simplify the description, they are all expressed as a series of action combinations, but those skilled in the art should know that the application is not limited by the described action sequence, because according to the application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the application.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software necessary for a general hardware platform, and of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a part of the prior art that contributes to the present application. The computer software product is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a plurality of instructions for causing an end device (which can be a mobile phone, a computer, a server, or a network device) to execute the method described in each embodiment of the present application.

[0098] Therefore, the above description is only a preferred specific embodiment of the present application, and the protection scope of the present application is not limited thereto. Any changes or replacements easily thought of by those skilled in the art within the technical scope disclosed by the present application should be covered within the protection scope of the present application, and the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A Multi-Agent based voice interaction method, characterized in that, The method comprises the following steps: S1, wake up the voice assistant, collect the voice data input by the user, convert the collected voice data into text input through an STT voice-to-text module, and store the text input into a historical dialogue database; S2, extract the historical dialogue data corresponding to the user and the dialogue in the historical dialogue database, and transmit the text in the historical dialogue data and the historical dialogue data to the intent classification large model , the intent classification large model classify the text and the historical dialogue data to obtain an intelligent home scene, a device manipulation scene, a tool calling scene and a question and answer scene, calculate the possibility of each scene, and determine which scene has the highest possibility; S3, according to the possibility of the scene, select appropriate knowledge supplement methods, external tool calls, smart home task dispatch verification and processing, and device manipulation task dispatch verification and processing, input the supplemented knowledge or rejected text input into a large question-answering model, the large question-answering model generates a final text output in real time, and sends the text output, the specified home device instruction called, and the device manipulation instruction completing the specified operation instruction to a TTS text-to-speech module to generate a voice reply; The step S2 comprises the following steps: S21, reconstructing the text input into multi-turn text input , extracting historical dialogue data related to the user and the session , and storing the text input to the historical dialogue database ; wherein the historical data related to the user and the session is represented as , that is, the historical dialogue data plus the current turn text input ; S22, the intention classification large model is fine-tuned, and the fine-tuned intention classification large model is called on the reconstructed multi-turn text input is classified to obtain an intelligent home scene, a device manipulation scene, a tool calling scene and a question and answer scene, and the possibility of the intelligent home scene, the device manipulation scene, the tool calling scene and the question and answer scene is calculated; wherein the calculation formula of the possibility vector of the intelligent home scene, the device manipulation scene, the tool calling scene and the question and answer scene is as follows: wherein a possibility vector representing as smart home scenario, device manipulation scenario, tool invocation scenario and question answering scenario, wherein SH represents as smart home scenario, DC represents as device manipulation scenario, Tools represents as tool invocation scenario, QA represents as question answering scenario; wherein respectively represent the possibility ranking of smart home scenario, device manipulation scenario, tool invocation scenario and question answering scenario; S23, inputting the text into a vector and storing into a preset vector library and calculating the similarity of the text input with the preset vector library of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene to obtain a possibility ranking The calculation formula of the possibility ranking is as follows: ； wherein, respectively represent the possibility rankings of the smart home scene, the device manipulation scene, the tool invocation scene, and the question and answer scene. 2.The Multi-Agent based voice interaction method according to claim 1, characterized in that, The step S2 further comprises the following steps: S24, the possibility vector of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene and the possibility ranking of the smart home scene, the device manipulation scene, the tool calling scene and the question and answer scene weighted average to obtain the final classification scene and according to the final classification scene determine which scene has the highest possibility; wherein the final classification scene The calculation formula is wherein , is represented as a weight. 3.The Multi-Agent based voice interaction method of claim 2, wherein, The step S3 comprises the following steps: S31, if the final classification scenario is a question and answer scenario, i.e. , , the question and answer scenario has the highest possibility; if the final classification scenario is a question and answer scenario and , the RAG mode is enabled for knowledge supplement; if the final classification scenario is a question and answer scenario and , the multi-engine crawler search mode is used for supplementing knowledge, and after the knowledge supplement is completed, a multi-round text input and a set of supplemented knowledge Knowledge are output, wherein , indicates a flag bit.

4. The Multi-Agent based voice interaction method of claim 3, wherein, The step S3 further comprises the following steps: S32, if the final classification scenario is a tool invocation scenario, i.e. , , the weighted ranking value of the tool invocation scenario is less than all other scenarios, the tool invocation scenario has the highest possibility; and a specified tool is invoked to obtain supplementary information, and a supplementary information is returned after the tool invocation ends, and a collection containing multiple rounds of text input and supplementary information is finally generated.

5. The Multi-Agent based voice interaction method of claim 4, wherein, The step S3 further comprises the following steps: S33, if the final classification scenario is a smart home scenario, i.e. , , the possibility of being a smart home scenario is the highest; it is judged whether it belongs to the smart home category; if it belongs to the smart home category, the home equipment called by the user and the function executed are analyzed, and the user demand is converted into a smart home calling statement and executed, which is encapsulated as , if it does not belong to the smart home category, it is rejected .

6. The Multi-Agent based voice interaction method of claim 5, wherein, The step S3 further comprises the following steps: S34, if the final classification scenario is a device manipulation scenario, i.e. , wherein, the weighted ranking value of the device manipulation scenario is less than all other scenarios, the possibility of the device manipulation scenario is the highest; it is judged whether it belongs to the device manipulation category, if it belongs to the device manipulation category, the device called by the user and the function executed are analyzed, and the user demand is converted into a device calling statement and executed, if it does not belong to the device manipulation category, it is rejected . 7.The Multi-Agent based voice interaction method of claim 6, wherein, In the step S3, the input of the supplemented knowledge or the multi-round text input rejected in the steps S31 to S32 is input into a large question-answering model, the large question-answering model generates a final text output in real time, and the text output is sent to a TTS text-to-speech module to generate a voice reply.

Images