Digital human intermediate layer-based multi-artificial intelligence agent voice interaction method and system

Abstract

The application discloses a multi-artificial intelligence agent voice interaction method and system based on a digital human intermediate layer, belongs to the technical field of artificial intelligence interaction, and aims to solve the problems of the split of voice interaction and artificial intelligence agent execution systems, the opaque agent execution process, the interruption of work flow by permission confirmation, and heavy burden of multi-task concurrent management in the prior art. The method comprises the following steps: user voice collection and transmission, automatic voice recognition and oral embellishment, conversation management agent conversation routing, greeting agent instant greeting, backend artificial intelligence agent task issuing and starting, speech synthesis streaming speech synthesis, digital human video streaming driving and playing, agent execution process real-time broadcasting, user midway query processing, permission request voice confirmation, task completion and final result broadcasting, and waiting for the next round of interaction. The system adopts a three-layer interaction architecture of a user layer, a digital human intermediate layer and an artificial intelligence agent execution layer.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence interaction technology, specifically relating to a multi-AI agent voice interaction method and system based on a digital human intermediary layer. Background Technology

[0002] In recent years, AI agent tools, such as Claude Code (an AI programming agent tool developed by Anthropic, which can be integrated into systems as a backend agent instance), Gemini CLI (a large-model command-line agent tool developed by Google, which can be integrated into systems as a backend agent instance), and Devin (an AI autonomous programming agent tool that only supports a text interface), have developed rapidly. These systems are driven by large language models and can autonomously perform complex, multi-step tasks such as file operations, code writing, and command-line calls, greatly expanding the practical value of artificial intelligence. Multi-agent collaboration platforms have integrated various mainstream agents into a unified platform, allowing users to manage multiple agents to execute tasks concurrently through a graphical interface.

[0003] Meanwhile, voice interaction technology (automatic speech recognition, text-to-speech) and digital human technology (photo-driven virtual face and lip-sync) have become relatively mature and have been widely used in fields such as voice assistants and digital human marketing, respectively.

[0004] However, existing technical solutions only cover a subset: voice assistants like ChatGPT Advanced Voice Mode (which is essentially a voice-based chatbot) are essentially voice-based chatbots, where the AI ​​can only answer questions and cannot perform tasks autonomously; D-ID / HeyGen (a real-time dialogue digital human video generation platform) only connects to a simple question-and-answer language model in its real-time dialogue digital human backend, lacking the ability for AI agents to perform autonomous tasks; and Devin / SWE-agent (an AI autonomous programming agent tool / AI software engineering agent) and other autonomous programming agents rely entirely on text interfaces, lacking any voice interaction capabilities. No existing product integrates voice interaction, digital human-driven operation, AI agent autonomous execution, proactive progress reporting, voice permission confirmation, and multi-session concurrent management into a unified system.

[0005] Therefore, there is an urgent need in this field for an intermediate layer system that can bridge voice interaction and the autonomous execution capabilities of artificial intelligence agents, enabling users to command agents to complete complex tasks through natural voice, while obtaining embodied digital human feedback and a completely touchless interactive experience. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, the present invention aims to provide a multi-AI agent voice interaction method and system based on a digital human middleware layer, which organically integrates voice interaction, digital human-driven operation, and AI agent autonomous execution capabilities, enabling users to complete complex tasks, receive progress, and receive results through voice in vision-free scenarios; thereby solving problems such as the separation of voice interaction and AI agent execution system, opaque agent execution process, workflow interruption due to permission confirmation, and heavy burden of multi-task concurrent management in existing technologies.

[0007] To achieve the above objectives, the present invention may adopt the following specific technical solutions: On the one hand, this invention provides a multi-AI agent voice interaction method based on a digital human intermediary layer, comprising the following steps: Step S1: Collect user voice commands, encode and upload them to the server, and call the automatic speech recognition engine to convert them into text and polish them to make them more conversational. Step S2: Query the active session status table through the session management agent. For new tasks, create a session and bind it to the backend agent instance. For existing tasks, route them to the corresponding session. Use the greeting agent to call the lightweight large language model in parallel to generate conversational greetings and push them into the synthesis queue to eliminate startup delays. Distribute tasks through a standardized communication protocol and start the execution loop of the backend artificial intelligence agent. Step S3: The text is segmented into sentences in real time by a sentence segmenter, and each sentence is synthesized into 16 Pulse Code Modulation audio blocks through the WebSocket (Network Socket Protocol, which supports full-duplex real-time communication and is used for speech synthesis streaming and digital human driving) streaming interface. The digital human service is forwarded in real time, and lip-synced joint image expert group image frames are generated frame by frame and rendered and played on the front-end canvas. Step S4: The progress broadcast agent broadcasts key execution events through the event stream subscription mechanism, and the broadcast agent broadcasts streaming text in segments through the message boundary detection mechanism. When a user queries midway, the status query agent reads the session and agent status to generate a description and broadcasts it. When a permission request is made, the permission confirmation agent rewrites it into a conversational question, collects the voice reply, and then calls the big language model tool to complete intent recognition and permission approval. Step S5: After the backend agent is completed, the broadcast agent will summarize the final result in spoken language and broadcast it through speech synthesis and digital human link. The digital human will then return to standby mode and wait for the user's next round of voice input.

[0008] Furthermore, the digital human intermediate layer includes six types of intelligent micro-agents, namely: Session management agent, used for session routing and multi-task concurrency management; A greeting agent used to generate instant greetings; Progress broadcasting agent, used to actively broadcast execution progress through event stream subscription; Status query agent, used to respond to user status queries midway; Broadcasting agent, used to broadcast segmented, conversational text output from an agent; Permission confirmation agent, used for permission confirmation in voice-driven applications.

[0009] Furthermore, in step S2, the greeting agent and the backend AI agent execute in parallel. The greeting agent generates a greeting and pushes it into the speech synthesis broadcast without waiting for the backend agent to finish starting, eliminating the interactive gap during the agent startup delay. The backend AI agent starts the execution loop in the following order: perceive the task; think about the solution; call the tool; observe the result; think again; repeat until the task is completed. Among them, calling the tool includes reading and writing files, executing commands, and calling application programming interfaces.

[0010] Furthermore, in step S3, the full-stream processing link is adopted in the following order: large language model streaming output; real-time sentence segmentation by sentence segmenter; speech synthesis by WebSocket sentence-by-sentence synthesis; pulse code modulation audio push to digital human service; frame-by-frame joint image expert group format and pulse code modulation distribution; real-time canvas rendering; the entire link can start the next stage of processing without waiting for any stage to complete, and the end-to-end latency is controlled at the second level.

[0011] Furthermore, in step S4, the broadcast agent uses a message identifier boundary detection mechanism to identify text segment boundaries. When a change in the message identifier is detected in the streaming output, it is determined that the previous text segment has been completely output. The text segment is then rewritten into a colloquial expression using a large language model and placed into a serial broadcast queue. Multiple text segments share the same speech synthesis WebSocket session for continuous broadcasting.

[0012] Furthermore, in step S4, the permission confirmation agent uses a large language model tool to identify five types of intents: allow this time, always allow, deny this time, always deny, and ambiguous. Based on the identification results, it automatically calls the corresponding permission confirmation interface of the backend agent to complete the approval, thus realizing a closed loop of permission processing without touching the screen.

[0013] On the other hand, the present invention provides a multi-AI agent voice interaction system based on a digital human intermediary layer, for executing the above-mentioned multi-AI agent voice interaction method based on a digital human intermediary layer, including: The user layer is used to provide voice input acquisition and multimodal output display, including a voice acquisition module, a digital human video rendering area, a speech bubble list, and a conversation switching bar; The digital human middleware layer, as an independent intelligent processing unit, includes an automatic speech recognition module, six types of intelligent micro-agent modules, a speech synthesis module, and a digital human video driving module. It is responsible for speech input parsing, intent routing, content conversationalization conversion, and multimodal output coordination. The AI ​​agent execution layer contains various pluggable AI agent instances that communicate with the digital human middleware layer through standardized protocols and are responsible for the autonomous execution of actual tasks.

[0014] Furthermore, the automatic speech recognition module, speech synthesis module, digital human video driving module, and six types of intelligent micro-agent modules in the digital human middleware layer are all decoupled from the core processing logic of the digital human middleware layer through standardized interfaces, and can be replaced independently at runtime; the digital human image supports dynamic initialization based on user-uploaded face photos, and the speech synthesis timbre supports user-level custom configuration.

[0015] Furthermore, the voice interaction system provides a dual view alongside the agent workbench user interface. The digital human user interface focuses on displaying the refined key information, while the agent workbench user interface retains the complete original output, allowing users to switch as needed.

[0016] Compared with the prior art, the present invention has the following advantages: This invention organically integrates voice interaction, digital human-driven operation, and the autonomous execution capabilities of AI agents through a digital human middleware layer. Users can complete all operations purely through voice, without needing to look at or touch the screen. Six types of intelligent micro-agents collaborate to achieve multi-task concurrent management, proactive progress reporting, touchless permission confirmation, and streaming voice broadcasting, transforming the human-computer interaction mode from "user-initiated polling" to "system-initiated reporting." The full-stream processing link achieves end-to-end, second-level latency multimodal synchronous driving, and can be widely applied in vision-free scenarios such as smart cockpits, education, and healthcare. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a system module diagram of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0019] Example 1 This embodiment provides a multi-AI agent voice interaction method based on a digital human middleware layer, including the following steps: (a) Step S1, Voice Acquisition and Recognition: Acquire user voice commands, encode and upload them to the server, and call the automatic speech recognition engine to convert them into text and polish them into spoken language.

[0020] Users issue voice commands via microphone, and the front end captures the microphone audio through the browser's MediaRecorder application interface, encodes it in webm / opus (open source audio and video container and audio encoding / front-end audio recording encoding) format, and uploads it to the local Express server via the Hypertext Transfer Protocol multipart / form-data (HTTP multipart form data format).

[0021] After receiving the audio, the server calls a configurable automatic speech recognition engine interface to convert the audio into text. Optionally, the recognized text is polished using a large language model to correct terminology errors in the speech recognition. The system has a safety fallback mechanism: if the polished text is more than twice the length of the original text, the polished result is discarded to prevent semantic bias caused by illusions in the large language model.

[0022] (ii) Step S2, Session Routing and Agent Startup: The active session status table is queried through the session management agent. New tasks create sessions and bind backend agent instances, while existing tasks are routed to the corresponding sessions. The greeting agent calls the lightweight large language model in parallel to generate colloquial greetings and pushes them into the synthesis queue to eliminate startup delays. Tasks are issued through standardized communication protocols, and the backend artificial intelligence agent starts the execution loop.

[0023] After receiving a text command from a user, the session management agent queries the current active session status table to determine the intent of the command. If it is a new task command, a new session is created, a unique session identifier is assigned, and the user-selected backend AI agent instance is bound to the session. If it is a follow-up question or operation on an existing task, the command is routed to the corresponding existing session based on the context.

[0024] While the session management agent completes routing, the greeting agent is triggered in parallel. The greeting agent invokes a lightweight large language model to generate a short, conversational greeting based on the user's command, and immediately pushes the greeting text into the speech synthesis and broadcast queue. The greeting agent executes in parallel with the backend AI agent, without waiting for the backend agent to complete startup. The purpose is to eliminate the interactive gap during agent startup delays, allowing the user to immediately perceive that the system has received the command.

[0025] Backend AI agent task issuance and initiation: After user commands are routed by the session management agent, they are issued to the bound backend AI agent via a standardized communication protocol (model context protocol / agent communication protocol, based on standard input / output JSON remote call method). The backend AI agent starts the execution loop in the following order: ① perceive the task; ② consider the solution; ③ call tools (read and write files, execute commands, call application programming interfaces); ④ observe the results; ⑤ further consider; repeat until the task is completed.

[0026] (III) Step S3, Speech Synthesis and Digital Human Driving: The text is segmented into sentences in real time by a sentence segmenter, and 16 Pulse Code Modulation audio blocks are synthesized sentence by sentence through the speech synthesis WebSocket streaming interface. The digital human service is forwarded in real time, and lip-synced joint image expert group image frames are generated frame by frame and rendered and played on the front-end canvas.

[0027] The greeting text generated by the greeting agent enters the speech synthesis streaming process. The text is segmented by a sentence segmenter module according to punctuation (with a 30-character threshold set) to ensure controllable speech synthesis latency. The segmented text is then pushed sentence by sentence to the speech synthesis engine via a WebSocket streaming interface. The speech synthesis engine synthesizes and returns Pulse Code Modulation (PCM) 16 audio blocks simultaneously. The PCM audio blocks are forwarded in real-time to the digital human video generation service via a WebSocket long connection. Upon receiving the PCM audio, the digital human service generates frame-by-frame Joint Image Experts Group (JEAG) format image frames synchronized with the audio lip movements and 40-millisecond PCM 16 audio segments, immediately sending them to the front end. The front end renders the JEAG format images frame-by-frame through the Canvas API, synchronously playing the corresponding audio. The digital human avatar is initialized from a pre-uploaded user face photo, employing a session reuse mechanism (recovery mode). Subsequent rounds of dialogue skip the initialization phase, significantly reducing the first frame latency.

[0028] This application employs a full-stream processing chain in the following order: ① Streaming output of a large language model; ② Real-time sentence segmentation by a sentence segmenter; ③ Speech synthesis via WebSocket sentence-by-sentence synthesis; ④ Pulse code modulation audio push to digital human service; ⑤ Frame-by-frame distribution of joint image expert group format and pulse code modulation; ⑥ Real-time canvas rendering; The entire chain can begin processing the next stage without waiting for any stage to complete, with end-to-end latency controlled in the second range.

[0029] (iv) Step S4, execution broadcast and interaction processing: The progress broadcast agent broadcasts key execution events through the event stream subscription mechanism, and the broadcast agent broadcasts streaming text in segments through the message boundary detection mechanism; when the user queries in the middle, the status query agent reads the session and agent status to generate descriptions and broadcasts them; when permission requests are made, the permission confirmation agent rewrites them into colloquial questions, collects voice responses, and then calls the big language model tool to complete intent recognition and permission approval.

[0030] During the execution loop of the backend agent, the following two micro-agents work in parallel: (a) Progress broadcast agent actively broadcasts progress: It continuously monitors the execution status of the backend agent through an event stream subscription mechanism, transforms technical execution logs into conversational progress descriptions, and actively pushes them into the speech synthesis broadcast queue. (b) Broadcast agent streams output: It subscribes to the streaming text output of the backend agent, uses a message identifier boundary detection mechanism to identify the end of text paragraphs, rewrites each text paragraph into a conversational expression through a large language model, and puts it into a serial broadcast queue. Multiple text paragraphs share the same speech synthesis WebSocket session for continuous broadcasting.

[0031] If the user sends a new voice message during the agent's execution, the user's voice is collected and automatically recognized through steps S1 to S2; the status query agent is triggered, reads the latest message record of the current session and the agent's execution status from the database, and calls the large language model to generate a description of the current status; the status description is then broadcast to the user through speech synthesis and digital human link in step S3.

[0032] If the backend agent issues a permission request during execution, the permission confirmation agent is triggered, rewriting the permission request into a conversational question and pushing it to the speech synthesis broadcast. The system enters voice monitoring mode, collects the user's voice response, and converts it into text through automatic speech recognition. The permission confirmation agent inputs the user's voice text and permission context into the large language model, and uses the large language model tool (non-streaming, low-latency mode) to perform intent recognition, outputting one of five intent types: allow_once, always allow, reject_once, always reject, or unclear (unclear, requires further inquiry). Based on the recognition result, the system automatically calls the corresponding permission confirmation interface of the backend agent to complete the approval, realizing a closed loop of permission processing without touch screen.

[0033] (V) Step S5, Task Completion and Standby: After the backend agent completes the task, the broadcast agent will summarize the final result in spoken language and broadcast it through speech synthesis and digital human link. The digital human will then return to standby mode and wait for the user's next round of voice input.

[0034] After the backend agent completes all execution steps in the agent execution loop, it outputs the final result text. The broadcast agent detects the end signal of the agent output, calls the large language model to summarize and rewrite the final result text in a conversational style, pushes it into the speech synthesis broadcast queue, and plays it to the user after speech synthesis and digital human-driven processing. The user receives the message and waits for the next round of interaction: after the broadcast is completed, the digital human returns to standby mode (silent but maintaining session reuse), and the system waits for the user's next round of voice input. The user can continue to follow up on the current session, switch to other sessions, or initiate a new task; the system then begins a new round of processing from step S1.

[0035] This invention enables AI agents to perform complex tasks via voice commands, proactive progress reporting, touchless permission management, and multi-session concurrent scheduling. It can be widely applied in touchless scenarios such as smart cockpits, education, and healthcare.

[0036] Example 2 like Figure 1 As shown, this embodiment provides a multi-AI agent voice interaction method based on a digital human middleware layer, including the following steps: (I) Step S1: Voice acquisition and recognition.

[0037] User voice capture and transmission. The user speaks into the microphone: "Help me refactor the code in the src / utils directory, and change all callback functions to async / await." The front-end media recorder captures the voice, encodes it in webm / opus format, and uploads it to the local Express server via HTTP.

[0038] Automatic speech recognition and conversational polishing. The server calls an automatic speech recognition engine (such as Alibaba Cloud Speech Recognition) to convert audio into text: "Help me refactor the code in the src / utils directory and change all callback functions to async / await." After polishing by the large language model, it becomes: "Help me refactor the code in the src / utils directory and change all callback functions to async / await."

[0039] (ii) Step S2, Session Routing and Proxy Startup.

[0040] Session management agent handles session routing. When a new task is identified by the session management agent, a session (identifier: session_001) is created and bound to the Claude Code agent instance.

[0041] The call agent issues an immediate call. Parallel execution: The call agent generates the call "Received, I'll help you with the reconstruction of the utils directory," and immediately pushes it into the speech synthesis queue; at the same time, the task instruction is sent to the Claude Code agent through the model context protocol.

[0042] Backend AI agent task issuance and startup. Claude Code starts the agent execution loop and begins executing code refactoring tasks.

[0043] (III) Step S3, speech synthesis and digital human driving.

[0044] Speech synthesis is streamed and played back as video-driven by the digital human. Greetings are streamed into pulse-code modulated audio and forwarded to the digital human service. Frame-by-frame, Joint Image Experts Group (JEPAG) format image frames and 40-millisecond pulse-code modulated audio clips are generated, and the front-end canvas renders and plays them frame-by-frame. The user sees the digital human say, "Received, I'll check the restructuring of the utils directory for you."

[0045] (iv) Step S4: Execute broadcast and interactive processing.

[0046] The agent execution process is broadcast in real time. During the Claude Code execution agent's execution loop: First, the agent scans the src / utils directory, broadcasting "Scanning the directory, found 8 TypeScript files"; Second, the agent issues a permission request, and upon permission confirmation, broadcasts "The agent wants to read the fileHelper, do you agree?", and the user says "Allow all". The large language model tool recognizes the intent as always allowing and automatically approves all subsequent read permissions; Third, the agent reconstructs each file one by one, broadcasting "The reconstruction of fileHelper has been completed, and the 3 callbacks have been changed to await" after each file is completed; Fourth, if the user asks "How is the progress now?", the status query agent broadcasts "The reconstruction of 5 files has been completed, and 3 more are being processed".

[0047] (v) Step S5, Task completion and standby.

[0048] The proxy task is completed and the final result is announced. After the proxy completes all the refactoring, it broadcasts the announcement: "Refactoring is complete. A total of 8 files were processed, 15 callback functions were changed to async / await, the code was reduced from 320 lines to 240 lines, and all tests have passed."

[0049] The user has received the command and is waiting for the next round of interaction. The digital human returns to standby mode, and the user can initiate the next round of voice commands at any time.

[0050] Example 3 This embodiment uses multi-session concurrent management as an example for illustration.

[0051] The user says, "Help me write a unit test using Claude." The system creates session A, binds it to the Claude Code agent, and instructs the agent to announce, "Okay, I'll write the unit test."

[0052] During session A, the user says, "Also, could you use Gemini to look up the documentation for this API?" The session management agent recognizes this as a new task, creates session B, binds it to the Gemini CLI agent, and instructs the agent to announce, "Also, I'll look up the API documentation for you."

[0053] The agents for the two sessions execute in parallel, with their respective progress broadcast agents and broadcast agents working independently. In session B, Gemini completes first, and its broadcast agent announces the API documentation query results. In session A, Claude Code continues execution, with its progress broadcast agent reporting the unit test progress midway through.

[0054] When a user asks, "Is the test finished?", the status query agent automatically routes to session A and announces the current progress. Once session A is complete, the broadcast agent announces the final result. The user does not need to switch between windows throughout the entire process.

[0055] Example 4 like Figure 2 As shown, this embodiment provides a multi-AI agent voice interaction system based on a digital human middleware layer, executing the interaction methods of embodiments 1, 2, or 3, and mainly includes the following modules: User layer: Provides microphone voice capture (browser media recorder application interface) and multimodal output (digital human video rendering area canvas rendering, voice broadcast, dialog bubble list, and conversation switching bar); includes microphone voice capture module, digital human video rendering area, dialog bubble list, and conversation switching bar.

[0056] The digital human middleware layer, as an independent intelligent processing unit, comprises four core functional modules: ① An automatic speech recognition module responsible for audio-to-text conversion and optional conversational polishing; ② Six types of intelligent micro-agent modules are core decision-making components, including a conversation management agent (conversation management and routing), a greeting agent (instant greeting), a progress broadcast agent (event-driven progress broadcast), a status query agent (status query response), a broadcast agent (streaming segmented broadcast), and a permission confirmation agent (voice permission confirmation); ③ A speech synthesis module segments sentences using a sentence segmenter and streams synthesizes pulse-code modulated audio via WebSocket; ④ A digital human video driving module forwards pulse-code modulated audio in real time via WebSocket and generates frame-by-frame frames and audio segments in the Joint Image Experts Group format synchronized with the audio lip movements. All four core functional modules are decoupled from the core processing logic of the digital human middleware layer through standardized interfaces, supporting independent replacement at runtime. The digital human avatar supports dynamic initialization using user-uploaded facial photos, and the speech synthesis timbre supports user-level custom configuration.

[0057] Artificial Intelligence Agent Execution Layer: Includes various pluggable artificial intelligence agent instances such as Claude Code, Gemini CLI, Codex, and Qwen Code. It communicates with the digital human middleware layer through the Model Context Protocol / Agent Communication Protocol standard protocol and is responsible for the autonomous execution of actual tasks.

[0058] The voice interaction system provides a dual view alongside the agent workbench user interface: the digital human user interface focuses on displaying the refined key information, while the agent workbench user interface retains the complete original output, which the user can switch between as needed.

[0059] The backend AI agent in the solution is not limited to the agent types listed above; new agent access only requires the implementation of a protocol adapter. The automatic speech recognition, speech synthesis, and digital human model all support independent replacement at runtime. The five types of intent recognition results can be expanded according to actual needs. All these modifications fall within the scope of protection of this invention.

[0060] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-AI agent voice interaction method based on a digital human middleware layer, characterized in that: Includes the following steps: Step S1: Collect user voice commands, encode and upload them to the server, and call the automatic speech recognition engine to convert them into text and polish them to make them more conversational. Step S2: Query the active session status table through the session management agent. For new tasks, create a session and bind it to the backend agent instance. For existing tasks, route them to the corresponding session. Use the greeting agent to call the lightweight large language model in parallel to generate conversational greetings and push them into the synthesis queue to eliminate startup delays. Distribute tasks through a standardized communication protocol and start the execution loop of the backend artificial intelligence agent. Step S3: The text is segmented into sentences in real time by a sentence segmenter, and 16 Pulse Code Modulation (PCM) audio blocks are synthesized sentence by sentence through the speech synthesis WebSocket streaming interface. The audio is forwarded to the digital human service in real time, and lip-synced joint image expert group image frames are generated frame by frame. The front-end canvas renders and plays the images. Step S4: The progress broadcast agent broadcasts key execution events through the event stream subscription mechanism, and the broadcast agent broadcasts streaming text in segments through the message boundary detection mechanism. When a user queries midway, the status query agent reads the session and agent status to generate a description and broadcasts it. When a permission request is made, the permission confirmation agent rewrites it into a conversational question, collects the voice reply, and then calls the big language model tool to complete intent recognition and permission approval. Step S5: After the backend agent is completed, the broadcast agent will summarize the final result in spoken language and broadcast it through speech synthesis and digital human link. The digital human will then return to standby mode and wait for the user's next round of voice input.

2. The multi-AI agent voice interaction method based on a digital human intermediary layer according to claim 1, characterized in that, The digital human middleware layer comprises six types of intelligent micro-agents, namely: Session management agent, used for session routing and multi-task concurrency management; A greeting agent used to generate instant greetings; Progress broadcasting agent, used to actively broadcast execution progress through event stream subscription; Status query agent, used to respond to user status queries midway; Broadcasting agent, used to broadcast segmented, conversational text output from an agent; Permission confirmation agent, used for permission confirmation in voice-driven applications.

3. The multi-AI agent voice interaction method based on a digital human intermediary layer according to claim 2, characterized in that, In step S2, the greeting agent and the backend AI agent execute in parallel. The greeting agent generates a greeting and pushes it into the speech synthesis broadcast without waiting for the backend agent to finish starting, eliminating the interactive gap during the agent startup delay. The backend AI agent starts the execution loop in the following order: perceive the task; think about the solution; call the tool; observe the result; think again; repeat until the task is completed. Among them, calling the tool includes reading and writing files, executing commands, and calling application programming interfaces.

4. The multi-AI agent voice interaction method based on a digital human intermediary layer according to claim 2, characterized in that, In step S3, the full-stream processing link is adopted in the following order: large language model streaming output; sentence segmentation in real time; speech synthesis by WebSocket sentence-by-sentence synthesis; pulse code modulation audio push to digital human service; frame-by-frame joint image expert group format and pulse code modulation distribution; canvas real-time rendering; the entire link can start the next stage of processing without waiting for any stage to complete, and the end-to-end latency is controlled at the second level.

5. The multi-AI agent voice interaction method based on a digital human intermediary layer according to claim 2, characterized in that, In step S4, the broadcast agent uses a message identifier boundary detection mechanism to identify text segment boundaries. When a change in the message identifier is detected in the streaming output, it is determined that the previous text segment has been completely output. The text segment is then rewritten into a colloquial expression using a large language model and placed into a serial broadcast queue. Multiple text segments share the same speech synthesis WebSocket session for continuous broadcasting.

6. The multi-AI agent voice interaction method based on a digital human intermediary layer according to claim 2, characterized in that, In step S4, the permission confirmation agent uses a large language model tool to identify five types of intents: allow this time, always allow, deny this time, always deny, and ambiguous. Based on the identification results, it automatically calls the corresponding permission confirmation interface of the backend agent to complete the approval, realizing a closed loop of permission processing without touching the screen.

7. A multi-AI agent voice interaction system based on a digital human intermediary layer, used to execute the multi-AI agent voice interaction method based on a digital human intermediary layer as described in any one of claims 1-6, characterized in that, include: The user layer is used to provide voice input acquisition and multimodal output display, including a voice acquisition module, a digital human video rendering area, a speech bubble list, and a conversation switching bar; The digital human middleware layer, as an independent intelligent processing unit, includes an automatic speech recognition module, six types of intelligent micro-agent modules, a speech synthesis module, and a digital human video driving module. It is responsible for speech input parsing, intent routing, content conversationalization conversion, and multimodal output coordination. The AI ​​agent execution layer contains various pluggable AI agent instances that communicate with the digital human middleware layer through standardized protocols and are responsible for the autonomous execution of actual tasks.

8. The multi-AI agent voice interaction system based on a digital human middleware layer according to claim 7, characterized in that, The automatic speech recognition module, speech synthesis module, digital human video driver module, and six types of intelligent micro-agent modules in the digital human middleware layer are all decoupled from the core processing logic of the digital human middleware layer through standardized interfaces, and can be replaced independently at runtime; the digital human image can be dynamically initialized by uploading a face photo by the user, and the speech synthesis timbre can be customized at the user level.

9. The multi-AI agent voice interaction system based on a digital human middleware layer according to claim 7, characterized in that, The voice interaction system provides a dual view alongside the agent workbench user interface. The digital human user interface focuses on displaying the refined key information, while the agent workbench user interface retains the complete original output, allowing users to switch as needed.

Images