Voice control browser operation method and system based on double-agent real-time communication

Abstract

The application provides a voice control browser operation method and system based on double-agent real-time communication, relates to the technical field of human-computer interaction and computer software, and aims at the defects of the prior art, such as the separation of voice and browser operation, the difficulty in cooperation between agents, the lack of real-time interruption mechanism and complete conversation persistence capability.The method is cooperatively executed by a voice agent and a browser agent which are connected in real time through a communication bridge, and comprises a voice input and recognition step, a voice input and recognition step, a browser operation and feedback step and a real-time interruption step.The application solves the problems in the prior art, and realizes the full process, interactivity and continuity of intelligent automation from natural language instructions to complex browser operation.

Description

Technical Field

[0001] This invention belongs to the field of human-computer interaction and computer software technology, and particularly relates to a voice-controlled browser operation method and system based on real-time communication between two intelligent agents. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence technology, human-computer interaction is undergoing profound changes. Among these changes, voice interaction, due to its natural and convenient characteristics, has become an important direction for improving user experience. Meanwhile, as the primary gateway to the internet, browsers are increasingly sophisticated in their automation technologies. Therefore, achieving end-to-end automation from user natural language commands to complex browser operations has become a key technology urgently needing breakthroughs in this field.

[0004] Currently, the technology is primarily based on an AI-driven browser automation framework. This technology allows the browser to perform tasks (such as filling out forms, fetching, scrolling, and logging in) through natural language and proxy commands. Operations are achieved through a visual perception and region selection mechanism. Playwright captures a screenshot of the current webpage and generates a visual semantic map based on page structure information. Then, the LLM (Layered Module) "draws" an action box on the screenshot based on the user's intent (e.g., clicking the login button), marking the target area. The browser automation layer executes specific actions (clicking, scrolling, inputting) based on the selected coordinates and provides a screenshot of the result for the model to make a second judgment. The model further analyzes the page after execution, and if the task is not completed, it automatically corrects the operation path, achieving a closed-loop interaction. It is currently in a rapid update phase, providing examples of scenarios such as form filling, grocery shopping lists, and PC accessory selection.

[0005] Furthermore, mainstream intelligent voice assistants (such as Siri and Alexa) typically employ a technical architecture comprising four core modules: an ASR (Action Speech, such as Whisper and Wenet) module, responsible for converting user speech into text; a NLU (Nature Language Understanding) module, responsible for semantic parsing of text, extracting user intent, or directly inferring meaning from a large language model; a dialogue management module, responsible for managing multi-turn dialogue states and context; and a TTS (Text-to-Speech, such as VITS and Edge-TTS) module, responsible for converting system responses into speech. This technology has evolved from early keyword recognition and rule matching to deep learning-based semantic understanding, and currently, its integration with large language models (LLMs) has enabled it to excel in knowledge-based question answering, emotional dialogue, and simple task execution.

[0006] Despite the significant progress made in the aforementioned technologies, certain shortcomings still exist:

[0007] (1) Existing intelligent voice assistant systems are mainly limited to simple information question and answer and basic device control, lacking the ability to convert complex and continuous natural language commands into a series of browser automated operations in real time and accurately.

[0008] (2) Existing browser automation tools (such as Selenium and Playwright) are powerful, but they rely heavily on pre-written scripts and lack the ability to conduct real-time, intelligent multi-turn voice interaction with users.

[0009] (3) Most existing multi-agent systems operate independently and lack real-time coordination mechanisms, which makes it difficult for the two agents to respond quickly to each other's state changes and achieve coordination and immediate intervention in task execution.

[0010] (4) When performing long-term or complex browser tasks, it cannot respond to voice interruption requests, and the browser session cannot maintain a continuous state between different running instances; at the same time, the browser session is usually based on simple cookies or local storage, which cannot fully save the browser state, resulting in poor session continuity and inability to maintain a continuous state between different running instances. Summary of the Invention

[0011] To overcome the shortcomings of the prior art, this invention provides a voice-controlled browser operation method and system based on real-time communication between two intelligent agents. Through efficient collaboration between the voice intelligent agent and the browser intelligent agent, combined with real-time interruption and state persistence mechanisms, it realizes intelligent automation of the entire process from natural language commands to complex browser operations in an interactive and continuous manner.

[0012] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0013] The first aspect of the present invention provides a voice-controlled browser operation method based on real-time communication between two agents, which is collaboratively executed by a voice agent and a browser agent connected in real time via a communication bridge, including:

[0014] In response to the user's voice input, the front-end interface acquires audio data and sends it to the voice agent through the first real-time communication link;

[0015] The voice-activated agent recognizes audio data to obtain the user's command text;

[0016] The voice agent parses the user's command text using a large language model, generates structured tasks, and distributes the structured tasks to the browser agent through a communication bridge.

[0017] The browser agent receives structured tasks and drives the browser to perform corresponding automated operations based on the structured tasks. During or after the browser operation, the browser agent feeds back the execution status or result information to the voice agent through the communication bridge, and the voice agent then broadcasts the information to the user.

[0018] If a new user voice input is detected during the process of a voice agent performing voice broadcast or a browser agent performing a structured task, an interruption mechanism is triggered. The interruption mechanism includes at least: immediately stopping the current voice broadcast and sending an interruption signal to the browser agent through the communication bridge to interrupt the currently executing structured task.

[0019] As one implementation method, session persistence is also included, the specific process of which is as follows:

[0020] During the execution of structured tasks by the browser agent, the state changes of the browser session are continuously monitored;

[0021] Serialize and persist the state of the browser session;

[0022] When a session needs to be resumed after it has been interrupted, the saved serialized state data is loaded, and the browser session is reconstructed based on the serialized state data.

[0023] The voice agent is notified to resume the conversation via a communication bridge.

[0024] As one implementation method, the specific process of obtaining user instruction text includes:

[0025] The front-end interface adopts a modular architecture based on Hypertext Markup Language and is designed with a state machine to trigger voice input operations and call the device's audio interface to collect audio data.

[0026] The acquired audio data is preprocessed, including at least format standardization and encapsulation.

[0027] The preprocessed audio data is sent to the voice agent in a streaming and chunked manner through the first real-time communication link;

[0028] The voice agent receives preprocessed audio data via a WebSocket server and stores it in a temporary audio buffer until the end of voice input is detected.

[0029] When the end of the voice input is detected, the audio data is extracted from the temporary audio buffer and recognized by the speech recognition model, and the audio data is converted into user command text.

[0030] As one implementation method, the voice agent parses the user's command text using a large language model, generates structured tasks, and distributes these structured tasks to the browser agent via a communication bridge. The specific process includes:

[0031] The voice-activated intelligent agent inputs user command text into a large language model and uses an intent parsing algorithm for semantic understanding and intent parsing;

[0032] Based on the intent parsing results, user commands are converted into structured tasks. The structured tasks adopt a predefined format and include key fields such as operation type, operation target, operation parameters and execution conditions.

[0033] The voice agent distributes structured tasks to the browser agent through a communication bridge, and the task distribution is handled using a message queue pattern.

[0034] As one implementation method, the browser agent receives structured tasks and drives the browser to perform corresponding automated operations based on the structured tasks, including user query operations. The specific process is as follows:

[0035] When the browser agent encounters a situation requiring user confirmation while performing a structured task, it pauses execution and generates a confirmation request.

[0036] The confirmation request is sent to the voice agent via a communication bridge connection;

[0037] The voice agent converts the received confirmation request into voice and broadcasts it to the user, then enters a waiting state for a response.

[0038] As one implementation method, the browser agent receives structured tasks and drives the browser to perform corresponding automated operations based on the structured tasks. This also includes user responses and result feedback. The specific process is as follows:

[0039] When a user answers a confirmation request, the user's voice response is collected and recognized.

[0040] Semantic analysis is performed on the recognized user voice response to obtain the user voice response analysis result;

[0041] The browser agent receives the parsing results of the user's voice response through the communication bridge and continues to execute the suspended structured task;

[0042] After the browser agent completes the operation corresponding to the structured task, it generates task execution result information and sends it to the voice agent through the communication bridge.

[0043] The voice agent converts task execution results into voice broadcast content and drives the front-end interface to play the audio to the user.

[0044] As one implementation method, if a new user voice input is detected during the process of a voice agent performing voice broadcasting or a browser agent performing structured tasks, an interruption mechanism is triggered. The specific process includes:

[0045] The front-end interface monitors the audio input signal in real time. When it detects the start of voice activity that meets the preset conditions, it immediately generates an interruption trigger event and sends it to the voice agent.

[0046] After receiving the interruption trigger event through the first real-time communication link, the voice agent sets the global interruption flag to an active state and simultaneously sends an interruption signal to the browser agent through the communication bridge.

[0047] Upon receiving an interruption signal, the browser agent suspends the currently executing task and clears the temporary state associated with the structured task.

[0048] After the browser agent interrupts its task, it returns an interruption confirmation message to the voice agent through the communication bridge.

[0049] The voice agent updates its status based on the received interruption confirmation information and stops the ongoing voice broadcast.

[0050] As one implementation method, the communication bridge uses an asynchronous message queue mechanism to realize bidirectional asynchronous communication between the browser agent and the voice agent. The message queue is used to manage the transmission of task requests, status feedback, queries and answers, and interruption signals, and assigns a unique identifier to each message to support task status tracking, execution progress monitoring, error handling, and retry mechanisms.

[0051] A second aspect of the present invention provides a voice-controlled browser operating system based on real-time communication between two agents, comprising:

[0052] The voice input module is used to respond to the user's voice input operation. The front-end interface obtains audio data and sends it to the voice agent through the first real-time communication link.

[0053] The speech recognition module is used by the voice intelligence agent to recognize audio data and obtain the user's command text;

[0054] The task parsing and distribution module is used by the voice agent to parse the user's command text using a large language model, generate structured tasks, and distribute the structured tasks to the browser agent through a communication bridge.

[0055] The browser operation and feedback module is used by the browser agent to receive structured tasks and drive the browser to perform corresponding automated operations based on the structured tasks. During or after the browser operation, the browser agent feeds back the execution status or result information to the voice agent through the communication bridge, and the voice agent then broadcasts the information to the user.

[0056] The real-time interruption module is used to trigger an interruption mechanism if a new user voice input is detected during the process of a voice agent performing voice broadcast or a browser agent performing a structured task. The interruption mechanism includes at least: immediately stopping the current voice broadcast and sending an interruption signal to the browser agent through the communication bridge to interrupt the currently executing structured task.

[0057] As one implementation, it also includes a session persistence management module, which continuously monitors the state changes of the browser session during the execution of structured tasks by the browser agent; serializes and persistently saves the state of the browser session; when the session needs to be resumed after interruption, it loads the saved serialized state data and reconstructs the browser session based on the serialized state data; and notifies the voice agent to resume the dialogue through the communication bridge.

[0058] The above one or more technical solutions have the following beneficial effects:

[0059] This embodiment constructs a collaborative architecture of voice agent, communication bridge, and browser agent, and integrates a large language model for end-to-end task parsing and planning. This enables the real-time and accurate decomposition and conversion of complex, continuous natural language commands (such as logging into my email, finding Zhang San's email from last week, and replying that it has been received) into a series of executable automated browser operation sequences. This expands the capabilities of the voice interaction system, upgrading it from simple question-and-answer to a productivity tool capable of executing complex workflows. It solves the problem that existing intelligent voice assistants cannot drive complex browser operations, achieving end-to-end automated operation from natural language to complex browser actions.

[0060] In this embodiment, the browser agent is deeply coupled with a voice agent capable of multi-turn dialogue management. When the browser agent encounters ambiguous or unclear situations requiring confirmation during execution, it can initiate a query in real time through a communication bridge. The voice agent then interacts with the user in natural language to clarify the intent and dynamically adjusts the execution path based on the user's response. This enables intelligent real-time negotiation and dynamic adjustment of the automation tool, evolving it from a static script execution tool into an interactive intelligent agent, thus solving the problem of existing browser automation tools lacking intelligent interaction capabilities.

[0061] In this embodiment, a dedicated real-time communication bridge based on a dual-channel asynchronous message queue and WebSocket was designed to construct the neural hub between the two agents. This bridge is not only responsible for low-latency synchronization of tasks and states, but also can instantly broadcast state changes of one party (such as an interruption signal when a user starts speaking) to the other party. This achieves millisecond-level state awareness and response between the two agents, ensuring a high degree of coordination and consistency between task execution, user intervention, and system feedback.

[0062] In this embodiment, a global real-time interruption mechanism and browser session deep persistence technology are employed. On one hand, the system can respond to user interruptions and clear task states at any time through front-end voice activity monitoring and global interruption flags. On the other hand, the browser's intelligent agent can serialize, store, and accurately restore the complete state of the session (including DOM structure, JavaScript context, network cache, etc.), ensuring both the autonomy and fluency of the interaction, and enabling seamless continuation of complex tasks across time and instances, greatly improving the reliability and user experience of long-term, multi-step operations.

[0063] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0064] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0065] Figure 1 This is a flowchart of a voice-controlled browser operation method based on real-time communication between two intelligent agents, according to Embodiment 1 of the present invention.

[0066] Figure 2 This is a schematic diagram of the framework of a voice-controlled browser operation method based on real-time communication between two intelligent agents, according to Embodiment 1 of the present invention. Detailed Implementation

[0067] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0068] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations of the present invention.

[0069] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0070] Example 1

[0071] This embodiment discloses a voice-controlled browser operation method based on real-time communication between two intelligent agents.

[0072] To more clearly illustrate this embodiment, the implementation process of voice-controlled browser operation based on real-time communication between two intelligent agents can be specifically described as follows:

[0073] A voice-controlled browser operation method based on real-time dual-agent communication is executed collaboratively by a voice agent and a browser agent connected in real time via a communication bridge, including:

[0074] S1. Responding to the user's voice input, the front-end interface acquires audio data and sends it to the voice agent through the first real-time communication link;

[0075] S2, the voice agent recognizes audio data and obtains the user's command text;

[0076] S3, the voice agent parses the user's command text using a large language model, generates structured tasks, and distributes the structured tasks to the browser agent through a communication bridge;

[0077] S4. The browser agent receives structured tasks and drives the browser to perform corresponding automated operations based on the structured tasks. During or after the browser operation, the browser agent feeds back the execution status or result information to the voice agent through the communication bridge, and the voice agent then broadcasts the information to the user.

[0078] S5. If a new user voice input is detected during the process of the voice agent performing voice broadcast or the browser agent performing a structured task, an interruption mechanism is triggered. The interruption mechanism includes at least: immediately stopping the current voice broadcast and sending an interruption signal to the browser agent through the communication bridge to interrupt the currently executing structured task.

[0079] like Figure 1 , Figure 2 As shown, in step S1, in response to the user's voice input operation, the front-end interface acquires audio data and sends it to the voice agent through the first real-time communication link.

[0080] In this embodiment, the specific process includes:

[0081] (1) The front-end interface adopts a modular architecture based on hypertext markup language and is designed with a state machine to trigger voice input operation and call the device audio interface to collect audio data.

[0082] The front-end interface adopts a modular architecture based on the HTML5 standard, including a network socket client and an HTML5 audio / status display module; a state machine is used to design the front-end interface, and audio data is acquired by calling the device audio interface through the HTML5 MediaRecorder API.

[0083] Specifically, the front-end recording button design uses the HTML5 MediaRecorder API to achieve real-time audio capture, obtaining microphone permissions through getUserMedia(). The recording button employs a state machine design, including three states: standby, recording, and processing, with each state corresponding to different UI feedback and functional behaviors.

[0084] (2) Preprocess the collected audio data, wherein the preprocessing includes at least format standardization and encapsulation; and send the preprocessed audio data to the voice agent in a streaming and block manner through the first real-time communication link.

[0085] Specifically, firstly, the audio data undergoes format processing. The recorded audio is in WebM format with a sampling rate set to 16kHz (the optimal input format for the Whisper model), and is recorded in mono to reduce data transmission. The audio data is encapsulated using Blob objects, supporting both real-time streaming and chunked delivery.

[0086] Transmission is carried out through the first real-time communication link, which adopts a WebSocket client architecture to establish a persistent WebSocket connection. A heartbeat mechanism is used to maintain connection stability. That is, the communication bridge / WebSocket connection uses a periodic heartbeat detection mechanism to monitor connection activity and trigger automatic reconnection when the connection is abnormally disconnected, thereby maintaining the stability and high availability of the communication link. The client implements an automatic reconnection mechanism, and performs exponential backoff reconnection when the connection is broken. Audio data is transmitted through binary frames to avoid the performance loss caused by Base64 encoding.

[0087] The front-end interface serves as the entry point for voice interaction and adopts a modular design, primarily comprising two core components: a network socket client and an HTML5 audio / status display. The network socket client, implemented based on the HTML5 WebSocket API, is responsible for establishing a persistent bidirectional communication connection with the voice agent's WebSocket server. It not only handles real-time uplink transmission of audio data but also receives status notifications, text replies, and synthesized speech data from the voice agent, enabling full-duplex real-time interaction. The HTML5 audio / status display module integrates audio acquisition and visual feedback functions: the audio acquisition part, based on HTML5's MediaRecorder and Web Audio API, implements microphone permission management, real-time audio stream capture, and format encapsulation; the status display part uses dynamic UI components to present the system's current status (standby, recording, processing, voice playback) in real time, and provides users with auxiliary information such as recording level visualization and connection status indicators, enhancing the intuitiveness and controllability of the interaction.

[0088] Furthermore, data transmission is optimized by implementing a chunked transmission mechanism for audio data. Each audio chunk contains a timestamp and sequence number, ensuring the server can correctly reassemble the audio stream. Compression algorithms are employed to reduce network transmission load, while also implementing error detection and retransmission mechanisms.

[0089] After steps (1) and (2), voice input is achieved, so that the transmitted audio data contains complete metadata information, providing the necessary format and quality assurance for subsequent speech recognition (ASR) processing.

[0090] like Figure 1 , Figure 2 As shown, in step S2, the voice agent recognizes the audio data to obtain the user command text.

[0091] (1) The voice agent receives the preprocessed audio data through the WebSocket server and stores it in a temporary audio buffer until the voice input ends.

[0092] Specifically, the voice agent audio receiving mechanism involves the voice agent receiving audio streams through a WebSocket server, implementing an asynchronous audio processing queue.

[0093] The received audio data undergoes format validation and integrity checks, and is then stored in a temporary buffer for further processing.

[0094] The process involves format verification. Specifically, the front-end uses MediaRecorder API parameters to force the recording format (mono, 16kHz sampling rate, WebM container + Opus encoding). It also checks for the existence of audio data blocks and records their size. On the back-end, FFmpeg performs WebM to WAV format conversion and standardization, and then verifies the conversion success using its return code.

[0095] Integrity check: Record and check the data size at each stage of transmission (not less than 1000 bytes).

[0096] (2) When the voice input ends, the audio data is extracted from the temporary audio buffer and recognized by the speech recognition model, and the audio data is converted into user instruction text.

[0097] Specifically, 1) Before recognition, the extracted audio data is denoised and normalized. FFmpeg is used for format conversion and audio enhancement. A silence detection algorithm is adopted to automatically identify valid speech segments, filter out meaningless silence parts, and normalize the audio to ensure that the audio input to the Whisper ASR speech recognition model has a consistent volume level.

[0098] The silence detection algorithm first calculates the short-time energy and zero-crossing rate to obtain an adaptive threshold, then makes a speech decision, and finally uses morphological processing to smooth the detection results and remove noise interference.

[0099] 2) The Whisper ASR speech recognition model is used to recognize the denoised and normalized data to obtain the user command text. The Whisper ASR model integrates the OpenAI Whisper model and supports multiple model size options (tiny, base, small, medium, large). The model is dynamically selected based on real-time and accuracy requirements, achieving a balance between accuracy and response speed. Whisper ASR model loading uses a lazy loading mechanism, initializing it only when needed and reusing the already loaded model instance in subsequent calls.

[0100] like Figure 1 , Figure 2 As shown, in step S3, the voice agent parses the user command text using a large language model, generates a structured task, and distributes the structured task to the browser agent through a communication bridge.

[0101] In this embodiment, task parsing and distribution are implemented, and the specific process includes:

[0102] (1) The voice agent inputs the user's command text into the large language model and uses the intent parsing algorithm to perform voice understanding and intent parsing;

[0103] Specifically, 1) the user command text is optimized before intent parsing.

[0104] Post-processing is performed on the raw text output by the Whisper ASR model, including punctuation addition, grammar correction, and terminology recognition, to achieve context-dependent text optimization and adjust the accuracy of the recognition results based on the content of historical dialogues.

[0105] The optimized user command text is encapsulated into structured data, including the original text, confidence level, timestamp, and other information, and then passed to the large language model for intent parsing and task generation.

[0106] 2) Based on the large language model, an intent parsing algorithm is used to parse the optimized user command text.

[0107] The large language model adopts the DeepSeek LLM integration architecture, using the DeepSeekChat model as the core inference engine. Integration is achieved through API calls, enabling request pooling and load balancing, and supporting concurrent request processing. Model parameter optimization is implemented, including dynamic adjustment of parameters such as temperature and top_p, ensuring a balance between output stability and creativity.

[0108] An intent parsing algorithm is used to parse the optimized user command text.

[0109] First, user commands are classified into coarse-grained intents using a predefined classification rule. Second, commands classified as browser operations are input into a large language model for fine-grained parsing and structured task generation under zero-shot conditions.

[0110] (2) Based on the intent parsing results, the user instructions are converted into structured tasks. The structured tasks adopt a predefined format and include key fields such as operation type, operation target, operation parameters and execution conditions.

[0111] Based on the intent parsing results, the natural language instructions are converted into a standardized task description format, including fields such as operation type, target URL, element selector, operation parameters, and execution conditions. The task description uses JSON format for easier subsequent parsing and execution.

[0112] (3) The voice agent distributes structured tasks to the browser agent through a communication bridge, wherein the task distribution is handled by a message queue pattern.

[0113] Specifically, the communication bridge design, agent_bridge.py implements a bidirectional communication mechanism, uses a message queue pattern to handle task distribution, supports task status tracking, execution progress monitoring, error handling and retry mechanisms, and the bridge maintains session state to ensure the context continuity of multi-turn dialogues.

[0114] The generated structured tasks are sent to the browser agent via a communication bridge. At the same time, a task execution monitoring mechanism is established to prepare for subsequent user inquiries and result feedback.

[0115] like Figure 1 As shown, in step S4, the browser agent receives a structured task and drives the browser to perform corresponding automated operations based on the structured task. During or after the browser operation, the browser agent feeds back the execution status or result information to the voice agent through the communication bridge, and the voice agent then broadcasts the information to the user.

[0116] The browser agent receives structured tasks and drives the browser to perform corresponding automated operations based on these tasks, including user inquiries, user responses, and result feedback.

[0117] S4-1. Drive the browser to perform corresponding automated operations based on structured tasks, including user prompts.

[0118] In this embodiment, the specific process is as follows:

[0119] (1) When the browser agent encounters a situation that requires user confirmation during the execution of a structured task, it pauses execution and generates a confirmation request;

[0120] Specifically, the browser agent performs monitoring operations. During task execution, the browser agent identifies intelligent decision points. When encountering situations requiring user confirmation (such as multiple matching elements, sensitive operations, ambiguous instructions, etc.), it automatically pauses execution and generates a confirmation request.

[0121] Based on the current execution context and the specific problem encountered, natural language queries are dynamically generated. Question generation takes into account the user's technical skill level and the complexity of the current task, employing concise and clear expressions. Multiple question types are supported: multiple choice, confirmation, and supplementary information questions, etc.

[0122] (2) Send the confirmation request to the voice agent through the communication bridge connection.

[0123] A bidirectional communication method is adopted, using a communication bridge to transmit questions from the browser agent to the voice agent, and an asynchronous message mechanism is used to ensure that the main execution process is not blocked. The message contains complete information such as question type, option list, and timeout settings.

[0124] (3) The voice agent converts the received confirmation request into voice and broadcasts it to the user, and then enters a waiting response state.

[0125] After receiving a question, the voice agent converts the text into speech using a TTS (Text-to-Speech) engine.

[0126] It supports multiple speech synthesis engines, including local TTS and cloud-based TTS services. It also implements speech playback queue management, supporting interruption and priority control.

[0127] After the voice broadcast is completed, the system enters a state of waiting for user response. The front-end recording system prepares to receive the user's voice reply, thus preparing for the next step of user response processing.

[0128] S4-2. Drive the browser to perform corresponding automated operations based on structured tasks, including user answers and result feedback.

[0129] In this embodiment, the specific process is as follows:

[0130] (1) When a user answers a confirmation request, the user's voice response is collected and recognized. The user's voice response is processed to obtain the user's voice response parsing result.

[0131] Steps S1 to S3 are executed sequentially. Specifically, when the user begins to answer, the front end automatically activates the recording function and uses a speech activity detection (VAD) algorithm based on energy detection to automatically identify the start and end of speech by continuously analyzing the short-term energy values ​​of the audio stream. The input audio is processed in frames, with each frame length set to 30 milliseconds. The root mean square energy value of each frame is calculated as the basic feature for determining speech activity. In non-recording mode, the system continuously learns the ambient noise level and updates the noise baseline using an exponential moving average algorithm, i.e., current noise baseline = α × historical noise baseline + (1-α) × current frame energy value, where α is set to 0.95 to ensure smooth learning and avoid sudden noise interference. The dynamic determination threshold for speech activity is set to max(minimum threshold, noise baseline × multiplier factor + fixed offset), where the minimum threshold is set to 0.015 to ensure basic sensitivity, the multiplier factor is set to 2.2 to allow the threshold to adapt to changes in the environment, and the fixed offset is set to 0.003 to provide additional tolerance. When the short-time energy value of 5 consecutive frames exceeds the dynamic threshold, it is determined that the voice has started and recording is automatically triggered; when the short-time energy value of 18 consecutive frames is lower than the dynamic threshold, it is determined that the voice has ended and recording is automatically stopped, thereby realizing intelligent silence detection, avoiding false triggering and premature end of recording, and automatically extracting audio segments containing valid user voice for subsequent transmission and processing.

[0132] (2) Perform semantic analysis on the recognized user voice response to obtain the user voice response analysis result.

[0133] The recognition results are semantically parsed to extract valid decision-making information. Next, the validity of the answer is verified by checking if it is empty. If not empty, a large language model is used for semantic detection to determine if the context is consistent. If consistent, the parsed information is valid, thus verifying the answer's validity. For invalid or ambiguous answers, clarification questions are automatically generated. Standardization of various answer formats is supported.

[0134] (3) The browser agent receives the parsing results of the user's voice response through the communication bridge and continues to execute the suspended structured task.

[0135] The communication bridge transmits the user's decision information to the browser agent, including the decision type, specific choice, and confidence level, and implements a transmission confirmation mechanism to ensure that the browser agent correctly receives and understands the user's decision.

[0136] After the user's answer is processed, the browser agent receives the decision information, resumes the suspended task, and adjusts the execution strategy and parameters according to the user's answer.

[0137] (4) After the browser agent completes the operation corresponding to the structured task, it generates task execution result information and sends it to the voice agent through the communication bridge.

[0138] 1) Structured task completion detection: The browser agent implements a multi-layered task completion judgment mechanism, including operation success verification, page status check, and goal achievement confirmation, and uses timeout mechanism and retry strategy to handle abnormal situations.

[0139] 2) Result information collection: Collect detailed result information of task execution, including operation success status, data content obtained, page screenshots, execution time, etc. The result data is stored in a structured format to facilitate subsequent analysis and user feedback.

[0140] 3) Intelligent Result Summary: The DeepSeek LLM large language model intelligently summarizes the execution results and generates user-friendly feedback. The summary includes task completion status, key information obtained, and possible follow-up operation suggestions.

[0141] (5) The voice agent converts the task execution result information into voice broadcast content and drives the front-end interface to play the voice to the user.

[0142] 1) Speech is generated through a TTS engine. After receiving the result information, the voice agent generates natural and fluent voice feedback. It supports emotional speech synthesis, adjusting the tone of voice according to the success or failure of the task, realizing personalized customization of voice content, and adjusting the broadcast style according to user preferences.

[0143] Among them, the TTS engine (Text-to-Speech Engine) is a text-to-speech engine.

[0144] 2) Drive the front-end interface to play voice messages, and implement intelligent voice playback control on the front-end interface; and use audio queue management to ensure the orderly playback of multiple voice messages.

[0145] After the result feedback is completed, the system returns to standby mode, ready to receive new voice commands, while maintaining the session context and preparing for possible interruption mechanisms.

[0146] like Figure 1 As shown, in step S5, if a new user voice input is detected during the process of the voice agent performing voice broadcast or the browser agent performing a structured task, an interruption mechanism is triggered. The interruption mechanism includes at least: immediately stopping the current voice broadcast and sending an interruption signal to the browser agent through the communication bridge to interrupt the currently executing structured task.

[0147] In this example, the specific process includes:

[0148] (1) The front-end interface monitors the audio input signal in real time. When it detects the start of a voice activity that meets the preset conditions, it immediately generates an interruption trigger event and sends it to the voice agent.

[0149] For real-time detection of voice activity, the front end continuously monitors voice activity and uses the audio analysis API of WebRTC to detect audio input level. A dynamic threshold algorithm is set to automatically adjust the detection sensitivity according to the ambient noise, realizing a fast response mechanism that triggers the interruption process immediately upon detecting voice activity.

[0150] Specifically, the front-end interface monitors the audio input signal in real time through the WebRTC audio analysis API, continuously calculates the instantaneous energy value of the audio stream, and compares it with a dynamic threshold. This dynamic threshold is not a fixed value but is adjusted in real time according to ambient noise using an adaptive algorithm: the system maintains a noise baseline value, which is updated only in non-recording states using an exponential moving average algorithm, i.e., N_new = 0.95 × N_old + 0.05 × E_current, ensuring that the noise baseline smoothly reflects environmental changes. The speech detection threshold T = max(0.015, N × 2.2 + 0.003), where the multiplier factor of 2.2 ensures effective differentiation between human voices and noise in noisy environments, and the minimum threshold of 0.015 prevents oversensitivity in extremely quiet environments. When the instantaneous energy value of 5 consecutive frames exceeds the dynamic threshold T, it is determined as the start of valid speech activity, immediately generating an interruption trigger event and sending it to the speech agent to trigger the interruption process, achieving a rapid response mechanism.

[0151] (2) After receiving the interruption trigger event through the first real-time communication link, the voice agent sets the global interruption flag to an effective state and simultaneously sends an interruption signal to the browser agent through the communication bridge.

[0152] 1) Set an interruption flag. The voice agent maintains a global state flag (should_interrupt). When the user starts speaking, the flag is immediately set to True. Atomic operations are used to ensure the thread safety of the flag setting and avoid concurrent access conflicts.

[0153] 2) A dual-path interruption signal transmission is adopted, namely, using both WebSocket and an internal bridge to ensure the reliability and real-time performance of signal transmission. The WebSocket path is responsible for fast signal transmission from the front end to the back end, while the internal bridge is responsible for state synchronization between agents.

[0154] (3) After receiving the interruption signal, the browser agent interrupts the currently executing task and cleans up the temporary state related to the structured task.

[0155] 1) Implement interruptible design for the browser agent's task execution. Check the interruption flag before each operation step and adopt a graceful interruption mechanism to ensure that the interruption is responded to only after the current operation is completed, thus avoiding inconsistencies in system state.

[0156] The graceful interruption mechanism achieves real-time interruption through state flag-driven, asynchronous collaborative cancellation, and global collaborative cleanup. When a user's interruption intention is detected, the voice agent immediately sets a global interruption flag and stops its own output process. Simultaneously, it sends an interruption signal to the browser agent via a dual-path communication bridge. Upon receiving the signal, the browser agent synchronizes its state and terminates the collaborative task by throwing a catchable cancellation exception to the currently executing asynchronous task. The task gracefully exits at the next waiting point and skips subsequent state persistence operations. Finally, the system uniformly cleans up the states of all components and broadcasts the interruption notification, enabling all agents to synchronously reset to a clean, ready state. This ensures the consistency of browser sessions, memory resources, and system states while responding to interruptions in milliseconds.

[0157] 2) State cleanup and recovery. When an interruption occurs, the system executes a complete state cleanup process, including canceling pending tasks, cleaning up temporary data, resetting the agent state, etc., implementing a state snapshot mechanism to support state recovery when necessary.

[0158] (4) After interrupting the task, the browser agent returns an interruption confirmation message to the voice agent through the communication bridge; the voice agent updates its status according to the received interruption confirmation message and stops the ongoing voice broadcast.

[0159] After the interruption is resolved, the system prepares to receive new voice commands, and the browser agent enters a ready state, waiting for the distribution and execution of new tasks.

[0160] like Figure 1 As shown, in step S6, the browser agent receives a new task and drives the browser to perform specific operations.

[0161] The browser agent receives a new task and employs a browser-use framework based on visual perception and artificial intelligence decision-making. This framework analyzes the visual interface of a webpage and combines reasoning from a large language model to convert structured tasks into specific coordinate instruction sequences, driving the underlying browser control engine to execute specific operations. The specific process is as follows:

[0162] (1) The structured task description is decomposed and planned into a sequence of instructions that conforms to the browser operation logic through the Browser-Use framework. The parsing process combines visual perception information with multi-strategy element positioning logic.

[0163] A browser automation (Browser-Use) framework based on visual perception and artificial intelligence decision-making is adopted, with Browser-Use as the core framework, providing high-level browser operation abstractions. The framework supports multiple browser engines, including Chromium and Firefox, achieving cross-browser compatibility.

[0164] The Browser-Use framework has a built-in intelligent task parser that converts structured task descriptions into specific browser operation sequences. It supports the decomposition and optimization of complex tasks, including operation path planning and element location strategy selection.

[0165] (2) The underlying browser control engine (Playwright) executes a sequence of instructions to perform operations on the target web page elements in a browser environment that supports dynamic page loading and asynchronous content rendering.

[0166] The underlying browser control engine is Playwright, providing stable and reliable browser automation capabilities. Playwright supports modern web technologies, including complex scenarios such as SPA applications, dynamic content, and asynchronous loading.

[0167] It employs a multi-strategy element location mechanism, including CSS selectors, XPath, text content, and visual features. Machine learning algorithms are used to optimize element recognition accuracy, and it supports adaptive handling of page structure changes.

[0168] (3) During the execution process, the page status changes are dynamically monitored, and mechanisms including intelligent waiting, operation retry and strategy adjustment are adopted to ensure the accuracy and robustness of the operation.

[0169] Implement an intelligent waiting mechanism to automatically handle timing issues such as page loading, animation effects, and asynchronous requests. Support automatic retries and strategy adjustments for failed operations to improve the success rate.

[0170] During browser operations, the system continuously monitors page state changes, providing necessary state information and change notifications for the session persistence mechanism.

[0171] like Figure 1 As shown, step S7 also includes session persistence operations.

[0172] In this embodiment, the specific process is as follows:

[0173] (1) During the execution of structured tasks by the browser agent, continuously monitor the state changes of the browser session.

[0174] During the operation of the browser agent, a lightweight state monitoring method is adopted to periodically check and record the following: 1) the current page URL and title; 2) the number of task execution steps; 3) the usage status of the Large Language Model (LLM); and 4) user data directory information, forming a recoverable session context. An event-driven pattern is adopted to capture state changes in real time and trigger corresponding processing.

[0175] (2) Serialize and persist the state of the browser session.

[0176] An automatic serialization mechanism and a metadata-based session persistence mechanism are employed. Specifically, a lightweight state serialization method is designed to convert key metadata of the browser session (URL, title, operation steps, model type) into storable JSON format data. Serialized content includes information such as URL, title, and timestamp. An incremental serialization strategy is adopted, saving only the changed parts to improve storage efficiency.

[0177] It enables persistent data storage and implements a multi-tiered storage architecture, including memory caching, local file storage, and database persistence. It supports configuring different storage strategies, allowing users to select the appropriate storage method based on data importance and access frequency.

[0178] (3) When the session needs to be resumed after it is interrupted, load the saved serialized state data and reconstruct the browser session based on the serialized state data; notify the voice agent to resume the conversation through the communication bridge.

[0179] First, the system checks for the existence of a recoverable session using an internal validation method defined by the agent. The `_is_session_valid` method determines if the session is valid within 24 hours, preventing the recovery of expired data. When a valid session is detected, the system interactively confirms with the user whether to restore the session, receiving the user's choice through a user decision variable to ensure the initiative and controllability of the restoration decision.

[0180] The recovery process employs a layered mechanism, retrieving basic session information from structured session state data (session_data), restoring the browser state from the browser runtime state descriptor (browser_state), and, where possible, restoring the agent's internal state from agent_state. The system prioritizes navigating to the saved target_url; if recovery fails, it automatically degrades to creating a new session, ensuring system robustness.

[0181] The entire recovery process uses SessionRestoreHelper to verify page navigation. If the recovery fails after multiple retries, it is abandoned. This strategy balances automated recovery with user control, enabling rapid restoration of the working state after system restarts or abnormal interruptions while avoiding forced recovery of invalid or outdated states.

[0182] It provides complete session recovery functionality, supporting scenarios such as state recovery after system restart and task continuation after abnormal interruption, and implements intelligent recovery strategies to select the recovery method based on the interruption time and state integrity.

[0183] Maintain the integrity and consistency of operation sequences to ensure the continuous execution of multi-step tasks. Implement transactional operation support, enabling operation rollback and redo. Establish an operation dependency graph to ensure the correct execution order of complex tasks.

[0184] The session persistence mechanism serves as the foundation of the system, providing state guarantees and continuity support for all other steps, ensuring that the entire dual AI Agent system can stably and reliably provide users with intelligent browser automation services.

[0185] Example 2

[0186] like Figure 2 As shown, the purpose of this embodiment is to provide a voice-controlled browser operating system based on real-time communication between two intelligent agents, including:

[0187] The voice input module is used to respond to the user's voice input operation. The front-end interface obtains audio data and sends it to the voice agent through the first real-time communication link.

[0188] The speech recognition and task parsing module is used by the speech agent to parse the user command text through a large language model, generate structured tasks, and distribute the structured tasks to the browser agent through a communication bridge.

[0189] The browser operation and feedback module is used by the browser agent to receive structured tasks and drive the browser to perform corresponding automated operations based on the structured tasks. During or after the browser operation, the browser agent feeds back the execution status or result information to the voice agent through the communication bridge, and the voice agent then transmits interactive feedback to the user.

[0190] The real-time interruption module is used to trigger an interruption mechanism if a new user voice input is detected during the process of a voice agent performing voice broadcast or a browser agent performing a structured task. The interruption mechanism includes at least: immediately stopping the current voice broadcast and sending an interruption signal to the browser agent through the communication bridge to interrupt the currently executing structured task.

[0191] The browser control and execution module is used by the browser agent to receive new tasks. The browser agent adopts a browser automation (Browser-Use) framework based on visual perception and artificial intelligence decision-making. This framework analyzes the visual interface of the webpage and combines the reasoning of the large language model to convert the structured task into a specific coordinate instruction sequence, and drives the underlying browser control engine to perform specific operations.

[0192] It also includes a session persistence management module, which continuously monitors the state changes of the browser session during the execution of structured tasks by the browser agent; serializes and persistently saves the state of the browser session; when the session needs to be resumed after an interruption, it loads the saved serialized state data and reconstructs the browser session based on the serialized state data; and notifies the voice agent to resume the dialogue through the communication bridge.

[0193] The voice-controlled browser operating system based on real-time communication between two agents specifically includes a front-end interface component, a language agent component, a browser agent component, a communication bridge, and an interruption mechanism component.

[0194] Among them, 1) the language intelligence agent component performs the following functions:

[0195] Handling user voice interaction;

[0196] Speech recognition: Chinese speech to text conversion using a local Whisper ASR model (small version);

[0197] Speech synthesis: Text-to-speech is achieved through Edge-TTS, supporting Chinese speech;

[0198] Natural Language Understanding: Dialogue understanding and generation using the DeepSeek large language model;

[0199] Audio processing: Convert WebM to WAV format using FFmpeg;

[0200] Interruption mechanism: Real-time detection of user recording start, immediately stop the current TTS playback.

[0201] 2) The browser smart body component performs the following functions:

[0202] Perform automated browser operations; based on the browser-use framework, integrate Playwright for browser control;

[0203] Visual understanding: Identify page elements using the visual capabilities of Claude Sonnet 4 or DeepSeek models;

[0204] Session persistence: Maintains browser state via BrowserProfile and supports session recovery.

[0205] Task execution engine: Intelligent task decomposition and execution based on Large Language Model (LLM);

[0206] EventBus mechanism: Implements an event-driven architecture using the bubus library;

[0207] Receive task instructions from the telephone agent via WebSocket;

[0208] Use the browser-use Agent to perform automated browser operations;

[0209] Supports task interruption: Task interruption can be achieved using asyncio.Task.cancel();

[0210] Session management: Saves and restores browser state, supporting cross-session operations.

[0211] Among them, Browser-Use automation engine: an advanced browser automation framework; Playwright browser driver: a low-level browser control driver that controls Chrome / Chromium browsers; multi-model support: Claude / DeepSeek dual-model decision-making, analyzing the page DOM structure through Large Language Model (LLM); identifying operable elements; generating operation sequences such as clicks, inputs, and scrolling based on user instructions; translating natural language instructions into specific browser operations; session persistence: saving and restoring browser state.

[0212] 3) The WebSocket communication bridge (AgentBridge) performs the following functions:

[0213] The system's core communication hub enables efficient collaboration between the two agents.

[0214] The system includes: message queue management (enabling bidirectional asynchronous message passing, supporting message ID tracking and response waiting mechanisms); connection status monitoring (real-time monitoring of the connection status between two agents, providing automatic reconnection); message routing and distribution (intelligent routing based on message type and target agent, supporting task distribution and result feedback); status synchronization mechanism (real-time synchronization of task execution status, progress, and system status information); question-and-answer coordination system (supporting browser agents to request confirmation information from users, managing the question-and-answer process); and concurrency control (supporting concurrent processing of multiple messages, preventing message blocking, and ensuring communication stability).

[0215] 4) The interruption mechanism component performs the following functions:

[0216] Enables users to interrupt ongoing tasks in real time, providing a smooth interactive experience;

[0217] Front-end recording event listener: Real-time detection of the user's action of starting recording, as an interruption trigger signal;

[0218] Multi-layered interruption strategies include: immediate stop of voice playback: setting the `should_interrupt` flag to immediately terminate the current TTS speech synthesis and playback process; task future cancellation: canceling the task being executed by the browser agent using the `asyncio.Task.cancel()` method; EventBus cleanup: gracefully shutting down and cleaning up the EventBus resources inside the browser agent to avoid resource leaks; state reset: clearing the task state and preparing to receive new user commands; dual-path interruption signal transmission: WebSocket path: sending an interrupt message to the Browser Agent via the VoiceAgentBridge's WebSocket connection; internal bridge path: sending an interrupt message via the AgentBridge's message queue system to ensure reliable signal transmission; front-end synchronization control: sending the `tts_stop` command to the front end to immediately stop the audio player.

[0219] State consistency maintenance: Ensure that the states of the two agents are synchronized after an interruption, and avoid anomalies caused by state inconsistency.

[0220] 5) The front-end interface is the direct entry point for user interaction with the system. It adopts a modular architecture based on the HTML5 standard, and its core consists of two main components:

[0221] The network socket client, implemented based on the HTML5 WebSocket API, is responsible for establishing a persistent, low-latency, bidirectional communication connection with the speech agent's WebSocket server. This client not only handles the chunked uploading of audio data but also receives various control commands and feedback data from the speech agent in real time, including transcription results, AI-generated response text, synthesized speech streams, system status updates, and interruption notifications. A heartbeat mechanism and automatic reconnection strategy ensure the stability and high availability of the communication link.

[0222] HTML5 Audio / Status Display Module: This module integrates audio acquisition and status visualization. The audio acquisition section, based on the MediaRecorder API and Web Audio API, dynamically requests microphone access, captures and processes audio streams in real time (WebM format, 16kHz sampling rate, mono), and intelligently controls recording start and stop using a voice activity detection algorithm. The status display section employs a responsive UI design, using dynamic icons, text prompts, and visual waveforms to present real-time system status (standby, recording, processing, voice playback), recording level, connection status, and error messages, providing users with clear and intuitive interactive feedback and enhancing the smoothness and controllability of the user experience.

[0223] The method steps in Embodiment 1 are implemented based on a voice-controlled browser operating system that provides real-time communication between two intelligent agents.

[0224] Example 3

[0225] The purpose of this embodiment is to provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described method.

[0226] Example 4

[0227] The purpose of this embodiment is to provide a computer-readable storage medium.

[0228] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the above method.

[0229] Example 5

[0230] The purpose of this embodiment is to provide a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods and functions involved in any of the above embodiments.

[0231] The steps and methods involved in the apparatus of the above embodiments correspond to those in Embodiment 1. For specific implementation details, please refer to the relevant description section of Embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and enabling the processor to perform any of the methods in this invention.

[0232] Those skilled in the art will understand that the modules or steps of the present invention described above can be implemented using general-purpose computer devices. Optionally, they can be implemented using computer-executable program code, thereby allowing them to be stored in a storage device for execution by a computer device, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. The present invention is not limited to any particular combination of hardware and software.

[0233] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A voice-controlled browser operation method based on real-time communication between two intelligent agents, characterized in that, Executed collaboratively by a voice agent and a browser agent connected in real time via a communication bridge, including: In response to the user's voice input, the front-end interface acquires audio data and sends it to the voice agent through the first real-time communication link; The voice-activated agent recognizes audio data to obtain the user's command text; The voice agent parses the user's command text using a large language model, generates structured tasks, and distributes the structured tasks to the browser agent through a communication bridge. The browser agent receives structured tasks and drives the browser to perform corresponding automated operations based on the structured tasks. During or after the browser operation, the browser agent feeds back the execution status or result information to the voice agent through the communication bridge, and the voice agent then broadcasts the information to the user. During the process of a voice agent performing voice broadcasting or a browser agent executing a structured task, if a new user voice input is detected, an interruption mechanism is triggered. This interruption mechanism includes at least: immediately stopping the current voice broadcast and sending an interruption signal to the browser agent via a communication bridge to interrupt the currently executing structured task. The specific process includes: The front-end interface monitors the audio input signal in real time. When it detects the start of voice activity that meets the preset conditions, it immediately generates an interruption trigger event and sends it to the voice agent. After receiving the interruption trigger event through the first real-time communication link, the voice agent sets the global interruption flag to an active state and simultaneously sends an interruption signal to the browser agent through the communication bridge. Upon receiving an interruption signal, the browser agent suspends the currently executing task and cleans up the temporary state related to the structured task. A graceful interruption mechanism is employed, which achieves real-time interruption through state flag-driven, asynchronous collaborative cancellation, and global collaborative cleanup. After the browser agent interrupts its task, it returns an interruption confirmation message to the voice agent through the communication bridge. The voice agent updates its status based on the received interruption confirmation information and stops the ongoing voice broadcast. The communication bridge uses an asynchronous message queue mechanism to realize bidirectional asynchronous communication between the browser agent and the voice agent. The message queue is used to manage the transmission of task requests, status feedback, queries and answers, and interruption signals, and assigns a unique identifier to each message to support task status tracking, execution progress monitoring, error handling, and retry mechanisms. It also includes session persistence operations.

2. The voice-controlled browser operation method based on real-time communication between two agents as described in claim 1, characterized in that, It also includes session persistence operations, the specific process of which is as follows: During the execution of structured tasks by the browser agent, the state changes of the browser session are continuously monitored; Serialize and persist the state of the browser session; When a session needs to be resumed after it has been interrupted, the saved serialized state data is loaded, and the browser session is reconstructed based on the serialized state data. The voice agent is notified to resume the conversation via a communication bridge.

3. The voice-controlled browser operation method based on real-time communication between two agents as described in claim 1, characterized in that, The specific process of obtaining the user instruction text includes: The front-end interface adopts a modular architecture based on Hypertext Markup Language and is designed with a state machine to trigger voice input operations and call the device's audio interface to collect audio data. The acquired audio data is preprocessed, including at least format standardization and encapsulation. The preprocessed audio data is sent to the voice agent in a streaming and chunked manner through the first real-time communication link; The voice agent receives preprocessed audio data via a WebSocket server and stores it in a temporary audio buffer until the end of voice input is detected. When the end of the voice input is detected, the audio data is extracted from the temporary audio buffer and recognized by the speech recognition model, and the audio data is converted into user command text.

4. The voice-controlled browser operation method based on real-time communication between two agents as described in claim 1, characterized in that, The voice agent parses user command text using a large language model, generates structured tasks, and distributes these structured tasks to the browser agent via a communication bridge. The specific process includes: The voice agent inputs user command text into a large language model and uses an intent parsing algorithm for speech understanding and intent parsing; Based on the intent parsing results, user commands are converted into structured tasks. The structured tasks adopt a predefined format and include key fields such as operation type, operation target, operation parameters and execution conditions. The voice agent distributes structured tasks to the browser agent through a communication bridge, and the task distribution is handled using a message queue pattern.

5. The voice-controlled browser operation method based on real-time communication between two agents as described in claim 1, characterized in that, The browser agent receives structured tasks and drives the browser to perform corresponding automated operations based on these tasks, including user queries. The specific process is as follows: When the browser agent encounters a situation requiring user confirmation while performing a structured task, it pauses execution and generates a confirmation request. The confirmation request is sent to the voice agent via a communication bridge connection; The voice agent converts the received confirmation request into voice and broadcasts it to the user, then enters a waiting state for a response.

6. The voice-controlled browser operation method based on real-time communication between two agents as described in claim 1, characterized in that, The browser agent receives structured tasks and drives the browser to perform corresponding automated operations based on these tasks. This also includes user responses and result feedback. The specific process is as follows: When a user answers a confirmation request, the user's voice response is collected and recognized. Semantic analysis is performed on the recognized user voice response to obtain the user voice response analysis result; The browser agent receives the parsing results of the user's voice response through the communication bridge and continues to execute the suspended structured task; After the browser agent completes the operation corresponding to the structured task, it generates task execution result information and sends it to the voice agent through the communication bridge. The voice agent converts task execution results into voice broadcast content and drives the front-end interface to play the audio to the user.

7. A voice-controlled browser operating system based on real-time communication between two intelligent agents, characterized in that: The method for operating a voice-controlled browser based on real-time communication between two agents as described in any one of claims 1-6 includes: The voice input module is used to respond to the user's voice input operation. The front-end interface obtains audio data and sends it to the voice agent through the first real-time communication link. The speech recognition module is used by the voice intelligence agent to recognize audio data and obtain the user's command text; The task parsing and distribution module is used by the voice agent to parse the user's command text using a large language model, generate structured tasks, and distribute the structured tasks to the browser agent through a communication bridge. The browser operation and feedback module is used by the browser agent to receive structured tasks and drive the browser to perform corresponding automated operations based on the structured tasks. During or after the browser operation, the browser agent feeds back the execution status or result information to the voice agent through the communication bridge, and the voice agent then transmits interactive feedback to the user. The real-time interruption module is used to trigger an interruption mechanism if a new user voice input is detected during the process of a voice agent performing voice broadcast or a browser agent performing a structured task. The interruption mechanism includes at least: immediately stopping the current voice broadcast and sending an interruption signal to the browser agent through the communication bridge to interrupt the currently executing structured task.

8. The voice-controlled browser operating system based on real-time communication between two agents as described in claim 7, characterized in that, It also includes a session persistence management module, which continuously monitors the state changes of the browser session during the execution of structured tasks by the browser agent; serializes and persistently saves the state of the browser session; when the session needs to be resumed after an interruption, it loads the saved serialized state data and reconstructs the browser session based on the serialized state data; and notifies the voice agent to resume the dialogue through the communication bridge.

Images