Intelligent digital employee demonstration control system and method supporting voice broadcast

Abstract

The application discloses a kind of intelligent body digital staff demonstration control system and method supporting voice broadcast, it is related to digital staff and intelligent demonstration technical field, the system is by demonstration control layer, page instance layer, audio subsystem, intelligent body subsystem and background recovery subsystem constitute;The method includes initialization, demonstration broadcast, interrupt and snapshot, intelligent body interaction, recovery and five stages of continuous broadcast.This application realizes the unified scheduling of demonstration and interaction dual mode by directed state machine, realizes the efficient page control of large model based on barrier-free tree reference mark, realizes the dynamic switching of multi-system page by closure binding, realizes the background recovery of page state by combining action classification and asynchronous parallel mechanism, while supporting double interruption of key and voice, unified audio channel management and multi-modal interaction.This application can realize the whole-process closed loop of broadcast, interruption, intelligent body takeover, parallel recovery and seamless continuous broadcast, significantly improve the flexibility of demonstration and user experience.

Description

Technical Field

[0001] This invention relates to the field of digital employee and intelligent presentation technology, specifically to an intelligent agent digital employee presentation control system and method that supports voice broadcasting. Background Technology

[0002] In existing technologies, there are two typical technical solutions for the automated manipulation and demonstration of web pages: web UI automated testing solutions based on visual language models and page understanding solutions in RPA scenarios. For example, Chinese patent application CN122111855A discloses an automated web UI testing method and system based on visual language models. This solution parses operation actions and target objects from test scripts, extracts interface elements based on web interface screenshots and accessibility trees through visual language models and rule parsing, and uses multimodal semantic alignment to fuse visual and semantic features, driving the visual language model to iteratively execute tests through interface observation, semantic reasoning, and operation generation. Its application is limited to web UI automated testing. For example, Chinese patent application CN119938040A discloses a data structure construction method for understanding graphical user interfaces in RPA scenarios. This scheme obtains a screenshot of the target page and an accessibility tree of the page's backend code. Based on the accessibility tree and the screenshot, it obtains visible elements, element boxes, and basic data. It sorts the element boxes by area to construct a view relationship tree and uses the view relationship tree and basic data to construct a view relationship graph, which serves as the data structure for RPA robots to understand graphical user interfaces. Its purpose is solely for page understanding in RPA process automation.

[0003] The aforementioned existing technical solutions have the following core defects:

[0004] 1. The demonstration process lacks a flexible user intervention and recovery mechanism. Existing demonstration systems are mostly one-way, uninterruptible scripted processes. Users cannot interrupt the demonstration in real time, interact with the page using natural language, or ask questions, and cannot seamlessly resume the original demonstration progress after the interaction. The two aforementioned solutions, respectively targeting batch execution of test scripts and offline data structure construction for RPA, do not support real-time user intervention for interruption or recovery after interaction; furthermore, user intervention is limited to physical button or GUI button operations, lacking support for real-time interruption and broadcasting via natural voice, as well as voice command input during the agent interaction phase, resulting in insufficient naturalness and flexibility in interaction.

[0005] 2. Large-scale models lack efficient and structured perception methods for manipulating web pages. In existing solutions, large language models mainly rely on visual screenshots or fixed CSS / XPath selectors to manipulate browser pages: solutions relying on visual model screenshots suffer from high on-device computing power consumption and poor real-time performance; solutions using accessibility trees are only used to build offline RPA data structures and are not designed as real-time operation interfaces that large language models can directly understand and reference.

[0006] 3. Lack of unified agent context management for multi-system page switching operations. When demonstrations involve multiple independent web system pages, existing solutions typically require users to switch manually or through predefined scripts. There is no mechanism that allows a large language model agent to autonomously perceive the current page position, dynamically bind operation tools to the target page, and freely switch between multiple systems while maintaining the continuity of the dialogue context.

[0007] 4. Modifications made to the page by the agent result in inconsistent states when returning to the original demonstration flow. When the agent performs operations such as navigation, filling in information, and clicking on the page during the takeover period, the page state deviates from the preset state of the original demonstration flow. Existing solutions either require the user to manually restore the page or abandon the restoration and continue directly, resulting in demonstration errors. There is a lack of technical solutions that can restore the page state in parallel during user interaction, so that the user can return without any waiting time. Summary of the Invention

[0008] To address the aforementioned shortcomings of existing technologies, the present invention aims to provide a digital employee demonstration control system and method that supports voice broadcasting. This system enables real-time interruption via either button presses or natural voice, page manipulation and questioning using natural language or voice, and seamless resumption of the demonstration from the interruption point after interaction. It manages pre-generated scripted broadcast voice and real-time generated agent response voice in a unified audio channel, providing a unified interruption mechanism covering both voice sources and interruption methods, and supporting voice input during agent interaction. It provides a structured, frame-independent Web page perception method for large language models, enabling them to understand Web page structure and accurately locate and manipulate page elements without relying on visual models or fixed selectors. It maintains the contextual continuity of agents in multi-system page environments, allowing agents to freely switch operations between multiple independent Web systems while maintaining the immediate availability of dialogue history, page perception, and operation tools. Simultaneously, the background restores the state of modified pages during user interaction, ensuring the page is ready to resume the demonstration directly from the breakpoint when the user interaction ends.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] In a first aspect, the present invention provides an intelligent agent digital employee demonstration control system that supports voice broadcasting, including a demonstration control layer, a page instance layer, an audio subsystem, an intelligent agent subsystem, and a background recovery subsystem;

[0011] The demonstration control layer is implemented based on a directed state machine and includes sequentially associated initialization nodes, demonstration broadcasting nodes, interruption handling nodes, intelligent agent interaction nodes, and state recovery nodes, which are used to schedule the system operation process and state transitions.

[0012] The page instance layer is built on a browser automation framework and is used to create an independent browser tab instance for each target business system. Each tab maintains an independent running state.

[0013] The audio subsystem includes a pre-generated TTS engine, a real-time TTS engine, a unified player, an ASR (Automatic Speech Recognition) speech recognition engine, and an interruption monitoring module, which are used to uniformly manage speech output, speech input recognition, and interruption monitoring.

[0014] The intelligent agent subsystem is built on the ReAct (Reasoning + Acting) paradigm and includes the PageAgent inference engine and a callable toolset, which are used to complete page manipulation and knowledge response based on the user's natural language instructions.

[0015] The background recovery subsystem includes an action classifier, a parallel recovery engine, and a page locator, which are used to asynchronously and in parallel restore the demonstration state of the modified page during agent interaction.

[0016] As a further aspect of the present invention, the functions of each node in the demonstration control layer are as follows:

[0017] Initialize nodes to load broadcast script configuration and multi-system page configuration, start browser and create independent tabs for each system, batch pre-generate TTS audio files for all broadcast steps, load speech keyword interruption dictionary, initialize ASR speech recognition engine and start background monitoring, and inject domain knowledge base into the intelligent agent dialogue context.

[0018] The demonstration broadcast node is used to iterate through the script steps in sequence, switch to the target system page and execute preset page actions, play pre-generated TTS voice synchronously, and listen for interruption signals from keyboard hotkeys and voice keywords. When an interruption is detected, a state transition is triggered.

[0019] The interrupt processing node is used to stop audio playback and set an interruption flag, save a snapshot of the presentation progress containing the current step index and a list of actions executed on each page, asynchronously start the background recovery engine, determine the currently active page, and initialize the intelligent agent toolset;

[0020] The agent interaction node is used to receive user text or voice input, route it to the agent inference engine to execute the ReAct loop, update the dialogue history and active page state, convert the agent's reply into voice broadcast, and wait for user input in a loop until it receives a command to continue or exit.

[0021] The state recovery node is used to wait for the background recovery task to complete, replace the old page controlled by the agent with the new page instance restored in the background, restore the demo progress snapshot, and return to the demo broadcast node to continue execution from the breakpoint.

[0022] As a further aspect of the present invention, in the audio subsystem:

[0023] The pre-generated TTS engine is used to batch convert the broadcast text of all demonstration steps into audio files and cache them locally during the initialization phase.

[0024] The real-time TTS engine is used to convert the agent's reply text into audio in real time during the agent interaction phase;

[0025] The unified player provides a unified playback control interface for pre-generated TTS and real-time TTS, and runs in non-blocking mode;

[0026] The ASR speech recognition engine runs continuously in the background thread and is used to demonstrate speech keyword recognition in the broadcasting stage and speech input to text conversion in the agent interaction stage.

[0027] The interruption monitoring module implements dual monitoring based on the keyboard channel and the voice channel. The keyboard channel monitors preset hotkeys through keyboard event hooks, and the voice channel matches preset keywords through ASR recognition results. When either channel is triggered, the current audio playback is stopped through a thread-safe event flag.

[0028] As a further aspect of the present invention, the toolset of the intelligent agent subsystem includes an accessibility tree snapshot tool, a page operation execution tool, a multi-system switching tool, a voice broadcast tool, and a knowledge retrieval tool;

[0029] The accessibility tree snapshot tool is used to obtain the complete accessibility tree of the page through the accessibility tree interface of the Chrome DevTools Protocol, assign reference tags to interactive elements and build an element attribute mapping table to generate structured page text;

[0030] The page operation execution tool is used to parse the operation commands output by the large language model, locate the target element by referencing the tag, locate the DOM element by step back through the multi-strategy locator and execute the corresponding page operation, and automatically refresh the page snapshot after the operation.

[0031] The multi-system switching tool is used to switch the browser focus to the target system tab according to instructions and automatically update the page snapshot and reference mapping.

[0032] The voice broadcasting tool is used to convert the agent's response into voice via real-time TTS and play it.

[0033] The knowledge retrieval tool is used to retrieve relevant knowledge slices from the domain knowledge base for the large language model to organize and respond to.

[0034] As a further aspect of the present invention, the PageAgent inference engine is built based on the LangGraph ReAct pattern, compiling a completely new inference subgraph each time a user command is triggered; the dialogue history is maintained at the main state machine level, and the first... The constraint on the number of reasoning steps for each interaction is:

[0035] ;

[0036] in, The maximum number of inference steps is preset, with a coefficient of 3 to accommodate the three graph nodes for each step, and a constant of 10 as a safety margin.

[0037] As a further aspect of the present invention, the toolset is implemented using the closure factory pattern, where each tool shares the accessibility tree snapshot text and reference mapping table through a variable dictionary within the closure; the multi-system switching tool dynamically changes the page references shared by all tools through the nonlocal keyword, so that subsequent operations are automatically applied to the new page.

[0038] As a further aspect of the present invention, in the background recovery subsystem:

[0039] The action classifier is used to preclassify script actions into destructive actions with irreversible server-side effects and non-destructive actions of pure UI navigation.

[0040] The parallel recovery engine is used to asynchronously create new tabs during agent interaction, replay only non-destructive actions according to the list of executed actions in the snapshot, and execute the recovery tasks of multiple pages concurrently.

[0041] The page locator is used to build a locator based on text descriptions, CSS selectors, or ARIA role information in the page configuration, and to perform a restore operation on a brand new page.

[0042] As a further aspect of the present invention, after the state is restored, the page... Action replay collection satisfy:

[0043] ;

[0044] In the formula, To interrupt the page List of executed action indexes For the first The name of the step. For the action classification indicator function, when The value is 1 if it belongs to the set of destructive actions, and 0 otherwise.

[0045] As a further aspect of the present invention, the page instance layer is built based on the Playwright framework, and each tab maintains an independent page URL, DOM tree and authentication session; during the demonstration and broadcasting phase, tabs are automatically switched and operations are executed according to the script; during the intelligent agent interaction phase, the control results are fed back in real time; after the interaction ends, the page instance is switched to the restored page instance in a hot-swap manner.

[0046] Secondly, the present invention also provides a method for controlling the demonstration of a digital employee with intelligent agent that supports voice broadcasting, comprising the following stages:

[0047] S1. Initialization: Parse configuration files and demo scripts, load domain knowledge base and interruption keyword library, initialize ASR engine and start background listening, batch generate pre-generated TTS audio, start browser and create independent tabs for each target system;

[0048] S2. Demonstration and Broadcast: Execute page operations in the order of the script and play the corresponding pre-generated voice, and monitor keyboard hotkeys and voice keywords for interruption signals in real time; if no interruption is detected, the current step is completed and the next step is performed; if an interruption is detected, the broadcast is interrupted.

[0049] S3. Interruption and Snapshot: Stop audio playback, save a state snapshot containing the current script step index and a list of actions executed on each page, determine the currently active page, and asynchronously start the background page recovery task;

[0050] S4. Intelligent Agent Interaction: Receives user text or voice commands, calls the intelligent agent to execute the ReAct inference loop to complete page manipulation or knowledge query, converts the reply into voice playback, and waits for user input in a loop; if an exit command is received, the system terminates; if a resume command is received, it enters the next stage.

[0051] S5. Restore and Resume Playback: Wait for the background restore task to complete, replace the old page that was manipulated with the new page after restoration, restore the demo progress snapshot, and return to the demo playback stage to continue execution from the breakpoint.

[0052] As a further aspect of the present invention, in step S2, the period for monitoring the interruption signal is 50 milliseconds; after detecting the interruption signal, the audio playback is immediately stopped and the broadcast loop is interrupted.

[0053] As a further aspect of the present invention, in step S3, the saved state snapshot satisfy:

[0054] ;

[0055] In the formula, This is an index of the script steps at the time of interruption. For the first A list of executed actions indexes for each page. This represents the total number of pages in the system.

[0056] As a further aspect of the present invention, in step S4, the specific process of the ReAct inference loop is as follows: obtain an accessibility tree snapshot of the current page, input the system prompts, dialogue history, page snapshot and user instructions into the large language model, the large language model generates a decision, execute the corresponding tool and obtain a new page snapshot, and iterate the inference until the final text response is generated.

[0057] As a further aspect of the present invention, in step S4, the voice command is recognized as text in real time by the ASR engine and then submitted to the intelligent agent; the intelligent agent's text reply is converted into voice in real time by TTS and then played.

[0058] As a further aspect of the present invention, in step S5, the specific method of page restoration is as follows: create a new tab for each manipulated page, navigate to the system base URL, and replay non-destructive actions according to the list of executed actions; the restoration tasks of multiple pages are executed concurrently.

[0059] As a further aspect of the present invention, in step S1, the pre-generated TTS audio is generated in batches asynchronously and stored in the local cache; the browser instance can reuse the local user data directory or pre-store the authentication status, and save the authentication status for subsequent reuse after the first login.

[0060] As a further aspect of the present invention, in step S2, after each page operation is performed, the corresponding step index is appended to the list of executed actions of the target page.

[0061] As a further aspect of the present invention, the interruption signal includes two types: keyboard hotkey triggering and voice keyword hitting. Both interruption methods apply to pre-generated speech and real-time generated speech.

[0062] Compared with existing technologies, the present invention provides a voice-enabled intelligent agent digital employee demonstration control system and method, which for the first time integrates barrier-free tree-driven LLM page operation, voice broadcasting and multimodal interruption of the demonstration process, multi-turn free interaction of the intelligent agent (including voice input), background parallel page recovery based on action classification, and unified audio channel management (including ASR) into a complete closed-loop system. It realizes fully automated control of the entire process, including broadcasting, button or voice interruption, intelligent agent multimodal takeover, parallel recovery, and seamless continuation. Specifically, it has the following beneficial effects:

[0063] 1. Significantly improves presentation flexibility and user experience. Compared with existing one-way, uninterruptible scripted presentation systems, this invention allows users to interrupt presentations at any time via hardware hotkeys or natural voice commands. Users can instruct the intelligent agent to control the page, query knowledge, and obtain voice responses via text or voice. After the interaction is completed, the user can resume from the interruption point with one click. The background parallel recovery mechanism ensures that the page is ready when the user's interaction ends, achieving a seamless transition without waiting. The pre-generated TTS and real-time TTS share a unified audio channel, avoiding channel conflicts and interruption failures.

[0064] 2. Achieve framework-independent, efficient large language model page manipulation. Compared to existing solutions that rely on visual model screenshots or fixed selectors, this invention constructs structured page text with reference tags through the Chrome DevTools Protocol's accessibility tree interface. This allows the large language model to precisely manipulate page elements without needing to understand the DOM structure or CSS rules. The accessibility tree, as a standard web interface, is universally applicable across front-end frameworks, and its compact and efficient text format significantly reduces the context consumption of the large language model. The step-by-step fallback mechanism of the multi-strategy locator further improves the success rate of operations.

[0065] 3. Provides unified control capabilities across multiple systems. Compared to existing single-system operation solutions, this invention, through a tool factory pattern with closure binding, enables agents to freely switch between multiple independent web systems. All tools are dynamically bound to the currently active page. When switching systems, there is no need to reinitialize the agent or interrupt the dialogue context. Each system page maintains an independent operating state, achieving true unified control across multiple systems.

[0066] 4. Solving the problem of restoring page consistency after intelligent agent manipulation. Compared with existing solutions that require manual page restoration or abandon restoration, this invention proposes a parallel background restoration method based on the classification of action effects. Script actions are divided into destructive and non-destructive categories. During user interaction, a new tab is created in parallel in the background, and only non-destructive actions are replayed to restore the page. The restoration process is executed concurrently with user interaction, and the user-perceived waiting time is zero.

[0067] 5. Ensuring the stability of inference in multi-turn dialogues. Compared with existing schemes that maintain a single inference graph for long conversations, this invention adopts an architecture that compiles a new inference graph for each interaction, avoiding the accumulation of state deviations in long dialogues. At the same time, it maintains the dialogue history at the main state machine level to ensure the continuity of multi-turn dialogues. The two-layer cascaded state machine architecture enables the two modes of demonstration broadcasting and agent interaction to share message context and page resources, and the state transition is clear and traceable.

[0068] 6. Achieving multimodal interaction to lower the operational threshold. Compared with existing demonstration systems that only support button operation and text input, this invention introduces an ASR speech recognition engine to enable voice keyword interruption and voice input commands. During the demonstration, users can interrupt the broadcast with natural speech without operating the keyboard. In the intelligent agent interaction stage, users can completely operate the page and ask questions by voice without the keyboard, significantly lowering the operational threshold. This is especially suitable for situations such as exhibition halls and press conferences where presenters need to balance body language and verbal expression, as well as for barrier-free operation scenarios for users with disabilities. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. In the drawings:

[0070] Figure 1 This is a block diagram of the overall architecture of the intelligent agent digital employee demonstration control system that supports voice broadcasting according to the present invention.

[0071] Figure 2 This is an architectural block diagram of the audio subsystem in the intelligent agent digital employee demonstration control system that supports voice broadcasting according to the present invention.

[0072] Figure 3 This is an architectural block diagram of the intelligent agent subsystem in the intelligent agent digital employee demonstration control system that supports voice broadcasting according to the present invention.

[0073] Figure 4 This is an architecture diagram of the background recovery subsystem in the intelligent agent digital employee demonstration control system that supports voice broadcasting according to the present invention.

[0074] Figure 5 This is a flowchart illustrating the overall control method for the intelligent digital employee demonstration system that supports voice broadcasting, as described in this invention. Detailed Implementation

[0075] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0076] This invention provides a digital employee demonstration control system and method supporting voice broadcasting. It enables real-time interruption via either button presses or natural voice, page manipulation and questioning using natural language or voice, and seamless resumption of the demonstration from the interruption point after interaction. Simultaneously, it manages pre-generated scripted broadcast voice and real-time generated agent response voice in a unified audio channel, providing a unified interruption mechanism covering both voice sources and interruption methods, and supporting voice input during agent interaction. It provides a structured, frame-independent Web page perception method for large language models, enabling them to understand Web page structure and accurately locate and manipulate page elements by understanding text, without relying on visual models or fixed selectors. It maintains the contextual continuity of agents in multi-system page environments, allowing agents to freely switch operations between multiple independent Web systems while maintaining the immediate availability of dialogue history, page perception, and operation tools. While the user interacts with the agent, the background parallelly restores the state of modified pages, ensuring the page is ready to continue the demonstration directly from the breakpoint when the user interaction ends.

[0077] This invention is applicable to scenarios requiring multi-system page demonstrations and flexible human-computer interaction, such as government exhibition halls, enterprise product demonstrations, and emergency command and dispatch. The demonstrator executes a multi-system synchronous demonstration using a pre-programmed system script, including voice announcements and page operations. When a viewer raises a question or request outside the script's predetermined scope, the demonstrator can interrupt the current demonstration using a hardware hotkey (such as the Alt key) or by speaking preset keywords (such as "stop," "wait," "pause," etc.), and the system enters an intelligent agent interaction mode. In this mode, the demonstrator can use text input or verbal input to instruct the intelligent agent to operate any page, query knowledge, or provide voice responses. After the interaction ends, the system automatically restores the state of all modified pages in the background, and the demonstrator can resume the original demonstration process with a single click.

[0078] See Figure 1 As shown, the present invention provides an intelligent agent digital employee demonstration control system that supports voice broadcasting, including a demonstration control layer, a page instance layer, an audio subsystem, an intelligent agent subsystem, and a background recovery subsystem.

[0079] The demonstration control layer is implemented based on a directed state machine and includes sequentially associated initialization nodes, demonstration broadcasting nodes, interruption handling nodes, agent interaction nodes, and state recovery nodes, which are used to schedule the system operation process and state transitions.

[0080] In this embodiment, the demonstration control layer is implemented based on a directed state graph framework and is responsible for scheduling the entire digital employee system's workflow. The functions of each node in the demonstration control layer are as follows:

[0081] The initialization node serves as the sole entry point during system startup. It loads the broadcast script configuration and multi-system page configuration, launches browser instances, creates independent tabs for each target system and navigates to the corresponding URL, pre-generates TTS audio files for all broadcast steps in batches, loads the speech keyword interruption dictionary, initializes the ASR speech recognition engine and continuously runs it in a background listening thread, and injects the domain knowledge base into the agent's dialogue context. If manual login is involved during the first run, the system will prompt the user to complete the login and save the authentication status for subsequent reuse.

[0082] The demo playback node, as the core execution node, iterates through the script steps sequentially. Each step switches the browser to the target system page and executes a preset page action (such as clicking or filling out a form), while simultaneously playing pre-generated TTS audio in non-blocking mode. During execution, it listens for system-level keyboard input via keyboard event hooks. Simultaneously, the ASR engine's background listening thread identifies the user's voice in real time and matches the results with a preset keyword set. Upon detecting a hotkey trigger or a matching voice keyword, audio playback is immediately stopped and an interruption status is marked, triggering a transition to the interruption handling module.

[0083] The interrupt handling node serves as a transition node after an interruption occurs. It stops audio playback, sets an interruption flag, records the source of the interruption, and ensures that all interruptions follow the same handling process. It saves a complete snapshot of the demonstration progress, including the current step index and a list of actions executed on each page. It starts the background recovery engine as an asynchronous task (this task is executed in parallel with subsequent agent interactions). It determines the currently active page as the initial operation target for the agent and initializes the agent toolset.

[0084] The agent interaction node, as a multi-turn dialogue loop node, is used to receive user text or voice input, supporting both text and voice input channels: voice input is converted into text in real time by the ASR engine and then routed to the agent; text input can be submitted directly; it is routed to the agent inference engine to execute the ReAct loop; the dialogue history and the currently active page are updated based on the agent's execution results; the agent's text reply is converted into voice broadcast; and it waits for the next round of user input in a loop until the user issues a command to continue broadcasting or exit.

[0085] The state recovery node, as the recovery node after the user issues the command to continue broadcasting, is used to wait for the background recovery task to complete, close the old page tabs that have been controlled by the agent, and replace them with the new page instance restored in the background; restore the list of executed actions and step indexes of each page to the snapshot value; and return to the demo broadcasting module to continue execution from the breakpoint.

[0086] The page instance layer is built on a browser automation framework to create independent browser tab instances for each target business system, with each tab maintaining its own independent running state. The page instance layer is the direct carrier for interaction between the system and the user, and is built on the Playwright automation framework. During the initialization phase, the system creates independent browser tabs for each target business system, with independent Page instances on the BrowserContext. Each tab maintains its own running state (page URL, DOM tree, authentication session, etc.) without interference. During the demonstration and broadcast phase, the system automatically switches the user's focus to the corresponding tab according to the script steps and executes preset operations, while simultaneously playing voice narration. The user can interrupt the demonstration at any time via buttons or voice commands. After the user interrupts, the system enters the intelligent agent interaction phase. The currently active page becomes the initial operation target of the intelligent agent. The user describes their operation intention in natural language or voice, such as "Help me open today's news" or "Check the provinces affected by the typhoon." The system transmits this intention to the intelligent agent subsystem for reasoning and page manipulation, and the operation results are fed back to the browser tab in real time, providing a WYSIWYG experience for the user. After the interaction ends, the system recreates a new tab instance for each manipulated page in the background and restores it to the presentation state before the interruption. It then seamlessly switches to the restored page using a hot-swap method, allowing the user to continue watching the original presentation from where they left off. Through this design, the page instance layer acts as a bridge for human-computer interaction in the closed loop of "presentation playback - user interruption - agent control - background restoration - continuation of playback," enabling users to seamlessly switch from passive viewing to active control and back to passive viewing in a multi-system page environment in the most natural way.

[0087] In this embodiment, the page instance layer is built based on the Playwright framework, and each tab maintains an independent page URL, DOM tree and authentication session; during the demonstration and broadcasting phase, tabs are automatically switched and operations are executed according to the script; during the intelligent agent interaction phase, the control results are fed back in real time; after the interaction ends, the page instance is switched to the restored page instance in a hot-swap manner.

[0088] See Figure 2 As shown, the audio subsystem includes a pre-generated TTS engine, a real-time TTS engine, a unified player, an ASR speech recognition engine, and an interruption monitoring module, used for unified management of speech output, speech input recognition, and interruption monitoring. In this embodiment, the audio subsystem is responsible for unified management of all speech output for demonstration broadcasts and agent responses, as well as user speech input recognition and interruption monitoring, avoiding channel conflicts through a single audio channel. Within the audio subsystem:

[0089] The pre-generated TTS engine is used to batch convert the broadcast text of all demonstration steps into audio files and cache them locally during the initialization phase. During initialization, the pre-generated TTS engine also batches and concurrently converts all steps with broadcast text in the demonstration script into MP3 audio files using a Text-to-Speech (TTS) service, storing them in the local audio cache directory. During broadcast, the audio is loaded and played directly according to the step index, without waiting for online generation, ensuring zero latency in the demonstration broadcast.

[0090] The real-time TTS engine is used to convert the agent's response text into audio in real time during the agent interaction phase; during the agent interaction phase, the real-time TTS engine converts the agent's response text into audio files in real time and plays them. Since the response text is unpredictable, a strategy of generating one response in real time is adopted.

[0091] The unified player provides a unified playback control interface for both pre-generated and real-time TTS, operating in non-blocking mode. Based on pygame.mixer, the unified player provides unified play and stop function interfaces for both pre-generated and real-time TTS. The demonstration broadcast module and the intelligent agent interaction module share the same player instance, achieving synchronized audio playback and action execution in non-blocking mode.

[0092] The ASR speech recognition engine runs continuously in a background thread, used for speech keyword recognition during the demonstration and broadcasting phase and speech input to text conversion during the agent interaction phase. It starts during the initialization phase and runs continuously as an independent background thread. It is responsible for two core tasks: (a) during the demonstration and broadcasting phase, it recognizes user speech in real time and matches it with a preset set of keywords, triggering an interrupt signal upon a match; (b) during the agent interaction phase, when the user selects the speech input mode, it receives the user's speech, extracts valid segments through endpoint detection, submits the recognized text for recognition, and returns the recognized text to the agent interaction module. The ASR engine supports both edge deployment and cloud API calls to meet the real-time and privacy requirements of different deployment environments.

[0093] The interruption monitoring module implements dual monitoring based on the keyboard channel and the voice channel. The keyboard channel monitors preset hotkeys through keyboard event hooks, while the voice channel matches preset keywords with ASR recognition results. When either channel is triggered, a thread-safe event flag stops the current audio playback. The interruption monitoring module is implemented based on a dual monitoring mechanism: (a) Keyboard channel: monitors system-level Alt key press events based on keyboard event hooks; (b) Voice channel: the ASR engine background thread continuously recognizes the real-time voice stream input from the microphone and performs fuzzy matching of the recognition results with a preset keyword set. When either channel detects a trigger condition, a thread-safe event flag (threading.Event) notifies the playback loop to stop the current playback. This interruption mechanism applies to both pre-generated and real-time generated voice, ensuring that both voice sources can be interrupted by either key presses or voice commands.

[0094] See Figure 3 As shown, the intelligent agent subsystem is built based on the ReAct pattern and includes a PageAgent inference engine and a set of callable tools. It is used to complete page manipulation and knowledge response based on the user's natural language instructions. Each user interaction compiles a brand new inference subgraph and is equipped with six types of callable tools.

[0095] The PageAgent inference engine is based on a LangGraph ReAct pattern inference subgraph. A completely new inference graph is compiled each time a user command is triggered: agent nodes invoke a large language model bound to a toolset for inference and generate tool invocation decisions; tool nodes execute the corresponding tools and return results; conditional edges determine whether to continue invoking tools or end inference. Dialogue history is maintained at the main state machine level, ensuring the continuity of multi-turn dialogues; compiling a new graph each time avoids state accumulation biases and page reference expiration issues in long-term sessions. In this embodiment, the PageAgent inference engine is built based on the LangGraph ReAct pattern, compiling a completely new inference subgraph each time a user command is triggered; dialogue history is maintained at the main state machine level. The constraint on the number of reasoning steps for each interaction is:

[0096] ;

[0097] in, The maximum number of inference steps is preset, with a coefficient of 3 to accommodate the three graph nodes for each step, and a constant of 10 as a safety margin.

[0098] In this embodiment, the toolset of the intelligent agent subsystem includes an accessibility tree snapshot tool, a page operation execution tool, a multi-system switching tool, a voice broadcast tool, and a knowledge retrieval tool;

[0099] The accessibility tree snapshot tool is used to obtain the complete accessibility tree of a page through the Chrome DevTools Protocol (CDP, Chrome Developer Tools Protocol) accessibility tree interface, recursively traverse the node tree, assign sequential reference tags [ref=eXX] to all interactive elements, where XX is an auto-incrementing number, and convert the page into an indented hierarchical structured text representation; at the same time, it constructs reference tags to element attributes. The mapping table.

[0100] The page operation execution tool is used to parse the operation commands output by the Large Language Model (LLM), locate the target element through reference tags, and then use a multi-strategy locator to backtrack and locate the DOM element step by step and execute the corresponding page operation. After the operation, the page snapshot is automatically refreshed. The page operation execution tool parses the operation commands output by the Large Language Model (LLM) (such as click e5, fill e7 "search term", goto https: / / ..., scroll, press Enter, hover e12, go-back), searches for the target element in the mapping table through reference tags, and uses a multi-strategy locator (exact text matching, fuzzy text matching, ARIA roles and names, roles only, CSS attribute selectors) to backtrack and locate the DOM element step by step. The corresponding click, fill, navigation, scroll, button, hover, and other operations are executed through the Playwright tool. After execution, the page snapshot is automatically refreshed, returning a new page structure description, forming an information loop of observation, operation, and re-observation.

[0101] The multi-system switching tool is used to switch the browser focus to the target system tab according to the instruction, and automatically update the page snapshot and reference mapping. The multi-system switching tool receives the system identifier, switches the browser focus to the tab of the corresponding system through bring_to_front, automatically obtains the accessibility tree snapshot of the new page and updates the page reference, so that the large language model can understand the user's natural language instructions (such as "help me switch to the integrated situation system") and complete the system switching autonomously.

[0102] The voice broadcasting tool is used to convert the agent's response into speech via real-time TTS and play it, and then play it in non-blocking mode through a unified player.

[0103] The knowledge retrieval tool is used to retrieve relevant knowledge slices from the domain knowledge base for the large language model to organize responses. It searches and queries the most relevant knowledge slices in the domain knowledge base based on keyword matching, sorts them by relevance score, and returns the top few slices with the highest scores for the large language model to organize responses.

[0104] The aforementioned toolset is implemented using the closure factory pattern. Each tool shares the accessibility tree snapshot text and reference mapping table through a mutable dictionary within the closure. The multi-system switching tool dynamically changes the page references shared by all tools using the nonlocal keyword, so that subsequent operations automatically apply to the new page.

[0105] See Figure 4 As shown, the background recovery subsystem includes an action classifier, a parallel recovery engine, and a page locator, used to asynchronously and in parallel restore the demonstration state of the modified page during agent interaction.

[0106] In this embodiment, the background recovery subsystem is responsible for restoring the page state modified by the intelligent agent in parallel as an asynchronous task during user-agent interaction. The background recovery subsystem includes:

[0107] The action classifier is used to pre-classify script actions into destructive actions with irreversible server-side effects and non-destructive actions such as pure UI navigation. Specifically, the action classifier pre-labels all actions defined in the script according to their effects. Operations with irreversible server-side effects, such as "creating a new session," "submitting a query to the server," "initiating a data request," and "sending a message," are marked as destructive actions; while pure UI navigation operations, such as "navigation clicks," "tab switching," and "filling in filter conditions," are marked as non-destructive actions.

[0108] The parallel recovery engine is used to asynchronously create new tabs during agent interaction, replaying only non-destructive actions according to the list of executed actions in the snapshot, with recovery tasks for multiple pages executed concurrently. Specifically, the parallel recovery engine is started in the interruption processing node by creating asynchronous tasks and runs concurrently with the agent interaction. It iterates through the list of executed action indices for each system page in the snapshot. For each page with an executed action record: a new tab is created in the browser context, navigation is performed to the system's base URL, and the executed actions are replayed one by one in index order, skipping actions marked as destructive. Recovery tasks for multiple pages are executed concurrently through asynchronous tasks, generating a reconstructed collection of page objects upon completion.

[0109] The page locator is used to construct a locator based on the text description, CSS selector, or ARIA role information in the page configuration, and to perform the recovery operation on the new page. During the page locator recovery process, the Playwright locator is directly constructed from the text description, CSS selector, or ARIA role information in the page configuration, bypassing the accessibility tree dependency and ensuring the executability of the recovery operation on the new page.

[0110] After the status is restored, the page Action replay collection satisfy: In the formula, To interrupt the page List of executed action indexes For the first The name of the step. For the action classification indicator function, when The value is 1 if it belongs to the set of destructive actions, and 0 otherwise. That is, only actions belonging to the page are replayed. Non-destructive actions from the list of executed actions.

[0111] See Figure 5 As shown, the present invention also provides a demonstration control method for intelligent digital employees that supports voice broadcasting, comprising the following stages:

[0112] S1. Initialization: Parse configuration files and demo scripts, load domain knowledge base and interruption keyword library, initialize ASR engine and start background listening, batch generate pre-generated TTS audio, start browser and create independent tabs for each target system; among them, the pre-generated TTS audio is generated asynchronously in batches and stored in local cache; the browser instance can reuse local user data directory or pre-store authentication status, and save the authentication status after the first login for subsequent reuse.

[0113] During initialization, after system startup, the initialization node performs the following operations: Parses the configuration file to obtain the URLs of multiple system pages, page element locator configurations, and predefined actions; parses the script document to obtain the demonstration step sequence (each step includes the broadcast text, target system page, and action name); loads the full text of the domain knowledge base; injects the knowledge base into the dialogue history of the large language model as the system context; loads the speech interruption keyword dictionary, pre-setting several keywords and their synonyms; initializes the ASR speech recognition engine and runs it continuously in a background thread; batches asynchronously generate pre-generated TTS audio files corresponding to all script texts; starts the Playwright browser (reusing the local Chrome persistent user data directory or injecting authentication status), creates an independent browser tab for each system, and navigates to the corresponding URL.

[0114] S2. Demonstration and Broadcast: Executes page operations sequentially according to the script and plays the corresponding pre-generated audio, while real-time monitoring for keyboard hotkeys and voice keyword interruptions. If no interruption is detected, the current step is completed and the next step is initiated; if an interruption is detected, the broadcast is interrupted. The period for monitoring interruption signals is 50 milliseconds; upon detection, audio playback immediately stops and the broadcast loop is interrupted. After each page operation is executed, the corresponding step index is appended to the list of executed actions on the target page.

[0115] In this demo broadcast, after initialization, the demo broadcast node is entered, and each step is executed in a loop according to the script index order: (a) If the current step contains a page action, switch to the system page tab corresponding to the action, execute the Playwright page operation according to the action definition in the configuration file (such as clicking a specified element or entering specified text in an input box), and append the current step index to the list of executed actions of the page. (a) If the current step contains text to be played, load the corresponding pre-generated TTS audio file and play it in non-blocking mode, while entering a listening loop: check for interruption instructions approximately every 50 milliseconds. If the audio plays normally, advance the index to the next step; if an interruption instruction is detected, immediately stop audio playback and interrupt the playback loop.

[0116] S3. Interruption and Snapshot: After interruption detection is triggered, the interruption handling node is entered. Audio playback is stopped, an interruption flag is set, and the source of the interruption is recorded; a state snapshot containing the current script step index and a list of actions executed on each page is saved; the currently active page is identified, and a background page recovery task is asynchronously started. This task runs in parallel during agent interaction and does not block the interaction flow; the saved state snapshot... satisfy: In the formula, This is an index of the script steps at the time of interruption. For the first A list of executed actions indexes for each page. The total number of system pages is used to enable background recovery tasks and agent interactions to run in parallel without blocking the interaction process.

[0117] S4. Agent Interaction: Receives user text or voice commands, invokes the agent to execute the ReAct inference loop to complete page manipulation or knowledge retrieval, converts the response into speech for playback, and loops waiting for user input. If an exit command is received, the system terminates; if a resume command is received, it proceeds to the next stage. The specific process of the ReAct inference loop is as follows: obtain a snapshot of the accessibility tree of the current page; input system prompts, dialogue history, page snapshot, and user commands into the large language model; the large language model generates a decision-making tool; executes the corresponding tool and obtains a new page snapshot; iterates the inference until the final text response is generated. Voice commands are recognized as text in real-time by the ASR engine and submitted to the agent; the agent's text response is converted into speech in real-time by TTS and then played.

[0118] In this agent interaction, after the agent interaction module takes over control, it waits for user input in a loop, supporting both text and voice input channels: (a) If the user inputs "continue broadcasting" (or an equivalent command), it proceeds to stage five; the user can input this command via text or verbally utter keywords that are recognized by the ASR engine to trigger the same logic; (b) If the user inputs "exit," the system operation terminates; (c) If it is another natural language command, the user can input text via keyboard or speak via microphone, which is then converted into text in real time by the ASR engine and submitted. The system calls the PageAgent agent to execute the ReAct inference loop: obtaining a snapshot of the accessibility tree of the current page, inputting system prompts, dialogue history, page snapshots, and user commands as context into the large language model, the large language model decides whether to call a tool and which tool to call, executes the tool and obtains a new snapshot as feedback, continues inference until the final text reply is generated, converts the reply text into speech via real-time TTS and plays it, and returns to the loop to wait for the next round of input. During the agent interaction, the background recovery task and user interaction are executed in parallel, and the user is unaware of this.

[0119] S5. Recovery and Resumption: After the background recovery task is completed, the manipulated old page is replaced with the restored new page, the demo progress snapshot is restored, and execution resumes from the breakpoint at the demo playback stage. Specifically, page recovery is performed as follows: a new tab is created for each manipulated page, navigating to the system base URL, and non-destructive actions are replayed according to the list of executed actions; recovery tasks for multiple pages are executed concurrently.

[0120] In this recovery and resume playback process, after the user issues a resume playback command, the state recovery module is entered: (a) waiting for the background recovery task to complete, which is usually completed during the interaction with the agent; (b) for each page with a record of executed actions, obtaining the page instance reconstructed by the background recovery engine, closing the old tabs manipulated by the agent, and replacing them with the recovered new tab instance; (c) restoring the list of executed actions for each page to the snapshot value. Reset the script walking index to (d) Clear the snapshot and restore the relevant state, return to the demo broadcast loop of stage two, and continue execution from the breakpoint.

[0121] In this embodiment, the interruption signal includes two types: keyboard hotkey trigger and voice keyword hit. Both interruption methods apply to pre-generated speech and real-time generated speech.

[0122] Verified by actual testing:

[0123] (1) Key interruption response time < 100 milliseconds (hardware hotkey trigger, audio stop).

[0124] (2) Voice keyword interruption response time <500 milliseconds (after the user finishes speaking the keyword, ASR recognition, matching hit and audio stops).

[0125] (3) Total time for backend page recovery: The recovery of two pages concurrently takes about 1.2 seconds, which is in parallel with user interaction and is imperceptible to the user.

[0126] (4) Success rate of page operation based on barrier-free tree snapshot of intelligent agent: more than 95% on standard web pages.

[0127] (5) Multi-system switching latency <500 milliseconds (tab switching and snapshot reconstruction).

[0128] This invention provides a digital employee demonstration control system and method that supports voice broadcasting. For the first time, it integrates barrier-free tree-driven LLM page operation, voice broadcasting and multimodal interruption of the demonstration process, multi-turn free interaction of the intelligent agent (including voice input), background parallel page recovery based on action classification, and unified audio channel management (including ASR) into a complete closed-loop system. It realizes fully automated control of the entire process, including broadcasting, button or voice interruption, multimodal takeover of the intelligent agent, parallel recovery, and seamless continuation.

[0129] This invention provides a digital employee demonstration control system and method supporting voice broadcasting, significantly improving demonstration flexibility and user experience. Compared with existing one-way, uninterrupted scripted demonstration systems, this invention allows users to interrupt the demonstration at any time via hardware hotkeys or natural voice commands. Users can instruct the agent to manipulate the page, query knowledge, and obtain voice responses via text or voice. After the interaction is completed, the user can resume from the interruption point with one click. The background parallel recovery mechanism ensures that the page is ready when the user's interaction ends, achieving a seamless transition without waiting. Pre-generated TTS and real-time TTS share a unified audio channel, avoiding channel conflicts and interruption failures.

[0130] This invention provides a digital employee demonstration control system and method supporting voice broadcasting, enabling efficient, framework-independent large language model page manipulation. Compared to existing solutions that rely on visual model screenshots or fixed selectors, this invention constructs structured page text with reference tags through the Chrome DevTools Protocol's accessibility tree interface, allowing the large language model to precisely manipulate page elements without needing to understand DOM structure or CSS rules. The accessibility tree, as a standard Web interface, possesses cross-frontend framework universality, and its compact and efficient text format significantly reduces the context consumption of the large language model. The step-by-step fallback mechanism of the multi-strategy locator further improves the success rate of operations.

[0131] This invention provides a digital employee demonstration control system and method for intelligent agents that supports voice broadcasting, offering unified control capabilities across multiple systems. Compared to existing single-system operation schemes, this invention, through a tool factory pattern with closure binding, allows intelligent agents to freely switch between multiple independent web systems, with all tools dynamically bound to the currently active page. System switching does not require re-initializing the intelligent agent or interrupting the dialogue context; each system page maintains an independent operating state, achieving true unified control across multiple systems.

[0132] This invention provides a digital employee demonstration control system and method supporting voice broadcasting, solving the problem of restoring page consistency after intelligent agent operation. Compared with existing solutions that require manual page restoration or abandoning restoration, this invention proposes a parallel background restoration method based on the classification of action effects. Script actions are divided into destructive and non-destructive categories. During user interaction, a new tab is created in parallel in the background, and only non-destructive actions are replayed to restore the page. The restoration process is executed concurrently with user interaction, and the user-perceived waiting time is zero.

[0133] This invention provides a digital employee demonstration control system and method supporting voice broadcasting, ensuring the stability of inference in multi-turn dialogues. Compared with existing schemes that maintain a single inference graph for long conversations, this invention adopts an architecture that compiles a new inference graph for each interaction, avoiding the accumulation of state deviations in long dialogues. Simultaneously, it maintains the dialogue history at the main state machine level to ensure the continuity of multi-turn dialogues. The two-layer cascaded state machine architecture allows both the demonstration broadcasting and agent interaction modes to share message context and page resources, making state transitions clear and traceable.

[0134] This invention provides a digital employee demonstration control system and method supporting voice broadcasting, achieving multimodal interaction to reduce the operational threshold. Compared with existing demonstration systems that only support button operation and text input, this invention introduces an ASR speech recognition engine to enable voice keyword interruption of broadcasting and voice input of commands. During the demonstration, users can interrupt the broadcasting with natural speech without operating the keyboard. In the intelligent agent interaction stage, users can completely operate the page and ask questions by voice without the keyboard, significantly reducing the operational threshold. It is especially suitable for occasions such as exhibition halls and press conferences where presenters need to balance body language and verbal expression, as well as barrier-free operation scenarios for users with disabilities.

[0135] The foregoing description of specific exemplary embodiments of the invention is for illustrative and explanatory purposes. These descriptions are not intended to limit the invention to the precise forms disclosed, and it will be apparent that many changes and variations can be made in accordance with the foregoing teachings. The exemplary embodiments were chosen and described in order to explain the specific principles of the invention and its practical application, thereby enabling those skilled in the art to implement and utilize various different exemplary embodiments of the invention, as well as various different choices and variations. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A digital employee demonstration control system for intelligent agents that supports voice broadcasting, characterized in that, It includes a presentation control layer, a page instance layer, an audio subsystem, an intelligent agent subsystem, and a background recovery subsystem; The demonstration control layer is implemented based on a directed state machine and includes sequentially associated initialization nodes, demonstration broadcasting nodes, interruption handling nodes, intelligent agent interaction nodes, and state recovery nodes, which are used to schedule the system operation process and state transitions. The page instance layer is built on a browser automation framework and is used to create an independent browser tab instance for each target business system. Each tab maintains an independent running state. The audio subsystem includes a pre-generated TTS engine, a real-time TTS engine, a unified player, an ASR speech recognition engine, and an interruption monitoring module, which are used to uniformly manage speech output, speech input recognition, and interruption monitoring. The intelligent agent subsystem is built on the ReAct model and includes the PageAgent inference engine and a callable toolset, which is used to complete page manipulation and knowledge response based on the user's natural language commands. The background recovery subsystem includes an action classifier, a parallel recovery engine, and a page locator, which are used to asynchronously and in parallel restore the demonstration state of the modified page during agent interaction.

2. The intelligent agent digital employee demonstration control system supporting voice broadcasting according to claim 1, characterized in that, The functions of each node in the demonstration control layer are as follows: Initialize nodes to load broadcast script configuration and multi-system page configuration, start browser and create independent tabs for each system, batch pre-generate TTS audio files for all broadcast steps, load speech keyword interruption dictionary, initialize ASR speech recognition engine and start background monitoring, and inject domain knowledge base into the intelligent agent dialogue context. The demonstration broadcast node is used to iterate through the script steps in sequence, switch to the target system page and execute preset page actions, play pre-generated TTS voice synchronously, and listen for interruption signals from keyboard hotkeys and voice keywords. When an interruption is detected, a state transition is triggered. The interrupt processing node is used to stop audio playback and set an interruption flag, save a snapshot of the presentation progress containing the current step index and a list of actions executed on each page, asynchronously start the background recovery engine, determine the currently active page, and initialize the intelligent agent toolset; The agent interaction node is used to receive user text or voice input, route it to the agent inference engine to execute the ReAct loop, update the dialogue history and active page state, convert the agent's reply into voice broadcast, and wait for user input in a loop until it receives a command to continue or exit. The state recovery node is used to wait for the background recovery task to complete, replace the old page controlled by the agent with the new page instance restored in the background, restore the demo progress snapshot, and return to the demo broadcast node to continue execution from the breakpoint.

3. The intelligent agent digital employee demonstration control system supporting voice broadcasting according to claim 1, characterized in that, In the audio subsystem: The pre-generated TTS engine is used to batch convert the broadcast text of all demonstration steps into audio files and cache them locally during the initialization phase. The real-time TTS engine is used to convert the agent's reply text into audio in real time during the agent interaction phase; The unified player provides a unified playback control interface for pre-generated TTS and real-time TTS, and runs in non-blocking mode; The ASR speech recognition engine runs continuously in the background thread and is used to demonstrate speech keyword recognition in the broadcasting stage and speech input to text conversion in the agent interaction stage. The interruption monitoring module implements dual monitoring based on the keyboard channel and the voice channel. The keyboard channel monitors preset hotkeys through keyboard event hooks, and the voice channel matches preset keywords through ASR recognition results. When either channel is triggered, the current audio playback is stopped through a thread-safe event flag.

4. The intelligent agent digital employee demonstration control system supporting voice broadcasting according to claim 1, characterized in that, The toolset of the intelligent agent subsystem includes an accessibility tree snapshot tool, a page operation execution tool, a multi-system switching tool, a voice broadcast tool, and a knowledge retrieval tool; The accessibility tree snapshot tool is used to obtain the complete accessibility tree of the page through the accessibility tree interface of the Chrome DevTools Protocol, assign reference tags to interactive elements and build an element attribute mapping table to generate structured page text; The page operation execution tool is used to parse the operation commands output by the large language model, locate the target element by referencing the tag, locate the DOM element by step back through the multi-strategy locator and execute the corresponding page operation, and automatically refresh the page snapshot after the operation. The multi-system switching tool is used to switch the browser focus to the target system tab according to instructions and automatically update the page snapshot and reference mapping. The voice broadcasting tool is used to convert the agent's response into voice via real-time TTS and play it. The knowledge retrieval tool is used to retrieve relevant knowledge slices from the domain knowledge base for the large language model to organize and respond to.

5. The intelligent agent digital employee demonstration control system supporting voice broadcasting according to claim 4, characterized in that, The PageAgent inference engine is built on the LangGraph ReAct pattern, compiling a completely new inference subgraph each time a user command is triggered; the dialogue history is maintained at the main state machine level. The constraint on the number of reasoning steps for each interaction is: ； in, The maximum number of inference steps is preset, with a coefficient of 3 to accommodate the three graph nodes for each step, and a constant of 10 as a safety margin.

6. The intelligent agent digital employee demonstration control system supporting voice broadcasting according to claim 4, characterized in that, The toolset is implemented using the closure factory pattern. Each tool shares the accessibility tree snapshot text and reference mapping table through a variable dictionary within the closure. The multi-system switching tool dynamically changes the page references shared by all tools through the nonlocal keyword, so that subsequent operations are automatically applied to the new page.

7. The intelligent agent digital employee demonstration control system supporting voice broadcasting according to claim 1, characterized in that, In the background recovery subsystem: The action classifier is used to preclassify script actions into destructive actions with irreversible server-side effects and non-destructive actions of pure UI navigation. The parallel recovery engine is used to asynchronously create new tabs during agent interaction, replay only non-destructive actions according to the list of executed actions in the snapshot, and execute the recovery tasks of multiple pages concurrently. The page locator is used to build a locator based on text descriptions, CSS selectors, or ARIA role information in the page configuration, and to perform a restore operation on a brand new page.

8. The intelligent agent digital employee demonstration control system supporting voice broadcasting according to claim 7, characterized in that, After the status is restored, the page Action replay collection satisfy: ； In the formula, To interrupt the page List of executed action indexes For the first The name of the step. For the action classification indicator function, when The value is 1 if it belongs to the set of destructive actions, and 0 otherwise.

9. A method for controlling a digital employee demonstration with voice broadcast support, executed based on the digital employee demonstration control system with voice broadcast support according to any one of claims 1-8, characterized in that, The demonstration control method includes the following steps: S1. Initialization: Parse configuration files and demo scripts, load domain knowledge base and interruption keyword library, initialize ASR engine and start background listening, batch generate pre-generated TTS audio, start browser and create independent tabs for each target system; S2. Demonstration and Broadcast: Execute page operations in the order of the script and play the corresponding pre-generated voice, and monitor keyboard hotkeys and voice keywords for interruption signals in real time; if no interruption is detected, the current step is completed and the next step is performed; if an interruption is detected, the broadcast is interrupted. S3. Interruption and Snapshot: Stop audio playback, save a state snapshot containing the current script step index and a list of actions executed on each page, determine the currently active page, and asynchronously start the background page recovery task; S4. Intelligent Agent Interaction: Receives user text or voice commands, calls the intelligent agent to execute the ReAct inference loop to complete page manipulation or knowledge query, converts the reply into voice playback, and waits for user input in a loop; if an exit command is received, the system terminates; if a resume command is received, it enters the next stage. S5. Restore and Resume Playback: Wait for the background restore task to complete, replace the old page that was manipulated with the new page after restoration, restore the demo progress snapshot, and return to the demo playback stage to continue execution from the breakpoint.

10. The intelligent agent digital employee demonstration control method supporting voice broadcasting according to claim 9, characterized in that, In step S3, a snapshot of the state is saved. satisfy: ； In the formula, This is an index of the script steps at the time of interruption. For the first A list of executed actions indexes for each page. This represents the total number of pages in the system.

Images