Real-time digital human double-channel question answering and automatic scheduling operation page execution system and method

Abstract

The invention relates to a real-time digital human double-channel question answering and automatic scheduling operation page execution system and method, and the system comprises an instruction routing center which is used for receiving a natural language instruction of a user and classifying the instruction to an operation channel or a question and answer channel; the operation channel module is used for disassembling an operation type instruction into a Playwright automatic script through an instruction analysis engine and driving and executing the Playwright automatic script; the question and answer channel module is used for generating technical principle instructions by combining LLM intention recognition and knowledge base retrieval; and the multi-thread synchronous controller is used for dynamically calibrating operation execution and digital human voice explanation. The method comprises the steps of dual-channel instruction processing, atomic operation disassembly and cross-modal time sequence synchronization. The defect that traditional RPA is only operated but not explained is overcome, and the information density of a training / demonstration scene is improved. No mature competitive product exists globally, and an operable digital human racing track core interaction technology is exclusively occupied. The method is suitable for government affair system training, system demonstration and software operation guidance high-value scenes.

Description

Technical Field

[0001] This invention relates to the intersection of artificial intelligence and automation technologies, specifically to a real-time digital human dual-channel Q&A and automatic scheduling operation page execution system and method. Background Technology

[0002] In the intersection of artificial intelligence and automation technologies, the current technology suffers from the following shortcomings: Operation and explanation are disconnected: Traditional RPA systems can only execute interface operations according to preset scripts, and cannot explain the content in real time or respond to users' questions about the principles in real time. Severe multimodal latency: When existing digital human systems concurrently perform voice narration and interface operations, the audio-visual deviation exceeds 500ms due to thread scheduling defects. For example, when the digital human narrates "click the submit button", the interface operation has already jumped to the next page, causing cognitive confusion. Limitations in instruction comprehension: A single NLP engine has insufficient accuracy in recognizing compound instructions (such as "demonstrate the operation and explain the reason") and cannot distinguish between operational actions and knowledge query needs.

[0003] Therefore, it is necessary to provide a real-time digital human dual-channel Q&A and automatic scheduling operation page execution system and method to solve the above problems. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time digital human dual-channel Q&A and automatic scheduling operation page execution system and method. The technical solution is as follows: A real-time digital human dual-channel Q&A and automatic scheduling operation page execution system includes: Command routing center: configured to receive natural language commands from user terminals and distinguish between operation commands and question-and-answer commands based on a pre-trained classification model; Operation channel module: includes an instruction parsing engine and a Playwright executor. The instruction parsing engine breaks down operation-type instructions into atomic operation steps and generates browser automation scripts. The Playwright executor drives the target web application interface to perform operations. Question and Answer Channel Module: Includes LLM engine and knowledge base. The LLM engine analyzes the user's potential question intent and combines the knowledge base search results to generate technical principle voice content. Multi-threaded synchronous controller: Real-time monitoring of the interface operation completion signal of the operation channel module and the voice output signal of the question and answer channel module, dynamically adjusting the voice start timing of the digital human rendering engine to ensure that the interface operation completion time T1 and the corresponding voice start time T2 satisfy |T1-T2|<300ms.

[0005] in: The instruction parsing engine uses a BERT-based sequence labeling model to map user instructions to a predefined set of atomic operations, which includes page navigation, form filling, button clicking, data validation, highlighting, page opening, and video opening operations.

[0006] The synchronization controller implements timing calibration through an event-driven architecture, specifically including: Listening module: Captures the DOM update completion event of the Playwright executor and the speech synthesis readiness event of the digital human rendering engine; Delay calculation module: Dynamically calculates the voice output delay compensation value based on the event timestamp; Priority scheduler: When an operation event conflicts with a voice event, it forces a delay in the execution of non-critical operation steps.

[0007] The second technical solution is as follows: A method for executing a real-time digital human dual-channel Q&A and automatic scheduling operation page includes the following steps: 1) Instruction classification steps: The instruction routing center classifies user-input instructions such as "demonstration refund approval" into two categories. If the instruction contains interface operation keywords, it is routed to the operation channel. If the instruction contains question words such as "why" or "principle", it is routed to the question and answer channel. 2) Operation breakdown steps: The instruction "Open the tax declaration system and fill in the form" is broken down into: S1. Launch the browser instance → S2. Navigate to the specified URL → S3. Locate the ID number input box → S4. Inject test data → S5. Submit the form; 3) Dual-channel response steps: Operation channel: After generating the Playwright script, execute steps S1-S5. After step S3 is completed, trigger the element location success event. Q&A Channel: When an S3 step event is detected, the LLM engine is activated to generate voice content on "ID Number Verification Rules"; 4) Timing synchronization step: Within 300ms after the interface element in step S3 is highlighted, start the digital human to narrate the corresponding voice content.

[0008] in: The timing synchronization step employs an adaptive audio speed-changing algorithm, using the DOM update event of the operation channel as a reference point to segment and accelerate or decelerate the audio stream of the question-and-answer channel.

[0009] When a user adds "Why is it necessary to upload a business license?", the Q&A module interrupts the current explanation flow and prioritizes calling the relevant clauses of the "XXX Management Regulations" in the knowledge base to generate an immediate response.

[0010] Compared with existing technologies, this invention solves three major technical defects of traditional RPA systems in training / demonstration scenarios: the separation of operation and explanation, severe multimodal latency, and limited understanding of compound instructions; it achieves a breakthrough in realizing the "operable digital human" interaction mode, improving information density in scenarios such as government system training; it overcomes the problem of concurrent latency in multimodal tasks, compressing the audio-visual deviation from the industry average of 500ms to within 300ms; and it supports the precise splitting and parallel response of compound instructions. Attached Figure Description

[0011] Figure 1 This is the system architecture diagram of the present invention.

[0012] Figure 2 This is a flowchart of the operation instructions breakdown of the present invention. Detailed Implementation Example

[0013] See Figure 1 This embodiment demonstrates a real-time digital human dual-channel Q&A and automatic scheduling operation page execution system. The system architecture includes the following modules: ● The user issues commands via voice, and the system collects these commands using a voice model. ●Command Routing Center: Configured to receive natural language commands from user terminals and distinguish between operation commands and question-and-answer commands based on a pre-trained classification model; ● Operation Channel Module: Includes instruction parsing engine and Playwright executor. The instruction parsing engine breaks down operation-type instructions into atomic operation steps and generates browser automation scripts. The Playwright executor drives the target web application interface to perform operations. ● Question and Answer Channel Module: Includes LLM engine and knowledge base. The LLM engine analyzes the user's potential question intent and combines the knowledge base search results to generate technical principle voice content. ● Multi-threaded synchronous controller: Real-time monitoring of the interface operation completion signal of the operation channel module and the voice output signal of the question and answer channel module, dynamically adjusting the voice start timing of the digital human rendering engine to ensure that the interface operation completion time T1 and the corresponding voice start time T2 satisfy |T1-T2|<300ms.

[0014] The synchronization controller implements timing calibration through an event-driven architecture, specifically including: ● Listening module: Captures DOM update completion events of the Playwright executor and speech synthesis readiness events of the digital human rendering engine; ●Delay calculation module: Dynamically calculates the voice output delay compensation value based on the event timestamp; ●Priority Scheduler: When an operation event conflicts with a voice event, it forces a delay in the execution of non-critical operation steps.

[0015] like Figure 2 As shown, the real-time digital human dual-channel Q&A and automatic scheduling operation page execution method provided by the present invention includes the following modules in its system architecture: including the following steps: ● Instruction classification steps: The instruction routing center classifies user-input instructions such as "demonstrate order creation" into two categories. If the instruction contains interface operation keywords, it is routed to the operation channel. If the instruction contains question words such as "why" or "principle", it is routed to the question and answer channel. ● Operation breakdown steps: For example, the instruction "Open the tax declaration system and fill in the form" can be broken down as follows: S1. Launch the browser instance → S2. Navigate to the specified URL → S3. Locate the ID number input box → S4. Inject test data → S5. Submit the form; ● Dual-channel response steps: Operation channel: Generate Playwright script to execute steps S1-S5, and trigger the element location success event after step S3 is completed; Q&A Channel: When an S3 step event is detected, the LLM engine is activated to generate voice content on "ID Number Verification Rules"; ● Timing synchronization step: Within 300ms after the interface element is highlighted in step S3, start the digital human to narrate the corresponding voice content.

[0016] The timing synchronization step employs an adaptive audio speed-changing algorithm, using the DOM update event of the operation channel as a reference point to segment and accelerate or decelerate the audio stream of the question-and-answer channel.

[0017] For example, when a user adds "Why do I need to upload my business license?", the Q&A channel module (40) interrupts the current explanation flow and prioritizes calling the relevant clauses of the "XXX Management Regulations" in the knowledge base (42) to generate an immediate response. Application Example 1:

[0018] A park's cockpit explains the dispatching digital human platform.

[0019] This system was deployed within a smart park's dashboard system to synchronize large-screen operation guidance and data interpretation. User command: "Demonstrate this month's energy consumption analysis and explain the reasons for peak-valley differences." Operation channel response:

[0020] The instruction parsing engine (31) is broken down into atomic operations: ① Focus on the energy management module → ② Select the time dimension "this month" → ③ Generate a bar chart comparison → ④ Highlight the peak and valley periods. The large screen WEB system is controlled by the Playwright executor (32), which calls the ECharts API to render the visualization chart (takes 380ms). Q&A Channel Response:

[0021] When step ④ is detected to be completed, the LLM engine (41) combines the "Park Energy Conservation Regulation Specification" in the knowledge base (42) to generate an interpretation: "Peak electricity price is 40% higher than valley price. Due to the transformer load rate exceeding 75%, the capacity expansion plan needs to be activated" (voice length 2.1 seconds). Synchronization control:

[0022] The multi-threaded synchronous controller (50) detects that the chart highlighting is complete (T1=2180ms), and the delay calculation module (52) predicts the speech synthesis time and starts the digital human narration at T1+170ms. The actual deviation is 189ms (the interface peak and valley markers and the "red bars" description in the speech are precisely matched). Innovation Value:

[0023] Overcoming the challenge of synchronizing dynamic data focus and policy interpretation during large-screen presentations, training efficiency has increased fourfold. Real-world testing at XX Smart Park showed that the operation-presentation latency in LED screen scenarios is consistently within 300ms.

[0024] Application Example 2: A park's cockpit explains the dispatching digital human platform.

[0025] This system was deployed within a smart park's dashboard system to synchronize large-screen operation guidance and data interpretation. User command: "Demonstrate this month's energy consumption analysis and explain the reasons for peak-valley differences." Operation channel response:

[0026] The instruction parsing engine (31) is broken down into atomic operations: ① Focus on the energy management module → ② Select the time dimension "this month" → ③ Generate a bar chart comparison → ④ Highlight the peak and valley periods. The large screen WEB system is controlled by the Playwright executor (32), which calls the ECharts API to render the visualization chart (takes 380ms). Q&A Channel Response:

[0027] When step ④ is detected to be completed, the LLM engine (41) combines the "Park Energy Conservation Regulation Specification" in the knowledge base (42) to generate an interpretation: "Peak electricity price is 40% higher than valley price. Due to the transformer load rate exceeding 75%, the capacity expansion plan needs to be activated" (voice length 2.1 seconds). Synchronization control:

[0028] The multi-threaded synchronous controller (50) detects that the chart highlighting is complete (T1=2180ms), and the delay calculation module (52) predicts the speech synthesis time and starts the digital human narration at T1+170ms. The actual deviation is 189ms (the interface peak and valley markers and the "red bars" description in the speech are precisely matched). Innovation Value:

[0029] Overcoming the challenge of synchronizing dynamic data focus and policy interpretation during large-screen presentations, training efficiency has increased fourfold. Real-world testing at XX Smart Park showed that the operation-presentation latency in LED screen scenarios is consistently within 300ms.

[0030] Compared with existing technologies, this invention solves three major technical defects of traditional RPA systems in training / demonstration scenarios: the separation of operation and explanation, severe multimodal latency, and limited understanding of compound instructions; it achieves a breakthrough in realizing the "operable digital human" interaction mode, improving information density in scenarios such as government system training; it overcomes the problem of concurrent latency in multimodal tasks, compressing the audio-visual deviation from the industry average of 500ms to within 300ms; and it supports the precise splitting and parallel response of compound instructions.

[0031] For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of this invention, and these all fall within the protection scope of this invention.

Claims

1. A real-time digital human dual-channel Q&A and automatic scheduling operation page execution system, characterized in that... : include: Command routing center: configured to receive natural language commands from user terminals and distinguish between operation commands and question-and-answer commands based on a pre-trained classification model; Operation Channel Module: Includes instruction parsing engine and Playwright executor. The instruction parsing engine breaks down operation-type instructions into atomic operation steps and generates browser automation scripts. The Playwright executor drives the target web application interface to perform operations. Question and Answer Channel Module: Includes LLM engine and knowledge base. The LLM engine analyzes the user's potential question intent and combines the knowledge base search results to generate technical principle voice content; Multi-threaded synchronous controller: Real-time monitoring of the interface operation completion signal of the operation channel module and the voice output signal of the question and answer channel module, dynamically adjusting the voice start timing of the digital human rendering engine to ensure that the interface operation completion time T1 and the corresponding voice start time T2 satisfy |T1-T2|<300ms.

2. The system according to claim 1, a real-time digital human dual-channel Q&A and automatic scheduling operation page execution system, is characterized in that: The instruction parsing engine uses a BERT-based sequence labeling model to map user instructions to a predefined set of atomic operations, including page navigation, form filling, button clicking, data validation, highlighting, page opening, and video opening.

3. The real-time digital human dual-channel Q&A and automatic scheduling operation page execution system according to claim 1, characterized in that... The synchronization controller implements timing calibration through an event-driven architecture, specifically including: Listening module: Captures the DOM update completion event of the Playwright executor and the speech synthesis readiness event of the digital human rendering engine; Delay calculation module: Dynamically calculates the voice output delay compensation value based on the event timestamp; Priority scheduler: When an operation event conflicts with a voice event, it forces a delay in the execution of non-critical operation steps.

4. A method for executing a real-time digital human dual-channel Q&A and automatic scheduling operation page, characterized in that: Includes the following steps: 1) Instruction classification steps: For example, the instruction "demonstration refund approval" type of instruction entered by the user is classified into two categories through the instruction routing center. If it contains interface operation keywords, it is routed to the operation channel. If it contains question words such as "why" or "principle", it is routed to the question and answer channel. 2) Operation breakdown steps: The instruction "Open the tax declaration system and fill in the form" is broken down into: S1. Launch the browser instance → S2. Navigate to the specified URL → S3. Locate the ID number input box → S4. Inject test data → S5. Submit the form; 3) Dual-channel response steps: Operation channel: After generating the Playwright script, execute steps S1-S5. After step S3 is completed, trigger the element location success event. Q&A Channel: When an S3 step event is detected, the LLM engine is activated to generate voice content on "ID Number Verification Rules"; 4) Timing synchronization step: Within 300ms after the interface element in step S3 is highlighted, start the digital human to narrate the corresponding voice content.

5. The method for executing a real-time digital human dual-channel Q&A and automatic scheduling operation page according to claim 4, characterized in that: The timing synchronization step employs an adaptive audio speed-changing algorithm, using the DOM update event of the operation channel as a reference point to segment and accelerate or decelerate the audio stream of the question-and-answer channel.

6. The method for executing a real-time digital human dual-channel Q&A and automatic scheduling operation page according to claim 4, characterized in that: When a user adds "Why is it necessary to upload a business license?", the Q&A module interrupts the current explanation flow and prioritizes calling the relevant clauses of the "XXX Management Regulations" in the knowledge base to generate an immediate response.

Images