Automatic operation backtracking and digital person continuous talk control method and system oriented to multi-modal interaction

Abstract

The invention relates to a multi-modal interaction-oriented automatic operation backtracking and digital person continuous talking control method and system, and the system comprises an operation snapshot module which records operation nodes through a semantic anchor point marking technology, such as an interface XPath page number and a PPT page number; the interruption detection module is used for capturing a user interruption instruction in real time; the progress prediction engine dynamically adjusts a follow-up demonstration path based on a program; and the continuous talk controller is used for coordinating RPA operation execution, digital human rendering and voice synthesis through a three-level priority thread pool. The problems of inaccurate operation flow backtracking and multi-modal thread conflict after demonstration interruption are solved, the breakpoint continuous talk error is obviously lower than that of a traditional scheme, and the method is suitable for PPT demonstration, cockpit explanation, business system training, course explanation and 3D modeling guidance scenes.

Description

Technical Field

[0001] This invention relates to the fields of human-computer interaction, automation control, and digital human technology. Specifically, it relates to an automated operation backtracking and digital human continuation control method and system for multimodal interaction. Background Technology

[0002] The core challenge currently facing automated presentation and digital human explanation systems lies in handling interruptions. When users interrupt presentations via voice or other means, such as asking follow-up questions, existing systems will exhibit the following issues: Unable to revert to the previous operation state: After an interruption, it is difficult to accurately restore the operation interface state before the interruption, such as PPT page numbers, software operation steps, and 3D model perspective. The presentation is awkward: When resuming the presentation, the digital human can only start from the beginning of the presentation script or a fixed point, and cannot intelligently skip the content that has already been explained or adjust the subsequent path according to the context; Multi-task coordination is difficult: When handling user interruptions, it is necessary to pause and coordinate multiple high-concurrency tasks, such as RPA operation execution, digital human rendering, and speech synthesis and recognition, which can easily lead to system lag or inconsistent states.

[0003] Existing technologies lack an efficient and accurate mechanism to record the status of interruptions, predict the optimal progress after recovery, and coordinate the smooth recovery of multiple tasks; for example, when a question is asked during a PowerPoint presentation, the digital human may repeat the explanation of a covered chapter, while the RPA operation has already moved to the next page.

[0004] Therefore, it is necessary to provide an automated operation backtracking and digital human continuation control method and system for multimodal interaction to solve the above problems. Summary of the Invention

[0005] The purpose of this invention is to provide an automated operation backtracking and digital human continuation control method and system for multimodal interaction.

[0006] The technical solution is as follows: An automated operation replay and digital human continuation control method for multimodal interaction, characterized by the following steps: During the execution of automated operations, real-time snapshots of the status of key operation nodes are recorded. These snapshots contain precise environmental status information, including: 1) During the explanation of the digital human, semantic anchors are created at semantic key points. The semantic anchors include at least an anchor identifier, associated explanation semantic information, and associated operation node identifiers. 2) When a user interruption is detected, the automated operation is immediately paused and the interruption point marker at the current moment is recorded. The interruption point marker includes the latest state snapshot identifier and / or the most recent semantic anchor identifier. 3) Handle user interruption requests; 4) After the user interruption request processing is completed, based on the interruption point marker and the context information during the interruption processing, predict the target starting point for demonstration recovery. The target starting point corresponds to a target state snapshot and / or a target semantic anchor point. 5) Based on the stated target starting point: Load the corresponding target state snapshot and drive the automated operation engine to restore the operation environment state recorded in the target state snapshot; The digital human engine continues the explanation from the semantic information associated with the target starting point; The speech synthesis engine is driven to continue synthesizing a speech stream from the explanatory content associated with the target starting point; 6) After the lecture resumes, restore the task priority of the dynamic priority thread pool to the default level before the interruption.

[0007] in: Precise environmental status information is determined based on the type of application being operated on, including at least one of the following: Document page numbers; documents include PDF, PPT, PPTX, DOC / DOCX formats. Web page element path identifier; Software interface control identifiers or coordinates; 3D model file path, viewpoint parameters, or selected object status; Document content block identifiers, such as paragraph IDs and chapter numbers.

[0008] The specific steps for predicting the target starting point include: By default, the state snapshot and semantic anchor point corresponding to the interruption point are used as the target starting point; Based on predefined rules or machine learning models, the interruption duration, user interruption request content, and historical demonstration data are analyzed to determine whether it is necessary to skip the part of the content that has been explained before the interruption point mark, and to determine the specific operation node and semantic anchor point to jump to as the target starting point.

[0009] The following core tasks are coordinated and executed using a dynamic priority thread pool: First task: Real-time speech stream synthesis and interruption detection; Second task: Digital human mouth shape and gesture generation; Third task: Automating the execution of operation instructions; Specifically, when a user interruption is detected, the thread pool executes: After the demonstration is completed, restore the task priority to the default level before the interruption; Pause the third task and trigger the recording of a state snapshot in the interruption point marker; During the demonstration of recovery, the thread pool operates based on the stated target starting point: Schedule the third task to load the target state snapshot and resume execution; The second task of scheduling starts from the explanatory semantic information associated with the target semantic anchor point; The first task of scheduling begins synthesis from the explanatory content associated with the target starting point.

[0010] The second technical solution is as follows: An automated operation replay and digital human continuation control system for multimodal interaction includes: The operation snapshot recording module is used to record the status snapshots of key operation nodes in real time during the execution of automated operations; The semantic anchor management module is used to create semantic anchors at semantic key points during the explanation of digital humans; The interrupt detection and capture module is used to detect user interrupts, pause automated operations, and record the interrupt point marker. The context analysis and progress prediction module is used to predict the target starting point for demonstration recovery based on the interruption point marker and context information after interruption handling. The dynamic priority thread pool scheduler is used to coordinate and schedule speech synthesis and interruption detection tasks, digital human rendering tasks, and automated operation execution tasks, and dynamically adjust task priorities and execution status during interruption and recovery. The execution engine control module is used to control the automated operation engine to resume its state, control the digital human engine to continue speaking, and control the speech synthesis engine to continue synthesizing, based on the target starting point output by the progress prediction module.

[0011] in: The precise environmental state information recorded by the operation snapshot recording module includes at least one of the following: document page number, web page element path identifier, software interface control identifier or coordinates, 3D model file path, viewpoint parameters, or selected object status.

[0012] When the progress prediction module predicts the target starting point, it is configured as follows: The default selection is to mark the corresponding point as the breakpoint; Alternatively, based on the interruption duration and interruption request content, rules or models can be applied to predict and select subsequent jump points.

[0013] The dynamic priority thread pool scheduler is configured as follows: When an interrupt is detected, the priority of the interrupt detection task is increased and the automated operation task is paused. Upon receiving a recovery instruction, the automated operation task is scheduled to load the snapshot recovery based on the target starting point, and the digital human task and speech synthesis task are scheduled to start from the continuation point.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: Precise backtracking: After a user interrupts the process, the interface can be restored to its previous state (page number / XPath / viewpoint) within milliseconds, providing a seamless user experience; Intelligent continuation of narration: The digital human's narration can naturally continue from the most relevant point and can intelligently skip redundant content, improving efficiency; Efficient coordination: Dynamic thread pools ensure that high-concurrency tasks run stably and smoothly during interruption / resumption, avoiding stuttering or state confusion; High versatility: Based on the mechanism of snapshot (page number / XPath / viewpoint parameters) and semantic anchors, it can be widely adapted to various presentation scenarios such as PPT, web applications, desktop software, and 3D modeling tools; Standardization potential: Provides a general framework for automated demonstration of interruption recovery. Attached Figure Description

[0015] Figure 1 This is the system architecture diagram of the present invention.

[0016] Figure 2 This is a schematic diagram illustrating the association between the operation node snapshot and the semantic anchor point of the present invention.

[0017] Figure 3 This is a flowchart / state diagram of the dynamic priority thread pool scheduling of the present invention.

[0018] Figure 4 This is a timing diagram / flowchart of the entire interrupt recovery process of the present invention. Detailed Implementation Example

[0019] See Figure 1-4 This embodiment demonstrates an automated operation replay and digital human continuation control method for multimodal interaction, characterized by the following steps: During the execution of automated operations, real-time snapshots of the status of key operation nodes are recorded. These snapshots contain precise environmental status information, including: 1) During the explanation of the digital human, semantic anchors are created at semantic key points. The semantic anchors include at least an anchor identifier, associated explanation semantic information, and associated operation node identifiers. 2) When a user interruption is detected, the automated operation is immediately paused and the interruption point marker at the current moment is recorded. The interruption point marker includes the latest state snapshot identifier and / or the most recent semantic anchor identifier. 3) Handle user interruption requests; 4) After the user interruption request processing is completed, based on the interruption point marker and the context information during the interruption processing, predict the target starting point for demonstration recovery. The target starting point corresponds to a target state snapshot and / or a target semantic anchor point. 5) Based on the stated target starting point: Load the corresponding target state snapshot and drive the automated operation engine to restore the operation environment state recorded in the target state snapshot; The digital human engine continues the explanation from the semantic information associated with the target starting point; The speech synthesis engine is driven to continue synthesizing a speech stream from the explanatory content associated with the target starting point; 6) After the lecture resumes, restore the task priority of the dynamic priority thread pool to the default level before the interruption.

[0020] in: Precise environmental status information is determined based on the type of application being operated on, including at least one of the following: Document page numbers; documents include PDF, PPT, PPTX, DOC / DOCX formats. Web page element path identifier; Software interface control identifiers or coordinates; 3D model file path, viewpoint parameters, or selected object status; Document content block identifiers, such as paragraph IDs and chapter numbers.

[0021] The specific steps for predicting the target starting point include: By default, the state snapshot and semantic anchor point corresponding to the interruption point are used as the target starting point; Based on predefined rules or machine learning models, the interruption duration, user interruption request content, and historical demonstration data are analyzed to determine whether it is necessary to skip the part of the content that has been explained before the interruption point mark, and to determine the specific operation node and semantic anchor point to jump to as the target starting point.

[0022] The following core tasks are coordinated and executed using a dynamic priority thread pool: First task: Real-time speech stream synthesis and interruption detection; Second task: Digital human mouth shape and gesture generation; Third task: Automating the execution of operation instructions; Specifically, when a user interruption is detected, the thread pool executes: After the demonstration is completed, restore the task priority to the default level before the interruption; Pause the third task and trigger the recording of a state snapshot in the interruption point marker; During the demonstration of recovery, the thread pool operates based on the stated target starting point: Schedule the third task to load the target state snapshot and resume execution; The second task of scheduling starts from the explanatory semantic information associated with the target semantic anchor point; The first task of scheduling begins synthesis from the explanatory content associated with the target starting point.

[0023] Figure 1 The core modules showcased include: User Interface (Voice / Vision), Interruption Detection Module, Operation Snapshot Recording Module, Semantic Anchor Management Module, Context Analysis & Progress Prediction Module, Dynamic Priority Thread Pool Scheduler, RPA Execution Engine, Digital Human Generation Engine, Speech Synthesis Engine, and Storage Module (Snapshot Library, Anchor Library, Script Library); and data flow between annotation modules (interruption signals, snapshot data, anchor data, prediction results, control commands). Figure 2 For example, in a PPT presentation, the snapshot record is set to page number 5; the semantic anchor record is created after the "Chart Analysis Conclusion" is explained, and is associated with page number 5. The interruption occurs after this anchor. Figure 3 Demonstrates dynamic priority thread pool scheduling: Normal state: Tasks A, B, and C are running; Interruption detection: Task A priority increases, Task C is paused (snapshot recording), and Task B is paused or put into standby mode; During interruption handling: processing user requests; Recovery instructions: Progress prediction module output results -> Thread pool scheduling: Task C loads snapshot recovery operation -> Task B resumes narration from the anchor point -> Task A synthesizes the resumed narration. Dynamic priority changes are displayed. Figure 4 Demonstrates the entire interruption recovery process: key steps and module interactions from user-initiated voice interruption -> interruption detection -> snapshot and anchor point recording -> handling user issues -> context analysis -> progress prediction -> thread pool scheduling of recovery tasks -> operation backtracking completion -> digital human resumes speaking -> voice synthesis resumes.

[0024] This embodiment also requires: 1. Snapshot Recording Implementation: Describe how to obtain page DOM / XPath / screenshots through the Playwright API; how to obtain page numbers through the PPT API; how to obtain viewpoint parameters through the 3D software API; snapshot data structure (JSON example).

[0025] 2. Semantic Anchor Creation Implementation: Describe the timing of anchor triggering (based on script tags, speech recognition keywords, and natural language processing analysis of the content); anchor data structure; how to bind it to operation nodes (through timestamp association or explicit ID association).

[0026] 3. Interruption Detection and Capture: Describes the keyword / semantic triggering mechanism of the speech recognition engine; how interrupt signals are transmitted to the snapshot and anchor modules for state freezing and recording.

[0027] 4. Progress Prediction Algorithm Implementation: Describe the core rules (e.g., if the interruption time is less than the threshold, return to the interruption point; if the interrupted content involves subsequent topics, jump to the next topic; if the user explicitly requests to skip, execute the next topic). Or describe the machine learning model (input: interruption point information, interruption duration, problem semantics, historical behavior; output: prediction starting node ID).

[0028] 5. Dynamic Thread Pool Implementation: Describe the thread pool management framework (such as Java ExecutorService); how to set the basic priority for tasks A / B / C; how to dynamically adjust the priority by calling APIs (Thread.setPriority or framework-specific methods) when interrupted / resumed; how to safely pause / resume RPA tasks (Playwright's Pause / Resume).

[0029] 6. Resuming narration from breakpoints: Describes how the RPA engine restores the state based on snapshots (Playwright's goto / evaluate restores the DOM state); how the digital human engine loads and generates the narration resume script; and how the speech engine starts synthesis from a specified text offset.

[0030] Therefore, this embodiment can have the following mechanism: 1. Precise operation backtracking mechanism: 1) Operation Node Snapshot Marking: At key nodes in automated operations (such as Playwright driving a browser, operating software) (e.g., PPT page turning completion, software interface switching, 3D model rotation into position), a state snapshot of that node is recorded and stored in real time. The snapshot includes: ● Unique node identifier example: # PPT scenario: snapshot_id = md5("PPT_" + file_path + "_page" + str (page_num))# Web scenario: snapshot_id = md5("WEB_" + url + "_xpath_" + xpath_hash) ●Precise environmental status: such as the current page number in PPT, the XPath / DOM structure of web page elements, the ID / coordinates of software interface controls, and the path / viewpoint parameters / object selection status of 3D model files.

[0031] ●Timestamp 2) Semantic Anchor Point Location: Create and store semantic anchor points at key semantic points in the digital human's explanation (such as chapter titles, the end of core concept explanations, and descriptions of completed operation steps). Anchor points include: ● Anchor point identifier ●Related explanatory text / semantic tags (e.g., "Starting from Chapter 2, Section 3", "After clicking the submit button") ●Associated operation node identifiers (establishing a strong association between semantics and operation state) ●Timestamp 3) Interrupt Status Capture: When the speech recognition module detects a valid user interruption (such as asking a question), it immediately: ● Freeze all current automated operation threads.

[0032] ● Record the latest snapshot of the operation node and the most recent semantic anchor as breakpoint markers.

[0033] 2. Intelligent continuation control mechanism: 1) Context-aware progress prediction: ● After the interruption process (such as answering a user question) ends, analyze the interruption point markers (operation snapshots and semantic anchors) and the contextual changes that occurred during the interruption (such as the content and duration of the user question).

[0034] ● Based on predefined rules or machine learning models, predict the optimal starting point for resuming the demonstration: ■ Typically, the operation node and semantic anchor point corresponding to the breakpoint marker are located directly.

[0035] ■ If the interruption is long or the context indicates that the content has been explained can be skipped, predict and jump to the appropriate subsequent node (such as the next subsection or key operation steps).

[0036] ■ Dynamically adjust the path of subsequent demonstration scripts (e.g., skip redundant steps).

[0037] 2) Resuming playback from breakpoint: Based on the predicted starting point, the control system: ● Operation backtracking: Drives the RPA engine to load the corresponding operation node snapshot and accurately restores the operation interface state (such as turning to a specific PPT page, locating a specific Web element, or restoring the 3D model view).

[0038] ●Digital Human Continues to Speak: Drives the digital human to generate lip movements, gestures, and speech starting from the semantic anchor points associated with the prediction starting point.

[0039] ●Speech stream continuation synthesis: Synthesize subsequent speech streams starting from the continuation point.

[0040] 3. Dynamic priority thread pool coordination mechanism: 1) Threading of the three core tasks: The system's core tasks are divided into: ●Task A (High Real-Time): Real-time speech stream synthesis and interruption detection (highest priority).

[0041] ●Task B (Medium Real-Time): Digital Human Lip / Gesture Generation and Rendering.

[0042] ●Task C (can be paused / resume): Execution of RPA operation instructions (Drivers such as Playwright).

[0043] 2) Dynamic priority scheduling: ●Normal Demonstration: Tasks A, B, and C run in a coordinated manner as needed.

[0044] ● An interrupt was detected: ■ Immediately elevate the priority of Task A to ensure a rapid response to interruptions.

[0045] ■ Pause task C (RPA operation) and record the current snapshot.

[0046] ■Task B (Digital Human) may be paused or switched to standby mode depending on the interruption type (e.g., silent thinking vs. needing feedback).

[0047] ●During interrupt handling: Resources are prioritized for handling interrupt logic (such as Q&A).

[0048] ●Recovery command issued: ■ Based on the progress prediction results, scheduled task C loads a snapshot and restores the operation state.

[0049] ■Scheduled task B starts rendering the digital human from the continuation point.

[0050] ■ Task A starts synthesizing speech from the continuation point.

[0051] ■ The thread pool dynamically adjusts the priority of each task to ensure a smooth recovery process.

[0052] 1. Initial state: Record semantic anchor points when playing to page 5: {"page":5, "voice_anchor":"This chart shows the peak sales in Q3"} 2. Interrupt triggering: ● User question: "Why did the data suddenly increase in Q3?" → Freeze the thread and save a DOM snapshot of page 5. 3. Continued speaking control: ●The progress prediction engine analyzes the user's identity (sales manager) and skips basic explanations when making decisions; ●Continue playback controller: Priority 1: RPA jumps to page 5 (via PPT API SlideShowWindow.View.GotoSlide(5)) Priority 2: Speech synthesis "Due to summer promotional activities..." Priority 3: Digital human finger chart motion generation 4. Performance indicators: ● Retrospective positioning error: ±0.3 seconds (traditional solution ±2 seconds) ● Thread conflict rate decreased by 76% Compared with the prior art, the beneficial effects of the present invention are as follows: Precise backtracking: After a user interrupts the process, the interface can be restored to its previous state (page number / XPath / viewpoint) within milliseconds, providing a seamless user experience; Intelligent continuation of narration: The digital human's narration can naturally continue from the most relevant point and can intelligently skip redundant content, improving efficiency; Efficient coordination: Dynamic thread pools ensure that high-concurrency tasks run stably and smoothly during interruption / resumption, avoiding stuttering or state confusion; High versatility: Based on the mechanism of snapshot (page number / XPath / viewpoint parameters) and semantic anchors, it can be widely adapted to various presentation scenarios such as PPT, web applications, desktop software, and 3D modeling tools; Standardization potential: Provides a general framework for automated demonstration of interruption recovery.

[0053] For those skilled in the art, various modifications and improvements can be made without departing from the inventive concept of this invention, and these all fall within the protection scope of this invention.

Claims

1. A method for multi-modal interaction oriented automated operation backtracking and digital human resuming speech control, characterized in that: The method comprises the following steps: During the execution of the automated operation, a state snapshot of a key operation node is recorded in real time, the state snapshot comprises accurate environment state information, and the accurate environment state information comprises: 1) During the digital human explanation process, a semantic anchor point is created at a semantic key point, and the semantic anchor point comprises at least an anchor point identifier, associated explanation semantic information, and an associated operation node identifier; 2) When a user interruption is detected, the automated operation execution is immediately paused, and a breakpoint marker at the current time is recorded, the breakpoint marker comprising a latest state snapshot identifier and / or a latest semantic anchor point identifier; 3) The user interruption request is processed; 4) After the user interruption request processing is completed, a target starting point for demonstration resumption is predicted based on the breakpoint marker and context information during the interruption processing, the target starting point corresponding to a target state snapshot and / or a target semantic anchor point; 5) According to the target starting point: The corresponding target state snapshot is loaded, and the automated operation engine is driven to resume to an operation environment state recorded by the target state snapshot; The digital human engine is driven to continue explanation from the explanation semantic information associated with the target starting point; The speech synthesis engine is driven to continue to synthesize a speech stream from the explanation content associated with the target starting point; 6) After the continuation of explanation, the task priority of the dynamic priority thread pool is restored to the default level before the interruption.

2. The method of claim 1, wherein: The accurate environment state information is determined according to the type of the application program being operated, and comprises at least one of the following: Document page number, the document comprising a PDF, PPT, PPTX, DOC\DOCX format; Web page element path identifier; Software interface control identifier or coordinate; 3D model file path, view angle parameter or selected object state; Document content block identifier, such as paragraph ID and chapter number.

3. The method of claim 1 or 2, wherein: The step of predicting the target starting point comprises: The state snapshot and the semantic anchor point corresponding to the breakpoint marker are taken as the target starting point by default; Based on a pre-defined rule or a machine learning model, the interruption duration, the user interruption request content and historical demonstration data are analyzed to determine whether the part of the explained content before the breakpoint marker needs to be skipped, and a specific operation node and a semantic anchor point in the subsequent part are determined as the target starting point.

4. The method of claim 1, wherein: The following core tasks are executed through the dynamic priority thread pool: The first task: real-time speech stream synthesis and interruption detection; The second task: digital human mouth shape and gesture generation; The third task: automated operation instruction execution; When the user interruption is detected, the thread pool executes: After the demonstration resumption is completed, the task priority is restored to the default level before the interruption; The third task is paused, and the state snapshot in the breakpoint marker is triggered to be recorded; When the demonstration resumption is performed, the thread pool executes according to the target starting point: The third task is scheduled to load the target state snapshot and resume execution; The second task is scheduled to start generation from the explanation semantic information associated with the target semantic anchor point; The first task is scheduled to start synthesis from the explanation content associated with the target starting point.

5. A multi-modal interaction oriented automated operation backtracking and digital human resuming control system, characterized in that: The method comprises: An operation snapshot recording module, configured to record a state snapshot of a key operation node in real time during the execution of an automated operation; A semantic anchor management module is configured to create a semantic anchor at a semantic key point during the digital human explanation process; An interruption detection and capture module is configured to detect a user interruption, pause the automated operation, and record an interruption point marker; A context analysis and progress prediction module is configured to predict a target starting point for resuming the demonstration after interruption processing based on the interruption point marker and context information; A dynamic priority thread pool scheduler is configured to coordinate and schedule voice synthesis and interruption detection tasks, digital human rendering tasks, and automated operation execution tasks, and dynamically adjust task priorities and execution states during interruption and resumption; An execution engine control module is configured to control the automated operation engine to resume the state, control the digital human engine to continue speaking, and control the voice synthesis engine to continue synthesizing based on the target starting point output by the progress prediction module.

6. The automated operation backtracking and digital human resuming speaking control system for multi-modal interaction according to claim 5, wherein: The precise environment state information recorded by the operation snapshot recording module includes at least one of a document page number, a Web page element path identifier, a software interface control identifier or coordinate, a 3D model file path, a view angle parameter, or a selected object state.

7. The automated operation backtracking and digital human resuming speaking control system for multi-modal interaction according to claim 5 or 6, characterized in that: When predicting the target starting point, the progress prediction module is configured to: Select the point corresponding to the interruption point marker by default; Or predict and select a subsequent jump point according to the interruption duration and interruption request content by applying rules or models.

8. The system of claim 5, wherein, The dynamic priority thread pool scheduler is configured to: When an interruption is detected, the interruption detection task priority is raised, and the automated operation task is paused; When a resumption instruction is received, the automated operation task is loaded to resume based on the target starting point, and the digital human task and voice synthesis task are scheduled to start from the continuation point.

Images