A browser plug-in based ticket process self-adaptive execution method and system

Abstract

The present application belongs to the technical field of browser plug-ins, and particularly relates to a work order process self-adaptive execution method and system based on a browser plug-in, which comprises the following steps: defining and compiling a process script containing semantic operation nodes and sensitive markers on a cloud process arrangement platform, and issuing the process script to a plug-in; the plug-in extracts multi-dimensional real-time features of page elements to generate semantic fingerprints, and locates a target by comparing the fingerprints with benchmark fingerprints through a multi-strategy engine of accurate matching, text semantic similarity matching and visual feature matching; after sending a locator and an operation instruction to an independent Web Worker sandbox thread to generate a personified track and a random delay time sequence, the track and the time sequence are returned to a main thread for execution; when a high-sensitive node is encountered, the process is paused and manual operation is guided with highlighting, and the process is resumed after completion; when positioning or operation fails, a DOM snapshot and a page screenshot of a failure context are reported to the platform, and a benchmark fingerprint library is updated through cluster analysis. Thus, the work order process is self-adaptive to interface changes, and personified safe execution and closed-loop optimization are realized.

Description

Technical Field

[0001] This invention belongs to the technical field of browser plugins, specifically relating to a method and system for adaptive execution of work order processes based on browser plugins. Background Technology

[0002] Currently, in automation scenarios such as customer service ticket processing that require interaction with web interfaces, there exists a macro recording solution based on browser plugins. This solution records the user's click and input sequences on the browser page, such as mouse coordinates, fixed element IDs, CSS selectors, or XPath paths, and generates a replayable script. When the same process needs to be repeated, the plugin automatically replays these pre-recorded operations based on fixed paths of specific page elements, thereby simulating manual operation.

[0003] However, the above-mentioned automated process is highly coupled with the specific Document Object Model (DOM) structure of the target webpage interface, exhibiting strong fragility. Because it relies entirely on fixed element attributes (such as ID and XPath) captured during recording for location, any non-functional changes to the front-end interface of the ticketing system, such as front-end framework upgrades, style class name changes, or page layout adjustments, will invalidate the previously recorded element location paths, causing the entire automated process to interrupt. This requires maintenance personnel to frequently re-record or manually adjust the script, resulting in high maintenance costs and preventing the reuse of the process across different versions or different ticketing systems. Summary of the Invention

[0004] The technical problem to be solved by this invention is to overcome the problem that traditional browser plugin-based automated processes cannot adapt to changes in the work order system interface due to their heavy reliance on fixed page element selectors, resulting in fragile processes and high maintenance costs.

[0005] To address the aforementioned technical problems, a first aspect of this invention discloses an adaptive execution method for work order processes based on browser plugins, the method comprising:

[0006] Define the customer service work order processing flow on the cloud-based workflow orchestration platform, compile it into a workflow script containing semantic operation nodes, and distribute it to the browser plugin;

[0007] The browser plugin receives the process script, and for the semantic operation nodes in the process script, it extracts the real-time features of the target element in the current work order page to generate a real-time semantic fingerprint. It then performs multi-strategy matching between the real-time semantic fingerprint and the pre-stored baseline semantic fingerprint in the process script, and determines the operable locator of the target element based on the matching confidence.

[0008] The browser plugin sends the operable locator and the corresponding operation instructions to an independent WebWorker sandbox thread; the WebWorker sandbox thread generates a human-like operation trajectory and operation sequence, and sends the final operation instructions back to the browser main thread to execute real DOM operations;

[0009] When execution reaches a node marked as requiring manual confirmation, the execution of the Web Worker sandbox thread is paused, and manual operation is received through the visual guidance interface provided by the browser plugin. The automated process is resumed after the manual operation is completed.

[0010] When semantic positioning or operation execution fails, the browser plugin will report failure data, including the page context at the time of failure, to the cloud-based process orchestration platform, which will then analyze the cause of the failure and update the baseline semantic fingerprint database.

[0011] As an optional implementation, in the first aspect of the present invention, defining a customer service ticket processing flow on a cloud-based workflow orchestration platform, compiling it into a workflow script containing semantic operation nodes, and distributing it to a browser plugin includes:

[0012] Arrange the work order processing logic flowchart, which includes sequential nodes and conditional branch nodes, using a visual drag-and-drop method;

[0013] The flowchart is compiled into a lightweight domain-specific language script, in which operation nodes are bound to semantic tags rather than fixed page element selectors;

[0014] The compiled script and the associated baseline semantic fingerprint library are sent to the browser plugin that executes the work order processing flow.

[0015] As an optional implementation, in the first aspect of the present invention, the browser plugin receives the process script, and for semantic operation nodes in the process script, extracts real-time features of target elements in the current work order page to generate real-time semantic fingerprints, performs multi-strategy matching between the real-time semantic fingerprints and pre-stored baseline semantic fingerprints in the process script, and determines the operable locator of the target element based on the matching confidence, including:

[0016] The text content features, structural path features in the DOM tree, visual style features, and contextual text features of neighboring elements of the target element are extracted in parallel and fused to generate the real-time semantic fingerprint.

[0017] The system executes a precise attribute matching strategy, a text semantic similarity matching strategy, and a visual feature matching strategy in parallel to match the real-time semantic fingerprint with the benchmark semantic fingerprint.

[0018] The results of each matching strategy are weighted and scored to obtain a comprehensive confidence level. When the comprehensive confidence level exceeds a preset threshold, the positioning is determined to be successful and the corresponding operable locator is output.

[0019] As an optional implementation, in the first aspect of the present invention, the browser plugin sends the operable locator and corresponding operation instructions to an independent Web Worker sandbox thread; the Web Worker sandbox thread generates a human-like operation trajectory and operation sequence, and sends the final operation instructions back to the browser main thread to execute real DOM operations, including:

[0020] The main thread of the browser plugin encapsulates operation instructions into standardized action objects, which include action type, element locator, input value and behavior mode identifier.

[0021] The action object is sent to the Web Worker sandbox thread via the postMessage interface;

[0022] The Web Worker sandbox thread calculates the Bézier curve trajectory for simulating mouse movement based on the coordinates of the target element in the action object, and generates a delay time that conforms to a preset random distribution model for each click and input operation.

[0023] The Web Worker sandbox thread sends the calculated sequence of operation instructions back to the main thread, which then calls the DOM API to execute the operation.

[0024] As an optional implementation, in the first aspect of the present invention, when execution reaches a node marked as requiring manual confirmation, the execution of the Web Worker sandbox thread is paused, manual operation is received through a visual guidance interface provided by a browser plugin, and the automated process is resumed after the manual operation is completed, including:

[0025] During the workflow orchestration phase, sensitivity levels are marked for the operation nodes in the workflow;

[0026] When the execution engine encounters an operation node marked as high sensitivity, it automatically triggers a manual intervention request and suspends the Web Worker sandbox thread.

[0027] Generate a highlighted visual feedback area around the target element and display a floating confirmation panel in a non-core area of ​​the page;

[0028] The system receives confirmation from the user on the floating confirmation panel or within the highlighted visual feedback area, and sends a resumption execution signal to the Web Worker sandbox thread.

[0029] As an optional implementation, in the first aspect of the present invention, when semantic positioning fails or operation execution fails, the browser plugin reports failure data containing the page context at the time of failure to the cloud-based workflow orchestration platform, whereby the platform analyzes the cause of failure and updates the baseline semantic fingerprint database, including:

[0030] When the browser plugin fails to locate or operate, it captures and records a failure context data packet containing a snapshot of the DOM structure at the time of failure, a page screenshot, and a sequence of operation instructions.

[0031] Upload the failure context data packet to the cloud-based workflow orchestration platform;

[0032] The cloud-based workflow orchestration platform performs cluster analysis on multiple failed cases to identify common interface change patterns that lead to failures.

[0033] Based on the common interface change pattern, adjust the weight coefficients of different features in the corresponding benchmark semantic fingerprint, or generate a new version of the benchmark semantic fingerprint and incrementally update it to the semantic fingerprint database.

[0034] As an optional implementation, in the first aspect of the present invention, before extracting the real-time features of the target element in the current work order page to generate a real-time semantic fingerprint, the method further includes:

[0035] The browser plugin dynamically loads the corresponding system adapter configuration file based on the Uniform Resource Locator (URL) characteristics of the currently accessed work order system page.

[0036] Based on the system adapter configuration file, establish a mapping relationship between standardized semantic operation nodes in the process script and specific page elements of the current work order system.

[0037] As an optional implementation, in the first aspect of the present invention, the process script further includes sensitive operation markers for identifying the risk level of operation nodes; the execution of the Web Worker sandbox thread is paused when a node marked as requiring manual confirmation is reached, specifically when a sensitive operation node marked as high risk is reached.

[0038] A second aspect of this invention discloses a browser plugin-based adaptive execution system for work order processes, used to implement the browser plugin-based adaptive execution method for work order processes described in any of the above embodiments. The system includes:

[0039] The compilation module is used to define customer service work order processing flow on the cloud-based workflow orchestration platform, compile it into a flow script containing semantic operation nodes, and distribute it to the browser plugin.

[0040] The positioning module is used by the browser plugin to receive the process script, extract the real-time features of the target element in the current work order page to generate a real-time semantic fingerprint for the semantic operation node in the process script, perform multi-strategy matching between the real-time semantic fingerprint and the pre-stored benchmark semantic fingerprint in the process script, and determine the operable locator of the target element based on the matching confidence.

[0041] The execution module is used by the browser plugin to send the operable locator and the corresponding operation instructions to an independent Web Worker sandbox thread; the Web Worker sandbox thread generates a human-like operation trajectory and operation sequence, and sends the final operation instructions back to the browser main thread to execute real DOM operations;

[0042] The first confirmation module is used to pause the execution of the Web Worker sandbox thread when it reaches a node marked as requiring manual confirmation, receive manual operation through a visual guidance interface provided by the browser plugin, and resume the automated process after the manual operation is completed.

[0043] The second confirmation module is used to report failure data, including the page context at the time of failure, to the cloud-based process orchestration platform when semantic positioning or operation execution fails. The platform then analyzes the cause of the failure and updates the baseline semantic fingerprint database.

[0044] A third aspect of this invention discloses another adaptive execution system for work order processes based on a browser plugin, the system comprising:

[0045] Memory containing executable program code;

[0046] A processor coupled to the memory;

[0047] The processor calls the executable program code stored in the memory to execute the work order process adaptive execution method based on browser plugin disclosed in the first aspect of the present invention.

[0048] The fourth aspect of the present invention discloses a computer-readable storage medium storing computer instructions, which, when invoked by a processor, are used to execute a work order process adaptive execution method based on a browser plugin disclosed in the first aspect of the present invention.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] By extracting real-time features from target elements across multiple dimensions, including text, structure, and visual aspects, to generate semantic fingerprints, and then performing multi-strategy matching with pre-stored benchmarks to determine operable locators, the system is freed from dependence on fixed attributes such as volatile IDs and XPaths. This decouples automated processes from specific interface implementations, allowing processes to adapt and execute automatically without manual re-recording when faced with changes to the work order system interface or switching between different systems, thus reducing maintenance costs. Secondly, the computationally intensive anthropomorphic trajectory and time-series generation are placed on a separate web application. The Worker sandbox thread executes the scripts, while the main thread is only responsible for the final DOM operations. This eliminates page lag issues caused by the execution of automated scripts, and the simulation of randomness in human operations helps to avoid anti-bot detection. By introducing an automatic identification and suspension mechanism for sensitive nodes in the human-machine collaboration steps, supplemented by non-intrusive visual guidance, a seamless switch between automated processes and necessary manual judgment is achieved. The self-learning step constructs a closed loop from failure on-site reporting to incremental updates of the semantic fingerprint database, enabling the system to learn the pattern of interface changes and optimize strategies from locating or executing failures. This achieves continuous adaptation of the automated process to interface changes, forming a stable, maintainable, and human-machine collaborative work order automation solution. Attached Figure Description

[0051] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0052] Figure 1 This is a flowchart illustrating an adaptive execution method for work order processes based on a browser plugin, as disclosed in an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram of the structure of a work order workflow adaptive execution system based on a browser plugin, as disclosed in an embodiment of the present invention.

[0054] Figure 3 This is a schematic diagram of another adaptive execution system for work order processes based on a browser plugin, as disclosed in an embodiment of the present invention. Detailed Implementation

[0055] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0057] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0058] This invention discloses a browser plugin-based adaptive execution method and system for work order processes. It generates semantic fingerprints by extracting multi-dimensional real-time features of target elements (text, structure, visual aspects), and performs multi-strategy matching with pre-stored benchmarks to determine operable locators. This eliminates dependence on fixed attributes such as volatile IDs and XPaths, decoupling the automated process from the specific interface implementation. Therefore, when facing changes to the work order system interface or switching between different systems, the process can adaptively execute without manual re-recording, reducing maintenance costs. Secondly, it places the computationally intensive anthropomorphic trajectory and time-series generation on a separate web application. Worker sandbox threads execute, while the main thread is only responsible for the final DOM operations, eliminating page lag issues caused by automated script execution. Simulating the randomness of human operations also helps avoid anti-bot detection. By introducing automatic identification and suspension mechanisms for sensitive nodes in the human-machine collaboration steps, supplemented by non-intrusive visual guidance, a seamless switch between automated processes and necessary manual judgment is achieved. The self-learning step constructs a closed loop from failure on-site reporting to incremental updates of the semantic fingerprint database, enabling the system to learn the rules of interface changes and optimize strategies from locating or executing failures, achieving continuous adaptation of the automated process to interface changes.

[0059] Example 1

[0060] Please see Figure 1 , Figure 1 This is a flowchart illustrating an adaptive execution method for work order processes based on a browser plugin, as disclosed in an embodiment of the present invention. Figure 1The described adaptive execution method for work order processes based on browser plugins is applied to data processing chips, processing terminals, or processing servers, and the processing server can be a local server or a cloud server; this embodiment of the invention does not limit the specific application. Figure 1 As shown, the browser plugin-based adaptive execution method for work order processes can include the following operations:

[0061] 101. Define the customer service work order processing flow on the cloud-based workflow orchestration platform, compile it into a workflow script containing semantic operation nodes, and distribute it to the browser plugin.

[0062] Specifically, by separating the business logic of work order automation from the front-end interface implementation of the specific work order system, the process is defined at the platform layer as an abstract "semantic operation node" instead of directly writing scripts that rely on volatile element selectors. This gives the process itself stability that is decoupled from specific system versions.

[0063] As can be seen, this invention achieves the effect of centralized management and one-click deployment of process assets. Administrators only need to orchestrate a process based on business semantics once in the cloud, and then compile and distribute it to the browser plugins of all customer service personnel. This realizes the unified update and efficient distribution of automated rules, and provides standardized input for the adaptive execution of subsequent processes.

[0064] 102. The browser plugin receives the process script, and for the semantic operation node in the process script, extracts the real-time features of the target element in the current work order page to generate a real-time semantic fingerprint, performs multi-strategy matching between the real-time semantic fingerprint and the pre-stored benchmark semantic fingerprint in the process script, and determines the operable locator of the target element based on the matching confidence.

[0065] Specifically, by using real-time semantic fingerprinting and multi-strategy matching mechanisms, the location of page elements in automated processes is transformed from relying on fixed and volatile code attributes to relying on stable features with business semantics. This makes automated scripts no longer bound to a specific version of the DOM structure, so that when non-functional changes occur in the front-end interface of the target work order system, the process can still find the correct operation element through feature matching.

[0066] As can be seen, this design enables automated processes to adapt to changes in the work order system interface. Even if the system's front-end code is upgraded, style class names are changed, or the layout is adjusted, as long as the core semantic features of the target element (such as text, relative position, and visual form) do not change fundamentally, the process can automatically locate and continue to execute, reducing the frequency of process failures and maintenance workload caused by interface redesigns.

[0067] 103. The browser plugin sends the operable locator and the corresponding operation instructions to an independent WebWorker sandbox thread.

[0068] Specifically, by transferring the computational task of generating anthropomorphic operation trajectories and timing to an independent Web Worker sandbox thread, the computational logic and the final interface operation are physically separated on the running thread, avoiding the blocking of the browser's main thread by intensive operations such as complex trajectory calculation and delayed generation.

[0069] As can be seen, this design allows the browser's main thread to remain lightweight, dedicated to responding to user interactions and updating the interface, thus avoiding page lag or response delays that may be caused by the execution of automated scripts. At the same time, the simulated human operating characteristics in the sandbox make the automated behavior more similar to that of a real user, reducing the likelihood of it being detected by the anti-automation mechanism of the ticketing system.

[0070] 104. When the execution reaches a node marked as requiring manual confirmation, the execution of the Web Worker sandbox thread is paused, and manual operation is received through the visual guidance interface provided by the browser plugin. The automated process is resumed after the manual operation is completed.

[0071] Specifically, by intelligently identifying and pausing at critical nodes requiring human decision-making in automated processes, a safe intervention point is provided for tasks that must be judged or operated by humans. This prevents automated processes from being completely interrupted due to complex situations that cannot be handled by preset rules, thus achieving an organic combination between automated execution and necessary human intervention.

[0072] As can be seen, this design maintains the continuity of the overall process while ensuring the safety and compliance of critical operations. After manual confirmation or specific operations are completed, the process can be seamlessly resumed from the interruption point to automate execution, thereby improving processing efficiency while avoiding the operational risks that may be brought about by full automation.

[0073] 105. When semantic positioning fails or operation execution fails, the browser plugin will report failure data containing the page context at the time of failure to the cloud process orchestration platform, which will then analyze the cause of failure and update the baseline semantic fingerprint database.

[0074] Specifically, a feedback and optimization closed loop has been established from the execution failure site to the strategy knowledge base. When an automated process fails due to reasons such as interface changes, the system can automatically capture and report detailed failure context. The cloud platform analyzes the failure mode and updates the baseline semantic fingerprint database accordingly.

[0075] It is evident that this design enables the system to learn from errors. By transforming the failure experience of individual cases into shared strategy updates, the entire system can gradually adapt to the iterative changes of the work order interface, thereby avoiding similar failures in subsequent executions and realizing the transformation of automated process maintenance from complete reliance on manual processing to partial autonomous optimization.

[0076] As an optional embodiment, the step described above, namely defining the customer service ticket processing flow on the cloud-based workflow orchestration platform, compiling it into a workflow script containing semantic operation nodes, and distributing it to the browser plugin, includes:

[0077] Arrange the work order processing logic flowchart, which includes sequential nodes and conditional branch nodes, using a visual drag-and-drop method;

[0078] The flowchart is compiled into a lightweight domain-specific language script, in which operation nodes are bound to semantic tags rather than fixed page element selectors;

[0079] The compiled script and the associated baseline semantic fingerprint library are sent to the browser plugin that executes the work order processing flow.

[0080] In this embodiment of the invention, a visual drag-and-drop method is used to arrange the work order processing logic flowchart containing sequential nodes and conditional branch nodes, which lowers the professional threshold for process design and enables business personnel to intuitively build complex automated process logic without writing code, thereby improving the efficiency and accessibility of process arrangement.

[0081] The flowchart is compiled into a lightweight domain-specific language script. Its core lies in binding the operation nodes in the script with semantic tags, rather than binding them with fixed page element selectors. This decouples the business logic layer of the process from the specific page implementation layer. The process definition no longer cares about the specific ID or CSS path currently used by the target interface, but only focuses on the semantics of the elements that need to be operated, such as the submit button or the customer name input box.

[0082] As can be seen, by sending the compiled script and the associated benchmark semantic fingerprint library to the executing browser plugin, the process execution end has a complete and consistent decision basis. The sent material includes high-level process instructions and feature data benchmarks for identifying interface elements, enabling the browser plugin to independently and accurately adapt the abstract process steps to the real-time changing actual page, providing the necessary foundation for subsequent semantic localization.

[0083] As an optional embodiment, in the above steps, the browser plugin receives the process script, extracts real-time features of the target element in the current work order page to generate a real-time semantic fingerprint for the semantic operation node in the process script, performs multi-strategy matching between the real-time semantic fingerprint and the pre-stored baseline semantic fingerprint in the process script, and determines the operable locator of the target element based on the matching confidence, including:

[0084] The text content features, structural path features in the DOM tree, visual style features, and contextual text features of neighboring elements of the target element are extracted in parallel and fused to generate the real-time semantic fingerprint.

[0085] The system executes a precise attribute matching strategy, a text semantic similarity matching strategy, and a visual feature matching strategy in parallel to match the real-time semantic fingerprint with the benchmark semantic fingerprint.

[0086] The results of each matching strategy are weighted and scored to obtain a comprehensive confidence level. When the comprehensive confidence level exceeds a preset threshold, the positioning is determined to be successful and the corresponding operable locator is output.

[0087] In this embodiment of the invention, by extracting the text content, DOM structure path, visual style and adjacent context text features of the target element in parallel and fusing them to generate a real-time semantic fingerprint, the description of the target element no longer depends on a single volatile attribute, but instead constructs a multi-dimensional, interference-resistant feature representation. This approach ensures that even if some attributes of the page change, as long as the features of other dimensions remain stable, the element can still be effectively identified.

[0088] By executing precise attribute matching, text semantic similarity matching, and visual feature matching strategies in parallel, the advantages of different matching mechanisms are comprehensively utilized. Precise attribute matching ensures fast and accurate positioning when the element identifier remains unchanged; text semantic similarity matching can handle minor changes in text expression or synonym replacement; and visual feature matching provides redundant recognition paths when the element style or position changes. The multi-strategy parallel mechanism enhances the adaptability of the positioning system to various interface change scenarios.

[0089] As can be seen, by weighting and scoring the output results of each matching strategy and calculating the overall confidence level, and finally comparing it with the preset threshold to determine whether the positioning is successful, a quantitative and controllable decision-making mechanism is established. This mechanism allows the weight of each strategy and the overall threshold to be adjusted according to the characteristics of different work order systems, thereby achieving a balance between positioning accuracy and recall, effectively reducing the risk of false positioning and missed positioning, and providing a reliable foundation for subsequent automated operations.

[0090] As an optional embodiment, in the above steps, the browser plugin sends the operable locator and corresponding operation instructions to an independent Web Worker sandbox thread; the Web Worker sandbox thread generates a human-like operation trajectory and operation sequence, and sends the final operation instructions back to the browser main thread to execute real DOM operations, including:

[0091] The main thread of the browser plugin encapsulates operation instructions into standardized action objects, which include action type, element locator, input value and behavior mode identifier.

[0092] The action object is sent to the Web Worker sandbox thread via the postMessage interface;

[0093] The Web Worker sandbox thread calculates the Bézier curve trajectory for simulating mouse movement based on the coordinates of the target element in the action object, and generates a delay time that conforms to a preset random distribution model for each click and input operation.

[0094] The Web Worker sandbox thread sends the calculated sequence of operation instructions back to the main thread, which then calls the DOM API to execute the operation.

[0095] In this embodiment of the invention, the operation instructions are encapsulated into standard action objects containing action type, element locator, input value, and behavior pattern identifier by the main thread. This achieves the abstraction and standardization of operation instructions. The encapsulation makes the transmission of instructions clear and the structure unified, facilitating data exchange between different threads and different processing stages, and also making it easier to expand new operation types in the future. Specifically, this includes:

[0096] By sending standardized action objects to a separate Web Worker sandbox thread through the postMessage interface, safe and asynchronous communication between the browser's main thread and the background computing thread is achieved. As a result, the main thread is freed from time-consuming trajectory calculation tasks and can continuously maintain its responsiveness to user interaction and page rendering, avoiding interface lag or freezing caused by automated script execution.

[0097] The Web Worker sandbox thread calculates the Bézier curve trajectory based on the received target element coordinates and generates an operation delay that conforms to a preset random distribution model. The Bézier curve can simulate the smooth, non-linear trajectory of human hand movement, while the random delay introduced for click and input operations breaks the inherent, mechanical periodic rhythm of automated operations, simulating the characteristics of human behavior in time and space. This makes automated operations closer to real users in external observation and reduces the risk of being judged as machine behavior by the underlying monitoring mechanism of the work order system.

[0098] Finally, the Web Worker thread sends the calculated sequence of operation instructions back to the main thread, which then calls the DOM API to execute the final operation. This design clearly defines the boundaries of responsibility: the Web Worker thread is dedicated to the "planning" of the simulated behavior, while the main thread is responsible for the final "execution." This not only ensures the reliability of the execution phase but also minimizes the impact of the entire anthropomorphic simulation process on page performance.

[0099] As an optional embodiment, in the above steps, when execution reaches a node marked as requiring manual confirmation, the execution of the Web Worker sandbox thread is paused, manual operation is received through a visual guidance interface provided by a browser plugin, and the automated process is resumed after the manual operation is completed, including:

[0100] During the workflow orchestration phase, sensitivity levels are marked for the operation nodes in the workflow;

[0101] When the execution engine encounters an operation node marked as high sensitivity, it automatically triggers a manual intervention request and suspends the Web Worker sandbox thread.

[0102] Generate a highlighted visual feedback area around the target element and display a floating confirmation panel in a non-core area of ​​the page;

[0103] The system receives confirmation from the user on the floating confirmation panel or within the highlighted visual feedback area, and sends a resumption execution signal to the Web Worker sandbox thread.

[0104] In this embodiment of the invention, by pre-marking the sensitivity level of operation nodes during the process orchestration stage, key operations requiring manual supervision are explicitly defined during process design. This allows the automated process to have clear risk control points before execution, embedding security strategies into the process logic rather than relying on temporary judgments at runtime, thereby improving the controllability of the entire automation solution.

[0105] When the execution engine encounters a node marked as highly sensitive, it automatically triggers a request for manual intervention and suspends the WebWorker sandbox thread. This enables a safe and immediate switch from fully automated execution to manual intervention mode. This mechanism ensures that the automated process can be proactively and reliably paused in critical operations that require human judgment or responsibility, preventing automated programs from performing high-risk operations without supervision and providing a guarantee for system security.

[0106] By generating a highlighted visual feedback area around the target element and displaying a floating confirmation panel in a non-core area of ​​the page, clear, intuitive, and non-intrusive guidance is provided for human operators. The highlighted area precisely guides the operator's attention to the target interface element to be processed, while the floating confirmation panel provides a clear interaction entry point. This design reduces the search and confirmation costs when human intervention is needed, lowers the possibility of misoperation, and improves the accuracy of human-computer collaboration.

[0107] By receiving human actions on the floating confirmation panel or in the highlighted visual feedback area and sending a resumption signal to the suspended WebWorker sandbox thread, a smooth transition from human intervention to automation is achieved. This allows humans to complete only the necessary key confirmations or operations, and the subsequent process can continue automatically, avoiding complete interruption of the process, maintaining the continuity of the overall processing task, and realizing the organic combination of automated processes and artificial intelligence in the task execution process.

[0108] As an optional embodiment, in the above steps, when semantic positioning fails or operation execution fails, the browser plugin reports failure data containing the page context at the time of failure to the cloud-based workflow orchestration platform. The platform then analyzes the cause of the failure and updates the baseline semantic fingerprint database, including:

[0109] When the browser plugin fails to locate or operate, it captures and records a failure context data packet containing a snapshot of the DOM structure at the time of failure, a page screenshot, and a sequence of operation instructions.

[0110] Upload the failure context data packet to the cloud-based workflow orchestration platform;

[0111] The cloud-based workflow orchestration platform performs cluster analysis on multiple failed cases to identify common interface change patterns that lead to failures.

[0112] Based on the common interface change pattern, adjust the weight coefficients of different features in the corresponding benchmark semantic fingerprint, or generate a new version of the benchmark semantic fingerprint and incrementally update it to the semantic fingerprint database.

[0113] In this embodiment of the invention, a closed-loop optimization system that continuously learns from failures is constructed. When a browser plugin fails, it automatically captures a complete failure context data packet containing a snapshot of the DOM structure, a page screenshot, and a sequence of operation instructions. This ensures that the state of each failure is recorded completely and accurately, providing a reliable data foundation for subsequent root cause analysis and avoiding the problem of relying on fuzzy logs for guesswork.

[0114] Uploading failure context data packages to the cloud-based process orchestration platform enables centralized management of failure experiences scattered across multiple terminals. This allows previously isolated and individual failure events to be aggregated, providing the platform with the data scale for macro-level analysis and making it possible to identify common and systemic problems.

[0115] The cloud-based workflow orchestration platform performs cluster analysis on multiple failed cases to identify common interface change patterns that lead to failures. This process can abstract common causes of failure from a large number of individual cases, such as regular changes in the style class name of a certain type of button, or systematic adjustments to the surrounding text prompts of an input box. By identifying these patterns through algorithms, problem localization is transformed from manual investigation to automated pattern discovery.

[0116] It is evident that adjusting the weight coefficients of different features in the corresponding baseline semantic fingerprint based on the identified common interface change patterns, or generating a new version of the baseline semantic fingerprint and incrementally updating it to the semantic fingerprint database, is a key step in transforming analytical conclusions into practical solutions. For example, when analysis reveals that the ID attribute of an element changes frequently but its visual features remain stable, the system will automatically reduce the weight of the ID attribute in the semantic fingerprint of that element and increase the weight of its visual features. By incrementally distributing the updated semantic fingerprint database, all online browser plugins can share this learning result, thereby avoiding similar failures in subsequent executions and enabling the entire automated system to cope with the continuous evolution of the interface.

[0117] As an optional embodiment, the above steps, before extracting the real-time features of the target element in the current work order page to generate a real-time semantic fingerprint, further include:

[0118] The browser plugin dynamically loads the corresponding system adapter configuration file based on the Uniform Resource Locator (URL) characteristics of the currently accessed work order system page.

[0119] Based on the system adapter configuration file, establish a mapping relationship between standardized semantic operation nodes in the process script and specific page elements of the current work order system.

[0120] In this embodiment of the invention, by dynamically loading the corresponding system adapter configuration file based on the Uniform Resource Locator (URL) feature of the currently accessed work order system page, the automatic identification and adaptation of the automated plugin to different work order systems is realized. This enables the same browser plugin to run on multiple different customer service work order systems without manual configuration, improving the versatility and deployment convenience of the tool.

[0121] By establishing a mapping relationship between standardized semantic operation nodes in the process script and specific page elements of the current work order system through the system adapter configuration file, a configurable conversion layer is built between the abstract and general automated process and the specific and variable system interface. This allows the standard process defined on the cloud orchestration platform based on business semantics to be accurately mapped to the unique interface of different work order systems through different adapters, achieving the effect of "orchestrate once, run in multiple places" and solving the problem of cross-system process reuse and migration.

[0122] As can be seen, this design encapsulates the differences between systems in independent adapter configuration files, reducing the complexity of the core process logic. When it is necessary to support a new work order system or deal with a major version update of the interface of a certain system, it is usually only necessary to add or modify the corresponding adapter configuration file, without rewriting the core process script or positioning algorithm, which reduces the cost of system expansion and maintenance and makes the whole solution have good scalability.

[0123] As an optional embodiment, in the above steps, the process script further includes sensitive operation markers for identifying the risk level of operation nodes; when the execution reaches a node marked as requiring manual confirmation, the execution of the WebWorker sandbox thread is paused, specifically when the execution reaches a sensitive operation node marked as high risk.

[0124] In this embodiment of the invention, by marking the risk level of operation nodes during the process orchestration stage, process designers are able to manage the safety requirements of different operations in a refined manner. The above design allows for the differentiation between high-risk operations and routine operations in the same automated process, thereby providing a basis for implementing differentiated execution strategies and enhancing the flexibility of process control.

[0125] By clearly defining the trigger conditions for manual confirmation as high-risk sensitive operation nodes, the automated process is ensured to request manual intervention only when necessary, avoiding unnecessary process pauses in non-critical or low-risk operation steps. This ensures the safety of critical operations while maximizing the continuity of automated execution and overall processing efficiency.

[0126] It is evident that this mechanism establishes a configurable balance between automation efficiency and operational safety. Through predefined risk markers, the system can automatically and reliably switch to manual supervision mode at nodes involving important data changes, final confirmation, or other critical business logic. This prevents the risk of misoperation that may arise from full automation, while also avoiding the inefficiency caused by setting all operations to be handled manually due to excessive conservatism.

[0127] Furthermore, a semantic fingerprint that does not rely on fixed selectors (such as XPath or CSS selectors) is constructed. The generation process of this fingerprint is as follows:

[0128] 1. Multi-dimensional feature extraction: For target operation elements, the locator does not rely on a single attribute, but extracts its multi-dimensional features in parallel:

[0129] a. Content characteristics: The text content, title, aria-label and other attribute values ​​of the element itself are normalized (whitespace and special characters are removed) and used as text fingerprints.

[0130] b. Contextual features: Extract the element's "contextual signature" in the DOM tree, such as the tag name of its neighboring sibling elements, part of the text content, and the id and class of the parent container, to assist in positioning when the page structure changes.

[0131] c. Structural features: Calculate the relative path hash of the element in the DOM tree, for example, hash the tag path starting from the body node (such as div[3] / section[1] / button) as a structural fingerprint.

[0132] d. Visual features: For buttons or icons with obvious visual features, extract key visual properties (such as background color, border-radius, and size) from their CSS to form a visual fingerprint vector.

[0133] 2. Fingerprint Vectorization and Fusion: The extracted text, context, structural, and visual features are vectorized (e.g., using MinHash to reduce the dimensionality of text features and converting structural features into one-hot encoding), and then a multidimensional semantic fingerprint vector is formed through weighted fusion. The initial values ​​of the weights are based on the historical stability preset of the features and are dynamically adjusted in the subsequent self-learning loop.

[0134] 3. Multi-strategy matching and confidence scoring: During execution, the plugin recalculates the real-time semantic fingerprint of the element to be operated on on the current page and performs multi-strategy matching with the fingerprint recorded in the process. Multiple algorithms (such as edit distance, cosine similarity, and Jaccard similarity) run in parallel during the matching process, with each algorithm outputting a matching score. Finally, a confidence scoring model (such as a weighted voting mechanism) integrates the results of each algorithm. Only when the overall confidence exceeds a preset threshold (such as 0.85) is the location considered successful and the operation executed. This architecture ensures that even if the front-end code of the ticketing system is completely refactored, the automated process can still be effectively executed as long as the core semantic information remains unchanged, achieving complete decoupling from UI implementation details.

[0135] Human-like execution engines based on Web Worker sandboxes include:

[0136] 1. Performance optimization and main thread unloading:

[0137] Operation offloading: All non-DOM operation logic, such as Bézier curve interpolation calculation of operation trajectory, random delay generation (Gaussian distribution), operation sequence pre-playing and conflict detection, is encapsulated and run in an independent Web Worker thread.

[0138] Lightweight main thread: The main thread is only responsible for receiving the final, calculated operation instructions (such as click(x, y, delay)) from the Worker and executing the lowest-level DOM API calls. This architecture reduces the event loop time of the main thread from an average of 18ms (common in industry automation tools) to below 3ms, and increases the page rendering frame rate (FPS) from an average of 45 frames to a stable 60 frames, completely eliminating UI stuttering and visual abruptness during the execution of automated scripts, making their behavior more like that of a real user.

[0139] 2. Anthropomorphic behavior modeling and anti-detection avoidance:

[0140] Bézier curve trajectory: The Web Worker dynamically generates a cubic Bézier curve path based on the coordinates of the target element and calculates about 50-80 discrete mouse movement points on the path to simulate the unavoidable slight jitter and arc movement of a human hand when operating a mouse, rather than mechanical straight-line movement.

[0141] Human-like operation timing: A behavior timing generator is implemented within the Worker. This generator randomly assigns intervals to each click, input, and other operation based on real user operation statistics. For example, the interval between two clicks follows a truncated normal distribution within 80-200ms, and the interval for keyboard input fluctuates randomly within 35-90ms. This non-fixed, statistically regular timing pattern makes the behavioral fingerprint of the automated process highly similar to that of a real user, thus effectively avoiding common anti-bot detection strategies based on operation frequency and fixed intervals.

[0142] By embedding a state machine-driven arbitrator module into the process execution engine, smooth collaboration between automation and manual operation is achieved. During the process orchestration phase, a sensitivity level can be defined for each operation (e.g., L0: low-risk query, L1: medium-risk data entry, L2: high-risk work order review / submission). The arbitrator monitors the execution process in real time. When encountering a sensitive L2-level operation, it automatically triggers a "manual intervention request" on the plugin side and suspends the automation thread. Instead of a pop-up blocking window, the system uses a highlighted, flashing visual guide (e.g., a semi-transparent halo above the "Confirm Submission" button) and a subtle prompt to inform the user that "This is a high-risk operation; please confirm the attachments are correct before clicking here to continue." After the user completes manual verification, they can seamlessly resume automation by clicking the guide or the preset "One-Click Restore" button. During the suspension, both the Web Worker sandbox and the main thread maintain their current state, including filled-in form data and page scroll position. Upon resumption, the arbitrator sends a recovery signal to the execution engine, and the automation process continues execution from the next instruction after the suspension point. The entire process achieves a seamless switch between "human judgment and intervention" and "automated execution," ensuring the security of sensitive operations while retaining the efficiency of automation. A typical application scenario is work order review: after the system automatically fills in all the information, it suspends the process for human review of the authenticity and compliance of the attachments. Once the human confirms, the submission can be resumed with a single click.

[0143] This invention establishes a complete closed loop from failure scene data collection to automatic strategy updates, achieving "self-healing" of the automated process. During each process execution, the plugin not only records operation instructions but also DOM snapshots, page screenshots, and confidence scores for key nodes. When the confidence score of any operation falls below a threshold (e.g., 0.6) or fails, the plugin immediately packages the complete context of that step (DOM snapshot at the time of failure, screenshot, operation instructions, and the semantic fingerprint of the expected element), along with the last 30 seconds of data from the operation video recording, and sends it back to the platform. Upon receiving the failure context, the platform initiates an offline analysis service. This service first uses the "failure screenshot" and "DOM snapshot" to locate the specific failed element. Then, it performs a structured comparison between the "DOM snapshot at the time of failure" and the "DOM snapshot during process design" to identify the root cause of the failure (e.g., the element's id changed, but its text content remained the same; or the button's CSS class name changed, but its relative position remained the same). Based on this analysis, the platform automatically generates a new, more robust localization strategy. For example, if the original fingerprint is found to overly rely on the volatile `id` attribute, the new fingerprint would increase the weight of the "text content" and "relative position" features, or directly add a new "text plus position" combined fingerprint. After manual review or automatic verification, this updated positioning strategy will be incrementally updated to the semantic fingerprint database. All other process instances that depend on this element will automatically download and use this new, more reliable positioning strategy on their next execution, thus forming a self-healing closed loop of "failure → analysis → learning → update → immunity," reducing the maintenance costs of automated scripts caused by page tweaks.

[0144] Separation of adapter and semantic mapping design:

[0145] Abstract Process Layer: On the platform side, all automated processes are orchestrated based on "semantic operation primitives," rather than specific page element selectors. For example, a process might be defined as "Search for work orders in the work order system → Click the first work order → Fill in the review comments → Submit." These primitives are independent of the specific system implementation.

[0146] Adapter Layer: Each ticket system has a separate adapter plugin. This adapter does not contain any business logic; it is only responsible for establishing the mapping relationship between "semantic operation primitives" and "target system UI implementation". For example, for the semantic primitive "search ticket", the adapter of system A maps it to "enter keywords in input#search-box and click button.search", while the adapter of system B maps it to "click div.search-icon to expand the search box, enter the keyword in textarea[name='keyword'] and press Enter".

[0147] Hot-swappable mechanism: Adapters exist as independent JSON configuration files or JavaScript modules and support dynamic loading at runtime. When the same automated process needs to be applied to a new work order system, there is no need to modify the core process code; simply install and activate the adapter corresponding to the new system. The system will automatically detect and load the installed adapter upon startup, achieving true "hot-swappability."

[0148] Cross-system mapping migration based on semantic fingerprints:

[0149] Semantic fingerprint database reuse: When migrating from system A to system B, although the UI is completely different, the core business semantics (such as "work order number" and "submit button") are the same. This architecture supports cross-system migration of semantic fingerprints.

[0150] Mapping Learning and Recommendation: During the initial configuration phase, the adapter can establish mappings in a semi-automatic manner. For example, an administrator defines a semantic fingerprint (including text features, location features, etc.) for "work order number" in the adapter for system A. When configuring the adapter for system B, the system can automatically scan the pages of system B and recommend elements with the highest similarity to the semantic fingerprint of system A (such as input boxes containing the text "work order number"). The administrator only needs to confirm to complete the mapping. This process fully utilizes the generalization capability of "semantic fingerprint" in Innovation Point 1, reusing the experience gained from one configuration to a new system.

[0151] Mapping Version Management: The platform manages the version of the mapping configurations for all adapters. When a UI redesign of a ticketing system causes mapping to fail, only the adapter configuration for that system needs to be updated (e.g., correcting the location fingerprint of an element), without modifying any business process code. Adapters and core processes in other systems remain unaffected, achieving isolation of maintenance impacts.

[0152] Adapter runtime dynamic switching and grayscale verification:

[0153] Multiple adapters coexisting: In large enterprises, there may be scenarios where two versions of the same work order system coexist, or different business departments use different work order systems. This architecture supports multiple adapters coexisting, and the core process can dynamically select which adapter to call for execution through routing strategies (such as based on user ID, department, or request parameters).

[0154] Seamless Switching and Rollback: When migrating automation for a business line from an old system to a new system, the adapter can be configured and verified on the new system first. Then, by modifying the routing policy, a portion of traffic (e.g., 1% of users) can be switched to the new system adapter for canary deployment testing. After successful verification, the switching ratio can be gradually increased until 100%. The entire process is completely transparent to business personnel. If a problem occurs in the new system, all traffic can be instantly switched back to the old system adapter using the routing policy.

[0155] Adapter sandbox isolation: Each adapter runs in its own independent JavaScript scope, without interfering with each other. Even if one adapter crashes due to a page parsing error, it will not affect the normal operation of other adapters, nor will it cause the entire browser plugin to crash.

[0156] The detailed implementation of this invention includes:

[0157] 1. Process Release and Loading Phase (ab)

[0158] a. Deployment of the process package: The orchestration platform compiles the visual flowchart into a lightweight DSL, which includes:

[0159] Node order, conditional branches;

[0160] Each operation node is bound to a target field semantic label (non-fixed selector);

[0161] Sensitive operation marker (whether manual confirmation is required);

[0162] b. Loading process: The plugin's main thread receives the DSL and constructs the execution context object.

[0163] 2. Semantic localization stage (cf)

[0164] c. Capture page metadata: The plugin extracts the DOM feature fingerprint of the current page.

[0165] Text labels, placeholders, and ARIA attributes for all interactive elements;

[0166] The relative positional relationships between elements;

[0167] Page URL characteristics and title characteristics;

[0168] d. Semantic location request: Package the "field name to be located" (e.g., customerPhone) with the current page features;

[0169] e. Multi-strategy matching engine:

[0170] Strategy 1: Exact match (ID / Name / XPath match);

[0171] Strategy 2: Semantic similarity (using Word2Vec or edit distance to match "phone number" ≈ "contact number");

[0172] Strategy 3: Visual features (identifying text labels above the input box);

[0173] Output the confidence score and select the highest-scoring locator.

[0174] f. Return locator: Returns an operable element selector.

[0175] 3. Sandbox execution phase (gk)

[0176] g. Instruction dispatch: The main thread encapsulates operation instructions into standardized action objects.

[0177] {

[0178] "action": "input",

[0179] "selector": "#phone",

[0180] "value": "13800138000",

[0181] "behavior": "humanized",

[0182] "sensitive": false

[0183] }

[0184] h. Forward to Sandbox: The main thread sends the instruction to the Web Worker sandbox via postMessage;

[0185] i. Simulated execution: Conducting operational rehearsals within a sandbox:

[0186] Calculate the path of the Bézier curve;

[0187] Generate random hover times (80-200ms);

[0188] Generate random input intervals (human-like typing rhythm);

[0189] It does not manipulate the actual DOM; it only generates an operation plan.

[0190] j. Real operation: The sandbox sends the final operation parameters back to the main thread, and the main thread executes the real DOM operation;

[0191] k. Execution status feedback: The operation result (success / abnormal) is returned to the sandbox log.

[0192] 4. Human-Machine Collaboration Stage (ln)

[0193] l. Sensitive operation judgment: Is the current node marked as needConfirm: true

[0194] m. Trigger manual confirmation:

[0195] The plugin displays a floating confirmation box at the edge of the page;

[0196] The target button features a breathing light-style high-intensity display.

[0197] Automated operations are paused and are in progress;

[0198] n. Continue execution: After manual confirmation, the automated process will resume.

[0199] 5. Self-learning feedback phase (op)

[0200] o. Reporting failure characteristics: When semantic positioning fails, the current page structure and failure context are reported to the platform;

[0201] p. Update semantic database: Platform administrators can manually mark the correct location, and the platform will automatically generate a new version of the semantic fingerprint, which will take effect automatically in subsequent processes.

[0202] Example 2

[0203] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a work order workflow adaptive execution system based on a browser plugin, as disclosed in an embodiment of the present invention. Figure 2 The described browser plugin-based adaptive execution system for work orders can be applied to data processing chips, processing terminals, or processing servers. The processing server can be a local server or a cloud server; this embodiment of the invention does not limit the application. Figure 2 As shown, the browser plugin-based adaptive execution system for work orders can include the following operations:

[0204] Compilation module 201 is used to define customer service work order processing flow on the cloud-based workflow orchestration platform, compile it into a flow script containing semantic operation nodes, and distribute it to the browser plugin.

[0205] Specifically, by separating the business logic of work order automation from the front-end interface implementation of the specific work order system, the process is defined at the platform layer as an abstract "semantic operation node" instead of directly writing scripts that rely on volatile element selectors. This gives the process itself stability that is decoupled from specific system versions.

[0206] As can be seen, this invention achieves the effect of centralized management and one-click deployment of process assets. Administrators only need to orchestrate a process based on business semantics once in the cloud, and then compile and distribute it to the browser plugins of all customer service personnel. This realizes the unified update and efficient distribution of automated rules, and provides standardized input for the adaptive execution of subsequent processes.

[0207] The positioning module 202 is used by the browser plugin to receive the process script, extract the real-time features of the target element in the current work order page to generate a real-time semantic fingerprint for the semantic operation node in the process script, perform multi-strategy matching between the real-time semantic fingerprint and the pre-stored benchmark semantic fingerprint in the process script, and determine the operable locator of the target element based on the matching confidence.

[0208] Specifically, by using real-time semantic fingerprinting and multi-strategy matching mechanisms, the location of page elements in automated processes is transformed from relying on fixed and volatile code attributes to relying on stable features with business semantics. This makes automated scripts no longer bound to a specific version of the DOM structure, so that when non-functional changes occur in the front-end interface of the target work order system, the process can still find the correct operation element through feature matching.

[0209] As can be seen, this design enables automated processes to adapt to changes in the work order system interface. Even if the system's front-end code is upgraded, style class names are changed, or the layout is adjusted, as long as the core semantic features of the target element (such as text, relative position, and visual form) do not change fundamentally, the process can automatically locate and continue to execute, reducing the frequency of process failures and maintenance workload caused by interface redesigns.

[0210] The execution module 203 is used by the browser plugin to send the operable locator and the corresponding operation instructions to an independent Web Worker sandbox thread; the Web Worker sandbox thread generates anthropomorphic operation trajectories and operation sequences, and sends the final operation instructions back to the browser main thread to execute real DOM operations.

[0211] Specifically, by transferring the computational task of generating anthropomorphic operation trajectories and timing to an independent Web Worker sandbox thread, the computational logic and the final interface operation are physically separated on the running thread, avoiding the blocking of the browser's main thread by intensive operations such as complex trajectory calculation and delayed generation.

[0212] As can be seen, this design allows the browser's main thread to remain lightweight, dedicated to responding to user interactions and updating the interface, thus avoiding page lag or response delays that may be caused by the execution of automated scripts. At the same time, the simulated human operating characteristics in the sandbox make the automated behavior more similar to that of a real user, reducing the likelihood of it being detected by the anti-automation mechanism of the ticketing system.

[0213] The first confirmation module 204 is used to pause the execution of the WebWorker sandbox thread when it reaches a node marked as requiring manual confirmation, receive manual operation through the visual guidance interface provided by the browser plugin, and resume the automated process after the manual operation is completed.

[0214] Specifically, by intelligently identifying and pausing at critical nodes requiring human decision-making in automated processes, a safe intervention point is provided for tasks that must be judged or operated by humans. This prevents automated processes from being completely interrupted due to complex situations that cannot be handled by preset rules, thus achieving an organic combination between automated execution and necessary human intervention.

[0215] As can be seen, this design maintains the continuity of the overall process while ensuring the safety and compliance of critical operations. After manual confirmation or specific operations are completed, the process can be seamlessly resumed from the interruption point to automate execution, thereby improving processing efficiency while avoiding the operational risks that may be brought about by full automation.

[0216] The second confirmation module 205 is used to report failure data, including the page context at the time of failure, to the cloud process orchestration platform when semantic positioning or operation execution fails. The platform then analyzes the cause of failure and updates the baseline semantic fingerprint database.

[0217] Specifically, a feedback and optimization closed loop has been established from the execution failure site to the strategy knowledge base. When an automated process fails due to reasons such as interface changes, the system can automatically capture and report detailed failure context. The cloud platform analyzes the failure mode and updates the baseline semantic fingerprint database accordingly.

[0218] It is evident that this design enables the system to learn from errors. By transforming the failure experience of individual cases into shared strategy updates, the entire system can gradually adapt to the iterative changes of the work order interface, thereby avoiding similar failures in subsequent executions and realizing the transformation of automated process maintenance from complete reliance on manual processing to partial autonomous optimization.

[0219] Example 3

[0220] Please see Figure 3 , Figure 3 This is a schematic diagram of another adaptive execution system for work order processes based on a browser plugin, as disclosed in an embodiment of the present invention. Figure 3 As shown, the device may include:

[0221] Memory 301 storing executable program code;

[0222] Processor 302 coupled to memory 301;

[0223] The processor 302 calls the executable program code stored in the memory 301 to execute some or all of the steps in the work order process adaptive execution method based on browser plugin disclosed in Embodiment 1 of the present invention.

[0224] Example 4

[0225] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in the work order process adaptive execution method based on a browser plugin disclosed in Embodiment 1 of this invention.

[0226] Example 5

[0227] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps of a browser plugin-based work order process adaptive execution method described in Embodiment 1.

[0228] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0229] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0230] Finally, it should be noted that the adaptive execution method and system for work order processes based on browser plugins disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A work order workflow adaptive execution method based on a browser plugin, characterized in that, The method includes: Define the customer service work order processing flow on the cloud-based workflow orchestration platform, compile it into a workflow script containing semantic operation nodes, and distribute it to the browser plugin; The browser plugin receives the process script, and for the semantic operation nodes in the process script, it extracts the real-time features of the target element in the current work order page to generate a real-time semantic fingerprint. It then performs multi-strategy matching between the real-time semantic fingerprint and the pre-stored baseline semantic fingerprint in the process script, and determines the operable locator of the target element based on the matching confidence. The browser plugin sends the operable locator and the corresponding operation instructions to an independent Web Worker sandbox thread; the Web Worker sandbox thread generates a human-like operation trajectory and operation sequence, and sends the final operation instructions back to the browser main thread to execute real DOM operations; When execution reaches a node marked as requiring manual confirmation, the execution of the Web Worker sandbox thread is paused, and manual operation is received through the visual guidance interface provided by the browser plugin. The automated process is resumed after the manual operation is completed. When semantic positioning or operation execution fails, the browser plugin will report failure data, including the page context at the time of failure, to the cloud-based process orchestration platform, which will then analyze the cause of the failure and update the baseline semantic fingerprint database.

2. The adaptive execution method for work order processes based on browser plugins according to claim 1, characterized in that, The process of defining a customer service ticket processing flow on a cloud-based workflow orchestration platform, compiling it into a workflow script containing semantic operation nodes, and then distributing it to a browser plugin includes: Arrange the work order processing logic flowchart, which includes sequential nodes and conditional branch nodes, using a visual drag-and-drop method; The flowchart is compiled into a lightweight domain-specific language script, in which operation nodes are bound to semantic tags rather than fixed page element selectors; The compiled script and the associated baseline semantic fingerprint library are sent to the browser plugin that executes the work order processing flow.

3. The adaptive execution method for work order processes based on browser plugins according to claim 1, characterized in that, The browser plugin receives the process script, and for semantic operation nodes in the process script, extracts real-time features of target elements in the current work order page to generate real-time semantic fingerprints. It then performs multi-strategy matching between the real-time semantic fingerprints and pre-stored baseline semantic fingerprints in the process script, and determines the operable locator of the target element based on the matching confidence, including: The text content features, structural path features in the DOM tree, visual style features, and contextual text features of neighboring elements of the target element are extracted in parallel and fused to generate the real-time semantic fingerprint. The system executes a precise attribute matching strategy, a text semantic similarity matching strategy, and a visual feature matching strategy in parallel to match the real-time semantic fingerprint with the benchmark semantic fingerprint. The results of each matching strategy are weighted and scored to obtain a comprehensive confidence level. When the comprehensive confidence level exceeds a preset threshold, the positioning is determined to be successful and the corresponding operable locator is output.

4. The adaptive execution method for work order processes based on browser plugins according to claim 1, characterized in that, The browser plugin sends the operable locator and the corresponding operation instructions to a separate Web Worker sandbox thread. The Web Worker sandbox thread generates a human-like operation trajectory and sequence, and sends the final operation instructions back to the browser's main thread to execute real DOM operations, including: The main thread of the browser plugin encapsulates operation instructions into standardized action objects, which include action type, element locator, input value and behavior mode identifier. The action object is sent to the Web Worker sandbox thread via the postMessage interface; The Web Worker sandbox thread calculates the Bézier curve trajectory for simulating mouse movement based on the coordinates of the target element in the action object, and generates a delay time that conforms to a preset random distribution model for each click and input operation. The Web Worker sandbox thread sends the calculated sequence of operation instructions back to the main thread, which then calls the DOM API to execute the operation.

5. The adaptive execution method for work order processes based on browser plugins according to claim 1, characterized in that, When execution reaches a node marked as requiring manual confirmation, the execution of the Web Worker sandbox thread is paused. Manual intervention is then received through a visually guided interface provided by a browser plugin. The automated process resumes after the manual intervention is completed, including: During the workflow orchestration phase, sensitivity levels are marked for the operation nodes in the workflow; When the execution engine encounters an operation node marked as high sensitivity, it automatically triggers a manual intervention request and suspends the Web Worker sandbox thread. Generate a highlighted visual feedback area around the target element and display a floating confirmation panel in a non-core area of ​​the page; The system receives confirmation from the user on the floating confirmation panel or within the highlighted visual feedback area, and sends a resumption execution signal to the WebWorker sandbox thread.

6. The adaptive execution method for work order processes based on browser plugins according to claim 1, characterized in that, When semantic positioning or operation execution fails, the browser plugin reports failure data, including the page context at the time of failure, to the cloud-based workflow orchestration platform. The platform analyzes the cause of the failure and updates the baseline semantic fingerprint database, including: When the browser plugin fails to locate or operate, it captures and records a failure context data packet containing a snapshot of the DOM structure at the time of failure, a page screenshot, and a sequence of operation instructions. Upload the failure context data packet to the cloud-based workflow orchestration platform; The cloud-based workflow orchestration platform performs cluster analysis on multiple failed cases to identify common interface change patterns that lead to failures. Based on the common interface change pattern, adjust the weight coefficients of different features in the corresponding benchmark semantic fingerprint, or generate a new version of the benchmark semantic fingerprint and incrementally update it to the semantic fingerprint database.

7. The adaptive execution method for work order processes based on browser plugins according to claim 1, characterized in that, Before extracting real-time features of target elements in the current work order page to generate a real-time semantic fingerprint, the method further includes: The browser plugin dynamically loads the corresponding system adapter configuration file based on the Uniform Resource Locator (URL) characteristics of the currently accessed work order system page. Based on the system adapter configuration file, establish a mapping relationship between standardized semantic operation nodes in the process script and specific page elements of the current work order system.

8. The adaptive execution method for work order processes based on browser plugins according to claim 1, characterized in that, The process script also includes sensitive operation markers for identifying the risk level of operation nodes; when the execution reaches a node marked as requiring manual confirmation, the execution of the Web Worker sandbox thread is paused, specifically when the execution reaches a sensitive operation node marked as high risk.

9. A browser plugin-based adaptive execution system for work order workflows, used to implement the browser plugin-based adaptive execution method for work order workflows as described in any one of claims 1-8, characterized in that, The system includes: The compilation module is used to define customer service work order processing flow on the cloud-based workflow orchestration platform, compile it into a flow script containing semantic operation nodes, and distribute it to the browser plugin. The positioning module is used by the browser plugin to receive the process script, extract the real-time features of the target element in the current work order page to generate a real-time semantic fingerprint for the semantic operation node in the process script, perform multi-strategy matching between the real-time semantic fingerprint and the pre-stored benchmark semantic fingerprint in the process script, and determine the operable locator of the target element based on the matching confidence. The execution module is used by the browser plugin to send the operable locator and the corresponding operation instructions to an independent Web Worker sandbox thread; the Web Worker sandbox thread generates a human-like operation trajectory and operation sequence, and sends the final operation instructions back to the browser main thread to execute real DOM operations; The first confirmation module is used to pause the execution of the Web Worker sandbox thread when it reaches a node marked as requiring manual confirmation, receive manual operation through a visual guidance interface provided by the browser plugin, and resume the automated process after the manual operation is completed. The second confirmation module is used to report failure data, including the page context at the time of failure, to the cloud-based process orchestration platform when semantic positioning or operation execution fails. The platform then analyzes the cause of the failure and updates the baseline semantic fingerprint database.

10. A work order workflow adaptive execution system based on a browser plugin, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the work order process adaptive execution method based on a browser plugin as described in any one of claims 1-8.

Images