A lightweight robotic process automation method based on browser extension

Abstract

The application relates to a lightweight robot process automation method based on a browser extension and belongs to the technical fields of robot process automation and browser plug-ins. The method comprises the following steps: initializing a drawing environment, preprocessing a node style, binding an input event to construct an interaction mechanism; communication is established through an event bus, an interface is constructed through a dynamic form component, instructions are encapsulated and syntax verification is performed; a command data format is defined, after syntax and semantic verification, the command data format is converted into executable code by an intermediate code generator, and error feedback visualization is supplemented; a content script is used to access a monitoring webpage DOM, an element operation and RPA instruction mapping library are established, and automatic conversion of a capture result into an instruction is realized; a lightweight execution kernel is realized through intermediate code acquisition, a process is monitored in an asynchronous non-blocking mode, and execution state global sharing and real-time updating are realized through a state management library. The lightweight architecture design of RPA, high-performance interaction optimization, intelligent process configuration and precise execution monitoring are realized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of robot process automation and browser plug-in, and particularly relates to a lightweight robot process automation method based on browser extension. BACKGROUND

[0002] The existing RPA tools mainly run in a desktop client mode, users build automation processes through a graphical interface by dragging and dropping components, and compile the processes into a script language for execution. Typical products include UiPath, Blue Prism, PowerAutomate, etc. These tools generally have the following characteristics: dependence on desktop system environment: specific client needs to be installed, some need administrator permission or compatibility with specific operating systems; deployment and use are heavy: installation package needs to be downloaded, service needs to be registered, and running environment needs to be configured; user experience is engineering-oriented: the interface is complex, it is difficult for ordinary users to get started, and the maintenance and upgrade cost is high. At the same time, with the maturity and popularity of Web technology, especially the high market share of Chrome browser in the world, more and more business processes are gradually migrated to Web applications. The heavy desktop architecture of traditional RPA has been difficult to adapt to the lightweight, high-frequency, distributed automation needs of the Web era. Therefore, there is an urgent need for a lightweight RPA system that can run on the basis of browser plug-ins without relying on traditional desktop environments. SUMMARY

[0003] To solve the above problems in the prior art, the application provides a lightweight robot process automation method based on browser extension, and the purpose of the application can be achieved through the following technical scheme:

[0004] A lightweight robot process automation method based on browser extension, comprising:

[0005] S1: initializing a drawing environment based on an HTML5 canvas programming interface, preprocessing node style resources through off-screen canvas technology; defining flow node set and node connection relationship set data structure; binding mouse and keyboard input events through an event system, processing drag and line event streams based on a reactive programming library, and constructing a user interaction response mechanism;

[0006] S2: the data association layer establishes a communication channel with the flow canvas through an event bus, defines node selection and parameter update event types; the user interface rendering adopts component-based design, constructs the interface based on a dynamic form component of a front-end framework, and realizes dynamic generation and destruction of form elements through a template engine; logically encapsulating basic operation instructions, and realizing syntax verification of instruction code based on a regular expression;

[0007] S3: define instruction data format based on standardized data receiving interface; perform structural verification on the instruction through a syntax analyzer, process parameter matching and logic verification based on a semantic verifier, and realize conversion of the instruction into executable code through an intermediate code generator; format convert the intermediate code based on the requirements of flow execution; define error feedback data structure to support visual feedback of the analysis result;

[0008] S4: access and monitor the current web document object model based on the content script technology of the browser extension program; establish a mapping relationship library of element operation and robot process automation instruction based on the standardized element information data structure, preset the corresponding operation instruction templates of clicking, inputting and selecting, and realize automatic conversion of the capture result into executable instructions;

[0009] S5: obtain the intermediate code through the standardized flow data interface to realize the lightweight execution kernel; monitor the execution of the flow steps and the processing of asynchronous operations through the asynchronous non-blocking mode design; record the flow execution state through the state machine model, and realize global sharing and real-time updating of the execution state based on the state management library.

[0010] As a preferred technical solution of the application, the initialization of the drawing environment in S1 requires initialization of the coordinate system and the viewport, and the method is:

[0011] The basic coordinate system of the canvas is defined, and the left upper corner of the canvas element is set as the origin by default, the right is the positive direction of the horizontal axis, and the downward is the positive direction of the vertical axis, which is consistent with the screen coordinate system; the viewport parameters are initialized, the field of view object is set to store the current view state, including the scaling ratio and the translation offset; the canvas is scaled and translated based on the scaling ratio and the translation offset; the mouse coordinates on the canvas element are converted into world coordinate system coordinates based on the coordinate conversion tool function; the interaction operation coordinates are calculated through the world coordinate system coordinates; the canvas background grid is initialized, the light grid lines are drawn, the dashed line style is set, and the viewport is output.

[0012] Specifically, the method for preprocessing the node style resource in S1 through the off-screen canvas technology is:

[0013] A canvas object matching the size of the target node is instantiated through the off-screen canvas technology, a two-dimensional drawing context is obtained based on the calling method; based on the system preset node type, the node background area is filled by using the rectangular filling method, the node border is drawn by using the stroke rectangle drawing method, the corresponding icon is cropped and rendered from the icon based on the image drawing method, and the node name identifier is added by using the text filling method; the main canvas draws the pre-drawing node style to the target canvas through the image drawing interface, generates a binary object based on the canvas conversion method, and caches the rendering result of the two-dimensional drawing context.

[0014] Specifically, the method for binding mouse and keyboard input events in S1 through the event system is:

[0015] The mouse event is bound through the event listener, the user triggers the interaction starting point recognition by pressing the mouse on the canvas, the coordinates of the mouse in the canvas are obtained, the consistency of the click position and the hit node is judged based on the coordinates, and the initial state is recorded; based on the response of mouse movement, the mouse displacement difference is calculated, the node coordinates are updated, the connection preview track is refreshed, the canvas is redrawn by requesting animation frames; based on the release of the mouse button, the end process of the interactive operation is triggered, the function resets the dragging state, saves the temporary connection to the data array, and synchronously updates the data; the shortcut key support for undo, copy and paste operations is realized by binding the key down event, and the conflict prevention program is used to avoid conflict with the default behavior of the browser;

[0016] The mouse event coordinates are converted into canvas world coordinates, the interaction state is managed through state variables, and the coherence of the event processing logic is realized; based on high-frequency triggered events, the event triggering frequency is limited through throttling; based on canvas reset, all events are unbound to avoid memory leaks; the event behavior differences of different browsers are realized by adding compatibility judgment.

[0017] Specifically, the interface is constructed based on the dynamic form component of the front-end framework in S2, including:

[0018] The reusable basic form component is encapsulated based on the front-end framework, covering text input components, drop-down selection components, numerical input components, and batch import components; the basic form component design follows the single responsibility principle, handles its own rendering and basic interaction; based on the received configuration parameters, the data changes are transmitted through callback functions;

[0019] Based on the basic form component, the received parameters are initialized, the field configuration is obtained through array parsing, and the storage is in the form of state variables; the corresponding basic component is selected through the attribute matching of the field to dynamically select the component; based on the initialization form data, the initial value of the field is determined, and the default value is supported to be overwritten from the outside;

[0020] Based on the node type selected by the user in the canvas, the attribute panel obtains the node type and requests the corresponding form mode; based on the monitoring of the changes of the form, the dynamic form container re-renders the form structure, retains the valid field data, clears the invalid fields, and integrates the dynamic form component into the attribute panel.

[0021] Specifically, the working mode of the syntax analyzer and the semantic verifier in S3 includes:

[0022] Based on the principle of instruction naming specification, the syntax rule system is defined, the rule system is converted into an executable regular expression, the instruction code transmitted by the attribute panel is received, the test method is called to perform regular matching, and it is verified whether the instruction conforms to the three-level naming structure; by secondary analysis of special instructions, it is detected whether the expression format is legal; the instruction structure is split, and it is verified whether the naming constraint is met; if the verification fails, the parsing is immediately terminated and the syntax error is marked; based on the syntax error, structured error information is generated, the error information is pushed to the attribute panel through the event bus, and the error field is displayed in the interface and the correction suggestion is displayed;

[0023] Predefine parameter templates for various instructions, store them in the instruction template library, and include a list of required parameters, optional parameter constraints, and logical verification functions; based on instruction code matching corresponding parameter templates, traverse the template array, check whether the instruction contains all required parameters, and record the missing parameters; for optional parameters that are not configured, automatically fill in the default values in the template; perform consistency detection on the parameter value type and the template, and based on the inconsistency of the detection results, attempt automatic conversion, and if conversion fails, mark a type error.

[0024] Specifically, the implementation of the visual feedback in S3 is:

[0025] Based on the failure of regular matching or the failure of structure verification, a syntax error dataset is generated by calling an error creation model; based on the error code generated by the intermediate code generator, the error code type, parameters and logic are detected to generate an error dataset; based on the failure of parameter injection or template assembly, the failed instruction template and parameter information are recorded; based on the collected error dataset, repeated errors of nodes are removed, the latest errors are retained, and error information of associated fields is aggregated and marked to output feedback information; based on the state management of the front-end framework, the feedback information is received, the error list is stored, and the errors are displayed on the canvas to realize visual feedback.

[0026] Specifically, the method for establishing the mapping relationship library in S4 is:

[0027] Based on the standardized element information data structure generated by element capture, the input basis of the mapping relationship library is clear, based on the web interaction scene, the common element operation types are sorted out; based on the combination of element type and operation type, the mapping rule is defined, and the core structure of the relationship library is constructed;

[0028] The relational database data is stored in an object notation format, mapping rule arrays are constructed based on the core structure, and parameter dynamic conversion is realized through template strings and function calls;The mapping rule arrays are stored as independent object notation files, and are loaded through an import process based on element capture initialization;An index table based on tag names is constructed, matching mapping rules are found according to element information and operation types, and document object models are generated by calling instruction generation methods.

[0029] Specifically, the method for accessing and monitoring the current webpage document object model in S5 is:

[0030] The injection rules of the content script are configured in the file of the extension program, the content script is dynamically injected through the background script of the extension program, the parent-child relationship and sibling elements of the elements are collected by traversing the document object model tree through obtaining single elements, the document object is accessed through the content document access instruction based on the embedded framework, the attribute, text content and style information of the element are obtained, the operation trajectory of the user is recorded by binding the mouse and keyboard interaction events, the components of the extension program that cannot be directly accessed by the content script are adapted by listening to the changes of the document object model structure, and the data interaction is realized through the construction of the message passing mechanism.

[0031] Specifically, the global sharing and real-time updating of the execution state in S5 are realized by:

[0032] Based on the execution characteristics of the robot automation process, a globally shared execution state model is designed to standardize the structure of the stored state, the node state is mapped through a node name mapping method, and the error and log are saved using an immutable data structure based on an independent array storage model, and the node is queried through the association of the node name;

[0033] According to the front-end framework and business requirements, a state management library suitable for adaptation is selected, the execution state slice is defined, and the state initial value and updating method are encapsulated: asynchronous operations are processed through asynchronous functions to realize asynchronous updating of the state, the canvas component subscribes to the execution state through the application programming interface of the state management library to realize real-time updating, the state linkage between different components is realized through the state management library, the node state of the process canvas component is updated, and the execution panel component realizes the display of the latest progress, the key execution state is persisted to the local storage through the middleware, the state change history is tracked based on the debugging tool of the state management library, and problem positioning is simplified.

[0034] The beneficial effects of the present application are:

[0035] The whole process is realized based on a browser extension program without deploying a backend server. Through the carrier of Chrome Extension plug-in, users only need to drag and drop to complete the installation, which greatly reduces the deployment complexity and hardware requirements of traditional RPA systems, and meets the low-threshold automation needs of individual users and small teams. The execution kernel realized based on JavaScript has small volume and fast start, and can complete process execution and debugging directly in the browser environment, avoiding the cumbersome installation and resource occupation problems of traditional client software.

[0036] The node style resources are preprocessed through the off-screen Canvas technology to reduce the calculation consumption during real-time rendering, and the affected area is only redrawn through the incremental rendering mechanism, so that the process canvas can still maintain smooth dragging and connection operation when the number of nodes is large. The triggering frequency of high-frequency events is limited through throttling processing, and the accuracy and continuity of node dragging, connection and other interactive operations are ensured through coordinate conversion and state management mechanism. The event behavior differences of different browsers are handled through compatibility judgment to ensure stable operation in mainstream Chrome browsers and avoid functional abnormalities caused by browser environment differences.

[0037] Based on the dynamic form component encapsulated by the front-end framework, the parameter editing interface can be automatically adapted according to the node type, supporting various configuration methods such as text input, drop-down selection and batch import. Combined with the form cache and reuse mechanism, the node parameter configuration efficiency is greatly improved. The visual process canvas constructed by HTML5 Canvas supports intuitive operations such as node dragging and connection editing, and users can complete automation process design without mastering complex programming languages, greatly reducing the learning threshold of RPA technology.

[0038] Through the syntax analyzer and semantic verifier, the instruction structure and parameter logic are double-checked, and combined with the custom error feedback data structure, the error position is accurately located and repair suggestions are provided. The intermediate code generator converts the user-configured instructions into code compatible with the JavaScript execution standard, and combined with the lightweight execution engine, the process is efficiently executed, supporting single-step debugging, breakpoint setting and other functions to improve the process debugging efficiency.

[0039] Based on the standardized element information data structure, the element ID, XPath, CSS selector and other multi-dimensional locators are extracted, and the optimal positioning strategy is generated combined with the priority sorting mechanism, so that the element recognition success rate is greatly improved, effectively dealing with the dynamic change scenario of the webpage. The mapping relationship library of preset element operations and RPA instructions supports automatic conversion of common operations such as clicking, inputting and selecting, and users can generate corresponding automation instructions through simple web interaction.

[0040] Based on the state management library, the global sharing of the execution state is realized, the process progress, node state, error information and other data are synchronized to the monitoring panel in real time, and the user can intuitively master the process execution. Through the state persistence mechanism, the state loss caused by page refresh is avoided, combined with Redux DevTools and other debugging tools to support state backtracking and problem positioning, and the process execution troubleshooting time is shortened. Asynchronous non-blocking execution improves resource utilization: the execution engine is designed in an asynchronous non-blocking mode, browser resources are reasonably allocated through task queues and concurrent control mechanisms, and the resource occupation of multiple process parallel execution is reduced.

[0041] In summary, the application builds a low-threshold, high-efficiency and high-reliability Web automation solution through lightweight architecture design, high-performance interaction optimization, intelligent process configuration and precise execution monitoring, which can be widely used in Web scenarios such as form filling, data entry and file download, and significantly improves the work efficiency and automation experience of users. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to facilitate those skilled in the art to understand, the application will be further described below in conjunction with the drawings.

[0043] Figure 1 A flowchart of a lightweight robotic process automation method based on browser extension of the application. DETAILED DESCRIPTION

[0044] In order to further illustrate the technical means and effects adopted by the application to achieve the predetermined application purpose, the specific embodiments, structures, features and effects according to the application are described in detail below in conjunction with the drawings and preferred embodiments.

[0045] Please refer to Figure 1 A lightweight robotic process automation method based on browser extension, comprising:

[0046] S1: initialize the drawing environment based on the HTML5 canvas programming interface, preprocess the node style resources through the off-screen canvas technology; define the data structure of the process node set and the connection relationship set between nodes; bind the mouse and keyboard input events through the event system, process the drag and line event stream based on the responsive programming library, and construct the user interaction response mechanism;

[0047] S2: the data association layer establishes a communication channel with the process canvas through the event bus, defines the node selection and parameter update event types; the user interface rendering adopts component-based design, constructs the interface based on the dynamic form components of the front-end framework, and realizes the dynamic generation and destruction of form elements through the template engine; logically encapsulate the basic operation instructions, and realize the syntax verification of instruction code based on regular expressions;

[0048] S3: define the instruction data format based on the standardized data receiving interface; perform structural checking on the instruction through a syntax analyzer, process parameter matching and logic verification based on a semantic checker, and realize the conversion of the instruction into executable code through an intermediate code generator; format convert the intermediate code based on the requirements of flow execution; define an error feedback data structure to support the visual feedback of the analysis result;

[0049] S4: based on the content script technology of the browser extension program, access and monitor the current web document object model; based on the standardized element information data structure, establish a mapping relationship library of element operation and robot process automation instruction, preset the corresponding operation instruction templates of clicking, inputting and selecting, and realize the automatic conversion of the capture result into executable instructions;

[0050] S5: obtain the intermediate code through the standardized flow data interface to realize the lightweight execution kernel; monitor the execution of the flow steps and the processing of asynchronous operations through the asynchronous non-blocking mode design; record the flow execution state through the state machine model, and realize the global sharing and real-time updating of the execution state based on the state management library.

[0051] Specifically, the initialization of the drawing environment in S1 requires the initialization of the coordinate system and the viewport, and the method is:

[0052] The basic coordinate system of the canvas is defined, and the left top corner of the canvas element is set as the origin by default, the right direction is set as the positive direction of the horizontal axis, and the downward direction is set as the positive direction of the vertical axis, which is consistent with the screen coordinate system; the viewport parameters are initialized, the field of view object is set to store the current view state, including the scaling ratio and the translation offset; the canvas is scaled and translated based on the scaling ratio and the translation offset; the mouse coordinates on the canvas element are converted into world coordinate system coordinates based on the coordinate conversion tool function; the interaction operation coordinates are calculated through the world coordinate system coordinates; the canvas background grid is initialized, the light grid lines are drawn, the dashed line style is set, and the viewport is output.

[0053] In the page layout of the flow canvas in this embodiment, a canvas carrier of the visual editing area is first created. The canvas is based on the HTML5 Canvas element, and the physical size is set to 1200 pixels wide and 800 pixels high. Through style setting, it is made to adapt to the container width and height in the browser, and a 1-pixel gray border is added to the edge as a visual boundary.

[0054] The origin of the base coordinate system is fixed at the top-left vertex of the canvas element, the horizontal direction to the right is the positive direction of the horizontal axis (X axis), and the vertical direction downward is the positive direction of the vertical axis (Y axis), which is consistent with the default coordinate system of the computer screen, ensuring that the user's visual perception and operation logic are unified. The unit of measurement of the coordinate system is pixel (px), and all node positions, sizes, and interaction coordinates are calculated based on this unit. After initialization, the system records and outputs the configuration information of the base coordinate system to confirm that the origin position, axis direction, and other core parameters are correct.

[0055] To realize the scaling and translation functions of the canvas, a viewport state management mechanism needs to be established. In the initial state, the scaling ratio of the viewport is set to 1.0 (i.e., 1:1 original ratio), the horizontal offset (X axis) and the vertical offset (Y axis) are both 0, ensuring that the canvas content is fully displayed in the initial field of view.

[0056] To avoid excessive scaling affecting the editing experience, the scaling range is limited: the minimum scaling ratio is 0.5 (reduced to 50% of the original size), and the maximum scaling ratio is 2.0 (enlarged to 200% of the original size). At the same time, the interaction state of the viewport is recorded through state markers, including whether it is in the drag and translate mode, etc., providing a basis for subsequent operation logic. These parameters are stored in the global canvas state object, which facilitates real-time access and modification.

[0057] The scaling operation of the canvas is triggered by the mouse wheel event. When the user scrolls the wheel forward, the viewport is enlarged by 10% each time; when scrolling backward, it is reduced by 10% each time. During the scaling process, the system automatically checks whether the new ratio is within the effective range of 0.5-2.0, and if it is outside the range, it ignores the operation, and if it is valid, it updates the scaling parameters and triggers the canvas redraw, making the content display at the new ratio.

[0058] The translation operation is realized by dragging the mouse in the blank area of the canvas. When the user presses the left mouse button in the blank area of the canvas, the system activates the translation mode and records the initial mouse position; when moving the mouse, the difference (horizontal and vertical displacement) between the current position and the initial position is calculated and added to the offset parameters of the viewport; when releasing the left mouse button, the translation mode is closed. After each displacement update, the canvas is redrawn immediately to ensure that the content shifts in real time with the mouse movement.

[0059] Specifically, the method for preprocessing node style resources in S1 through off-screen canvas technology is:

[0060] A canvas object matching the size of the target node is instantiated through an off-screen canvas technology, a two-dimensional drawing context is obtained based on a calling method, based on a system preset node type, a node background area is filled by using a rectangular filling method, a node border is drawn by using a rectangle drawing method, a corresponding icon is cropped and rendered from an icon based on an image drawing method, and a node name identifier is added by using a text filling method; the main canvas draws the pre-drawing node style to the target canvas through an image drawing interface, generates a binary object based on a canvas conversion method, and caches the rendering result of the two-dimensional drawing context.

[0061] Specifically, the method for binding mouse and keyboard input events through the event system in S1 is:

[0062] The mouse event is bound through the event listener, the user triggers the interaction starting point recognition by pressing the mouse on the canvas, the coordinates of the mouse in the canvas are obtained, the consistency of the click position and the hit node is judged based on the coordinates, and the initial state is recorded; based on the response of the mouse movement, the mouse displacement difference is calculated, the node coordinates are updated, the connection line preview track is refreshed, and the canvas is redrawn by requesting an animation frame; based on the release of the mouse button, the interaction operation end process is triggered, the function resets the dragging state, saves the temporary connection line to the data array, and synchronously updates the data; the shortcut key support for undo, copy and paste operations is realized by binding the key down event, and the conflict between the shortcut key and the default behavior of the browser is avoided based on the anti-collision program;

[0063] The mouse event coordinates are converted into canvas world coordinates, the interaction state is managed through state variables, and the coherence of the event processing logic is realized; the event triggering frequency is limited through throttling based on high-frequency triggered events; memory leakage is avoided by unbinding all events based on canvas reset; the event behavior differences of different browsers are realized by adding compatibility judgment.

[0064] In this embodiment, after the flow canvas is loaded, the system automatically starts the mouse event listening mechanism. When the user moves the mouse to the canvas area and presses the left button, the mouse down event is triggered: the canvas first records the accurate position of the current mouse in the canvas (taking the top left corner of the canvas as the origin, the horizontal as the X axis, and the vertical as the Y axis coordinate value), and then checks whether the position coincides with the area of the existing flow node through a coordinate matching algorithm. If the click position hits a node (such as the blue rectangular area of the “click operation” node), the system immediately marks the node as “selected state”, and records the initial click coordinates and the current position of the node, preparing for the subsequent dragging operation; if the blank area is clicked, it is determined as the starting point of the canvas translation operation, and the translation starting coordinates are recorded.

[0065] When the user holds down the left mouse button and moves, the mouse move event is continuously triggered. The system calculates the displacement difference between the current mouse position and the initial click position in real time: if in the node dragging state, the node will be updated in synchronization with the mouse movement, and the connection around the node will be adjusted in real time to maintain the connection state, forming a smooth visual following effect; if in the canvas translation state, the entire canvas background and all nodes and connections will be translated in the direction of mouse movement, ensuring that the view always focuses on the operation area. To avoid interface lag caused by frequent drawing, the system only redraws the canvas every 100 milliseconds, ensuring the smoothness of visual updates through the request animation frame mechanism, even if the mouse is moved quickly, the interface can still remain stable.

[0066] When the user releases the left mouse button, the mouse release event triggers the end of the interaction process. If it was in the node dragging state before, the system will save the final position of the node to the flow data, and at the same time cancel the "selected state" highlight effect of the node; if it is in the connection drawing state (such as dragging from the anchor point of one node to another node), the temporary connection is formally saved to the connection data array, completing the logical association of the two nodes. After all state updates, the system will refresh the canvas display synchronously to ensure that the visual effect is consistent with the data state, and at the same time clear the temporary markers in the dragging process to prepare for the next interaction.

[0067] Specifically, the interface of the dynamic form component based on the front-end framework in S2 is constructed, including:

[0068] Encapsulate reusable basic form components based on the front-end framework, covering text input components, drop-down selection components, numerical input components, and batch import components; the design of the basic form component follows the single responsibility principle, handles its own rendering and basic interaction; based on the received configuration parameters, data changes are passed through callback functions;

[0069] Initialize the received parameters based on the basic form component, obtain the field configuration through array parsing, and store it in the form of a state variable; dynamically select components by matching the attributes of the fields with the corresponding basic components; based on the initialization form data, determine the initial value of the field, and support overriding the default value from the outside;

[0070] Based on the node type selected by the user in the canvas, the attribute panel obtains the node type and requests the corresponding form mode; based on the changes in the monitored form, the dynamic form container re-renders the form structure, retains valid field data, clears invalid fields, and integrates the dynamic form component into the attribute panel.

[0071] Specifically, the working mode of the syntax analyzer and the semantic verifier in S3 includes:

[0072] The syntax rule system is defined based on the instruction naming specification principle, the rule system is converted into an executable regular expression, instruction code transmitted by the attribute panel is received, a test method is called to perform regular matching, and it is verified whether the instruction conforms to the three-level naming structure; the expression format is detected whether it is legal by secondary analysis on the special instruction; the instruction structure is split, and it is verified whether the naming constraint is met; if the verification fails, the parsing is immediately terminated and a syntax error is marked; based on the syntax error, structured error information is generated, the error information is pushed to the attribute panel through an event bus, and the error field is displayed in the interface and a correction suggestion is displayed;

[0073] The parameter templates of various instructions are predefined and stored in an instruction template library, including a list of mandatory parameters, optional parameter constraints, and logical verification functions; based on instruction code matching corresponding parameter templates, the template array is traversed, it is checked whether the instruction contains all mandatory parameters, and if not, it is recorded, and for unconfigured optional parameters, the default value in the template is automatically filled; the parameter value type and the template are consistent, and based on the inconsistency of the detection result, automatic conversion is attempted, and type error marking is performed if the conversion fails.

[0074] In this embodiment, the system predefines the instruction naming specification as a three-level structure of "namespace, module, method", for example, in "rpa.web.click", "rpa" is the namespace, "web" is the module, and "click" is the operation method. Based on the specification, a regular expression rule is generated, which requires that each level of name only contains lowercase letters, numbers and underscores, and each level must be separated by a period. At the same time, for special instructions of process control class (such as "rpa.flow.if"), an expression nesting rule is additionally defined, which allows the instruction parameters to contain condition statements in the format of "${variable name>threshold value}".

[0075] When the user inputs the instruction code "rpa.webclick" in the attribute panel and submits, the syntax analyzer starts the verification process: first, the regular matching method is called to detect the input content, it is found that "webclick" is not separated by a period according to the specification as "web.click", it is determined that the three-level structure is incomplete. Immediately terminate the parsing and mark it as a syntax error, generate structured information containing error type ("incomplete structure"), error location (instruction code input box), current value ("rpa.webclick"), and automatically generate a correction suggestion "please supplement the separator between the module and the method, the correct format example: rpa.web.click.

[0076] Specifically, the implementation of the visual feedback in S3 is:

[0077] Based on the failure of regular matching or the failure of structure verification, a syntax error dataset is generated by calling an error creation model; based on the error code generated by the intermediate code generator, the error code type, parameters and logic are detected to generate an error dataset; based on the failure of parameter injection or template assembly, the failed instruction template and parameter information are recorded; based on the collected error dataset, repeated errors of nodes are removed to retain the latest errors, and errors of associated fields are aggregated and marked for output feedback information; based on the front-end framework state management, the feedback information is received, the error list is stored, and errors are displayed on the canvas to realize visual feedback.

[0078] Specifically, the method for establishing the mapping relationship library in S4 is:

[0079] Based on the standardized element information data structure generated by element capture, the input basis of the mapping relationship library is determined, and common element operation types are sorted based on web interaction scenarios; based on the combination of element types and operation types, mapping rules are defined, and the core structure of the relationship library is constructed;

[0080] The relationship library data is stored in the object notation format, the mapping rule array is constructed based on the core structure, and parameter dynamic conversion is realized through template strings and function calls; the mapping rule array is stored as an independent object notation file, and is loaded based on element capture initialization through an import process; an index table based on tag names is constructed, the matching mapping rule is found according to the element information and operation type, and the document object model is generated by calling the instruction generation method.

[0081] In this embodiment, when the element tag is input and the type attribute is text number, and the operation type is input, the matching rule 2 is: the instruction template is rpa.web.input, and the parameter mapping includes the locator (id=searchInput), the input value (the search keyword input by the user), and the input before emptying switch (default on). The rule specially marks that the disabled attribute of the element needs to be verified as false to ensure that the input operation is valid.

[0082] When the element tag is select and the operation type is select, the matching rule 3 is: the parameter mapping includes the locator, the selected item (the product category name selected by the user), and the matching method (text matching). The rule supports the value attribute or the text content of the option tag as the selection basis.

[0083] These rules collectively constitute the core structure of the relationship library, and each rule includes four parts of a unique identifier (such as rule_click_001), a trigger condition (an element tag + an operation type + a property constraint), an instruction template and a parameter mapping logic.

[0084] The mapping rules described above are stored in object notation format, and a mapping rule array containing 3 rules is constructed. Each rule is described in the form of a key-value pair, for example, the storage structure of rule 1 is:

[0085] Trigger condition: {"tagName": ["button", "a"], "attributes": {"type": ["button", "submit", null]}, "interactionType": "click"}

[0086] Instruction template: "rpa.web.click"

[0087] Parameter mapping: {"target": "${locators.primaryLocator}", "timeout": 5000, "waitForElement": true}

[0088] The array is stored as an independent element-mapping-rules.json file, which is loaded through the import process when the element capture module is initialized. During the loading process, the file format integrity is automatically checked, and if a missing required field (such as instructionTemplate) is detected, a warning is triggered and the default rule is used instead.

[0089] Specifically, the method for accessing and monitoring the current web document object model in S5 is:

[0090] Configure the injection rules of the content script in the file of the extension program, dynamically inject the content script through the background script of the extension program; collect the parent-child relationship and sibling elements of the elements by traversing the document object model tree through the acquisition of individual elements; access the document object based on the embedded framework, and obtain the attribute, text content and style information of the element; record the user's operation trajectory by binding the mouse and keyboard interaction events; adapt the components of the extension program that the content script cannot directly access by listening to the changes in the document object model structure; realize the interaction of data through the construction of a message passing mechanism.

[0091] In this embodiment, first, the injection rule of the content script is specified in the core configuration file of the browser extension. The staff sets the applicable webpage range of the script in the configuration file of the extension program, such as supporting all websites or specifying a certain type of platform, and specifies the script file name to be injected and the injection time, usually after the webpage DOM is loaded to avoid capture failure due to element not being loaded. When the user starts the RPA tool and opens the target webpage, the background script of the extension automatically detects the webpage state, and dynamically injects the content script into the current webpage environment when the condition is met, establishing a direct interaction channel between the extension program and the webpage DOM.

[0092] After the content script is injected, the element information collection process is started immediately. For the target elements that the user may operate, the query method provided by the document object model is used to accurately locate a single element, such as finding a button by ID or filtering an input box by tag name. At the same time, the hierarchical traversal of the entire DOM tree is performed, and the system automatically records the parent-child relationship and the position of the sibling elements of each element, such as identifying that a certain button is located at the child node position of the form container, and its adjacent sibling element is an input box. For complex webpages containing embedded frames (iframes), the frame content document access interface is used to penetrate the frame hierarchy to obtain the element information in the nested page, including the basic attributes of the element such as the tag type, ID, class name, the current text content of the input box, the display text of the button, and the rendering information such as the size, color, and border style of the element, providing complete data support for subsequent element positioning and operation.

[0093] To track the user's operation behavior on the webpage, the content script binds a series of mouse and keyboard interaction events. When the user performs operations such as clicking, inputting, and selecting on the webpage, the event listener responds in real time: records the element position of the mouse click, and distinguishes different operations such as left-click button and right-click menu trigger; captures the keyboard input content, including the characters input in the text box, the delete operation, and the shortcut key combination. For example, when the user clicks the submit button, the system records the element identifier, operation time, and mouse coordinates; when filling in the content in the input box, the input text is synchronized in real time and the input box element information is marked. These operation data are arranged in chronological order into an operation track, clearly presenting the interaction process between the user and the webpage.

[0094] Specifically, the global sharing and real-time updating of the execution state in S5 are as follows:

[0095] Based on the robot automation process execution characteristics, a globally shared execution state model is designed to standardize the structure of the stored state; the node state is mapped through a node name mapping method; based on an independent array storage model, an immutable data structure is used to save errors and logs, and the node is queried through the association of the node name;

[0096] According to the front-end framework and the business requirement, a state management library is selected, an execution state slice is defined, and an initial value and an updating method of the state are encapsulated; asynchronous operations are processed through an asynchronous function, and asynchronous updating of the state is realized; the canvas component subscribes to the execution state through an application programming interface of the state management library, and real-time updating is realized; state linkage between different components is realized through the state management library, the node state of the process canvas component is updated, and the execution panel component realizes the latest progress display; key execution states are persisted to local storage through middleware, the state change history is tracked based on a debugging tool of the state management library, and problem positioning is simplified.

[0097] The above is only a preferred embodiment of the present application, and is not intended to limit the present application in any form. Although the present application has been disclosed as above with a preferred embodiment, it is not intended to limit the present application. Any person skilled in the art can make some changes or modifications to the above disclosed technical content to obtain equivalent embodiments with equivalent changes, without departing from the technical solution of the present application. Any modification, change, and modification of the above embodiments, which is based on the technical essence of the present application, is still within the scope of the technical solution of the present application.

Claims

1. A browser extension based lightweight robotic process automation method, characterized in that, Comprise: S1: initialize drawing environment based on HTML5 canvas programming interface, preprocess node style resources through off-screen canvas technology; Define flow node set and node connection relationship set data structure; bind mouse and keyboard input events through event system, handle drag and line events based on reactive programming library, and build user interaction response mechanism; S2: data association layer establishes communication channel with flow canvas through event bus, defines node selection and parameter update event types; user interface rendering adopts component-based design, builds interface based on dynamic form components of front-end framework, and realizes dynamic generation and destruction of form elements through template engine; encapsulate basic operation instructions, and realize syntax verification of instruction code based on regular expression; S3: define instruction data format based on standardized data receiving interface; check the structure of the instruction through the syntax analyzer, process parameter matching and logic verification based on the semantic verifier, and convert the instruction into executable code through the intermediate code generator; Format conversion is performed on the intermediate code based on the requirements of process execution; Define error feedback data structure to support visual feedback of parsing results; S4: based on the content script technology of browser extension program, access and monitor the current web document object model; based on the standardized element information data structure, establish the mapping relationship library of element operation and robot process automation instruction, preset the corresponding operation instruction template of click, input and selection, and realize automatic conversion of capture results to executable instructions; S5: get intermediate code through standardized process data interface to realize lightweight execution kernel; monitor the execution of process steps and asynchronous operation processing through asynchronous non-blocking mode design; Record the execution state of the process through the state machine model, and realize the global sharing and real-time updating of the execution state based on the state management library; wherein the global sharing and real-time updating of the execution state is: Based on the characteristics of robot automation process execution, design a globally shared execution state model to standardize the structure of the stored state; map the node state through the node name mapping method; save errors and logs using immutable data structures based on independent array storage model, and query nodes by associating node names; According to the front-end framework and business requirements, select the appropriate state management library, define the execution state slice, encapsulate the state initial value and update method: process asynchronous operations through asynchronous functions to realize asynchronous updating of the state; the canvas component subscribes to the execution state through the application programming interface of the state management library to realize real-time updating; realize the state linkage between different components through the state management library, and based on the node state update of the process canvas component, the execution panel component realizes the latest progress display; persist key execution states to local storage through middleware, and track state change history based on the debugging tool of the state management library to simplify problem positioning.

2. The method of claim 1, wherein, The initialization of the drawing environment in S1 needs to initialize the coordinate system and viewport, and the method is: Define the basic coordinate system of the canvas, with the upper left corner of the canvas element as the origin by default, the right direction as the positive direction of the horizontal axis, and the downward direction as the positive direction of the vertical axis, consistent with the screen coordinate system; initialize the viewport parameters, set the field of view object to store the current view state, including the scaling ratio and translation offset; Based on the scaling ratio and translation offset, the canvas is scaled and translated; based on the coordinate conversion tool function, the mouse coordinates on the canvas element are converted into world coordinate system coordinates; the interactive operation coordinates are calculated through the world coordinate system coordinates; the canvas background grid is initialized, the light grid lines are drawn, the dashed line style is set, and the viewport is output.

3. The method of claim 1, wherein, The method for preprocessing node style resources in S1 through off-screen canvas technology is: Through off-screen canvas technology, a canvas object matching the size of the target node is instantiated, a two-dimensional drawing context is obtained based on the calling method; based on the system preset node type, the node background area is filled using the rectangle filling method, the node border is drawn using the stroke rectangle drawing method, the corresponding icon is cropped and rendered from the icon based on the image drawing method, and the node name identifier is added through the text filling method; the main canvas draws the pre-drawn node style to the target canvas through the image drawing interface, generates a binary object based on the canvas conversion method, and caches the rendering result of the two-dimensional drawing context.

4. The method of claim 1, wherein, The method for binding mouse and keyboard input events through the event system in S1 is: Through the event listener, the mouse event is bound, the user triggers the interaction starting point recognition by pressing the mouse on the canvas, the coordinates of the mouse in the canvas are obtained, the consistency of the click position and the hit node is judged based on the coordinates, and the initial state is recorded; Based on the response of mouse movement, the mouse displacement difference is calculated, the node coordinates are updated, the connection preview track is refreshed, and the canvas is redrawn by requesting an animation frame; based on the release of the mouse button, the interactive operation end process is triggered, the function resets the dragging state, saves the temporary connection to the data array, and synchronously updates the data; through the binding of the key down event, the shortcut key support for undo, copy and paste operations is realized, and the conflict between the shortcut keys and the default behavior of the browser is avoided based on the anti-collision program; Convert the mouse event coordinates to the canvas world coordinates, manage the interaction state through state variables, and realize the coherence of the event handling logic; based on the high-frequency triggering event, the event triggering frequency is limited through throttling; based on the canvas reset, all events are unbound to avoid memory leakage; through the addition of compatibility judgment, the event behavior differences of different browsers are realized.

5. The method of claim 1, wherein, The interface is constructed based on the dynamic form component of the front-end framework in S2, including: Encapsulate reusable basic form components based on the front-end framework, covering text input components, drop-down selection components, numerical input components, and batch import components; the basic form component design follows the single responsibility principle, handles its own rendering and basic interaction; based on the received configuration parameters, data changes are passed through callback functions; Based on the received parameters of the basic form component initialization, the field configuration is obtained by parsing the array, and is stored in the form of a state variable; the corresponding basic component is selected through the attribute matching of the field, and the dynamic component selection is performed; the initial value of the field is determined based on the initialized form data, and the default value is supported to be covered from the outside; Based on the node type selected by the user in the canvas, the attribute panel obtains the node type and requests the corresponding form mode; based on the change of the monitored form, the dynamic form container re-renders the form structure, retains the valid field data, clears the invalid field, and integrates the dynamic form component into the attribute panel.

6. The method of claim 1, wherein, The working mode of the syntax analyzer and the semantic verifier in S3 comprises: Based on the instruction naming specification principle, a syntax rule system is defined, the rule system is converted into an executable regular expression, instruction codes transmitted by the attribute panel are received, a test method is called to perform regular matching, it is verified whether the instruction conforms to the three-level naming structure, the expression format is detected through secondary parsing of special instructions, whether the naming constraint is met is verified by splitting the instruction structure, if the verification fails, the parsing is immediately terminated and a syntax error is marked, based on the syntax error, structured error information is generated, the error information is pushed to the attribute panel through an event bus, and the error field is displayed in the interface and a correction suggestion is displayed; Parameter templates of various instructions are predefined and stored in an instruction template library, including a list of mandatory parameters, optional parameter constraints and logical verification functions; based on the matching of the instruction code with the corresponding parameter template, the template array is traversed, it is checked whether the instruction contains all mandatory parameters, and if not, the missing parameters are recorded, the default values in the template are automatically filled for the unconfigured optional parameters; the consistency of the parameter value type and the template is detected, inconsistent detection results are automatically converted, and type error markers are generated for conversion failures.

7. The method of claim 1, wherein, The implementation of the visual feedback in S3 is: Based on the failure of regular matching or the failure of structure verification, a syntax error data set is generated by calling an error creation model; based on the error code generated by the intermediate code generator, the error code type, parameters and logic are detected, and an error data set is generated; Based on the failure of parameter injection or template assembly, the failed instruction template and parameter information are recorded; Based on the collected error data set, repeated errors of nodes are removed, the latest error is retained, errors of associated fields are aggregated, and feedback information is output; Based on the state management of the front-end framework, the feedback information is received, the error list is stored, and the error is displayed on the canvas to realize visual feedback.

8. The method of claim 1, wherein, The method for establishing the mapping relationship library in S4 is: Based on the standardized element information data structure generated by element capture, the input basis of the mapping relationship library is determined, common element operation types are sorted based on web interaction scenarios; based on the combination of element types and operation types, mapping rules are defined, and the core structure of the relationship library is constructed; The relationship library data is stored in the object notation format, the mapping rule array is constructed based on the core structure, and the parameter dynamic conversion is realized through the template string and function call. The mapping rule array is stored as an independent object representation file, loaded through an import flow based on element capture initialization; an index table based on a tag name is constructed, a matching mapping rule is looked up according to element information and an operation type, and a document object model is generated by calling an instruction generation method.

9. The method of claim 1, wherein, The method for accessing and monitoring the current webpage document object model in S5 is: Configuring injection rules of the content script in a file of the extension program, and dynamically injecting the content script through a background script of the extension program; Collecting parent-child relationships and sibling elements of the elements by traversing the document object model tree through acquisition of single elements; Based on an embedded framework, accessing a document object to acquire attribute, text content and style information of the elements through a content document access instruction; recording operation trajectories of the user by binding mouse and keyboard interactive events; adapting components of the extension program that cannot be directly accessed by the content script by listening to changes in the document object model structure; and realizing data interaction through construction of a message passing mechanism.

Images