Web workflow generation method and device based on large language model, equipment and medium

Abstract

This invention provides a method, apparatus, device, and medium for generating web workflows based on a large language model, relating to the field of browser technology. The method includes: acquiring user-inputted requirement information and acquiring browser context data corresponding to the browser; acquiring a constraint template for the browser context data, inputting the requirement information and browser context data into a large language model for prediction, and constraining the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information; acquiring an executable operation context, inputting the executable operation context and workflow plan into the large language model for prediction to obtain an executable atomic operation sequence corresponding to the requirement information; cyclically executing the executable atomic operation sequence, and converting the executable atomic operation sequence into an automated task template after the execution of the executable atomic operation sequence is completed.

Description

Technical Field

[0001] This invention relates to the field of browser technology, and in particular to a web workflow generation method based on a large language model, a web workflow generation device based on a large language model, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Browser automation refers to the technology of controlling web browsers to perform various operations programmatically, commonly used in scenarios such as testing, data scraping, and task automation. In browser automation, developers need to write scripts to control the browser, which offers high flexibility but requires programming skills and has high debugging and maintenance costs. Alternatively, actions can be recorded by clicking to generate corresponding executable scripts or processes. While this lowers the technical barrier, the generated code is often not robust enough to handle dynamic pages or heavy workloads, failing to meet actual user needs. Summary of the Invention

[0003] The present invention provides a method, apparatus, electronic device, and computer-readable storage medium for generating web workflows based on a large language model, in order to solve or partially solve the problem of high usage threshold in browser automation processes.

[0004] This invention discloses a method for generating web workflows based on a large language model, comprising:

[0005] Obtain the user's input request information, as well as the browser context data corresponding to the browser;

[0006] Obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and constrain the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information.

[0007] Obtain the executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction to obtain the executable atomic operation sequence corresponding to the requirement information;

[0008] The executable atomic operation sequence is executed repeatedly, and when the executable atomic operation sequence has been completed, it is converted into an automated task template.

[0009] In some feasible implementations, the browser context data includes at least a page routing table. The step of inputting the requirement information and the browser context data into the large language model for prediction, and constraining the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information, includes:

[0010] The requirement information and the page routing table are input into the large language model for prediction, and the output of the large language model is constrained by the constraint template during the prediction process to obtain the workflow plan corresponding to the requirement information.

[0011] In some feasible implementations, the page routing table includes at least one of page path, page title, and at least one sub-page. The step of inputting the requirement information and the page routing table into the large language model for prediction, and constraining the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information, includes:

[0012] Determine the page path, the page title, and the page path context corresponding to the sub-page;

[0013] The page path context and the requirement information are packaged to obtain the first input data;

[0014] The first input data is input into the large language model for prediction, and the output of the large language model is constrained by the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information.

[0015] In some feasible implementations, the browser context data includes a DOM tree, and obtaining the executable operation context corresponding to the browser context data includes:

[0016] The DOM tree is shrunk to obtain the corresponding target DOM tree;

[0017] Obtain the operation type corresponding to the browser;

[0018] The target DOM tree is shrunk according to the operation type to obtain an executable operation context;

[0019] The operation type includes at least one of the following: click trigger, double-click trigger, right-click trigger, jump trigger, input trigger, and text acquisition.

[0020] In some feasible implementations, the step of reducing the DOM tree to obtain the corresponding target DOM tree includes:

[0021] Starting from the root node of the DOM tree, traverse each node of the DOM tree level by level;

[0022] If the target node being traversed is an operable element or has text content, then the target node is retained;

[0023] If the target node being traversed is not the operable element or does not have text content, then delete the target node, save the color attribute corresponding to the target node, and remove other style attributes;

[0024] Assign a unique identifier to the retained nodes to obtain the corresponding target DOM tree.

[0025] In some feasible implementations, the executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the cyclic execution of the executable atomic operation sequence includes:

[0026] Obtain semantic tags, wherein the semantic tags include at least one of element tags, path tags, behavior tags, and execution target tags;

[0027] Obtain the target path context corresponding to the operation to be performed, and the target DOM element corresponding to the execution target from the target DOM tree;

[0028] The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain the corresponding second input data;

[0029] The second input data is input into the large language model to obtain the operable element corresponding to the target DOM element or the execution path corresponding to the target path context;

[0030] Obtain the operation function corresponding to the operable element or the execution path;

[0031] The operation function is used to perform the operation corresponding to the operable element or the execution path.

[0032] In some feasible implementations, the step of converting the executable atomic operation sequence into an automated task template after the executable atomic operation sequence has been executed includes:

[0033] The atomic operation sequence is transformed into a target sequence, which is composed of the operation function and a target value. The target value includes at least one of the following: DOM element, page path context, operation to be performed, and execution target.

[0034] By performing persistence operations on each of the target sequences and the constraint template, the corresponding automated task template is obtained.

[0035] This invention also discloses a web workflow generation device based on a large language model, comprising:

[0036] The data acquisition module is used to acquire the user's input requirements and the browser context data corresponding to the browser.

[0037] The workflow prediction module is used to obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and constrain the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information.

[0038] The operation sequence prediction module is used to obtain the executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction to obtain the executable atomic operation sequence corresponding to the requirement information;

[0039] An execution module is used to repeatedly execute the executable atomic operation sequence, and when the executable atomic operation sequence has been executed, convert the executable atomic operation sequence into an automated task template.

[0040] In some feasible implementations, the browser context data includes at least a page routing table, and the workflow prediction module is specifically used for:

[0041] The requirement information and the page routing table are input into the large language model for prediction, and the output of the large language model is constrained by the constraint template during the prediction process to obtain the workflow plan corresponding to the requirement information.

[0042] In some feasible implementations, the page routing table includes at least one of page path, page title, and at least one subordinate page, and the workflow prediction module is specifically used for:

[0043] Determine the page path, the page title, and the page path context corresponding to the sub-page;

[0044] The page path context and the requirement information are packaged to obtain the first input data;

[0045] The first input data is input into the large language model for prediction, and the output of the large language model is constrained by the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information.

[0046] In some feasible implementations, the browser context data includes the DOM tree, and the operation sequence prediction module is specifically used for:

[0047] The DOM tree is shrunk to obtain the corresponding target DOM tree;

[0048] Obtain the operation type corresponding to the browser;

[0049] The target DOM tree is shrunk according to the operation type to obtain an executable operation context;

[0050] The operation type includes at least one of the following: click trigger, double-click trigger, right-click trigger, jump trigger, input trigger, and text acquisition.

[0051] In some feasible implementations, the operation sequence prediction module is specifically used for:

[0052] Starting from the root node of the DOM tree, traverse each node of the DOM tree level by level;

[0053] If the target node being traversed is an operable element or has text content, then the target node is retained;

[0054] If the target node being traversed is not the operable element or does not have text content, then delete the target node, save the color attribute corresponding to the target node, and remove other style attributes;

[0055] Assign a unique identifier to the retained nodes to obtain the corresponding target DOM tree.

[0056] In some feasible implementations, the executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the execution module is specifically used for:

[0057] Obtain semantic tags, wherein the semantic tags include at least one of element tags, path tags, behavior tags, and execution target tags;

[0058] Obtain the target path context corresponding to the operation to be performed, and the target DOM element corresponding to the execution target from the target DOM tree;

[0059] The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain the corresponding second input data;

[0060] The second input data is input into the large language model to obtain the operable element corresponding to the target DOM element or the execution path corresponding to the target path context;

[0061] Obtain the operation function corresponding to the operable element or the execution path;

[0062] The operation function is used to perform the operation corresponding to the operable element or the execution path.

[0063] In some feasible implementations, the execution module is specifically used for:

[0064] The atomic operation sequence is transformed into a target sequence, which is composed of the operation function and a target value. The target value includes at least one of the following: DOM element, page path context, operation to be performed, and execution target.

[0065] By performing persistence operations on each of the target sequences and the constraint template, the corresponding automated task template is obtained.

[0066] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0067] The memory is used to store computer programs;

[0068] When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0069] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0070] The embodiments of the present invention have the following advantages:

[0071] In this embodiment of the invention, by acquiring user-inputted requirement information and browser context data, a constraint template for the browser context data is obtained. The requirement information and browser context data are then input into a large language model for prediction. During the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the requirement information. Then, the executable operation context corresponding to the browser context data is acquired, and the executable operation context and workflow plan are input into the large language model for prediction to obtain an executable atomic operation sequence corresponding to the requirement information. The executable atomic operation sequence is executed cyclically, and after the executable atomic operation sequence is completed, it is transformed into an automated task template. Thus, based on the user-input requirement information and the acquired relevant data, the browser automation process can be generated. Users do not need to write code, effectively reducing the technical threshold for automation. At the same time, the generated automated task template can automatically adapt to changes such as page layout adjustments, making the automated task template more stable. Subsequent similar tasks can be directly called and adjusted, realizing template reuse. Attached Figure Description

[0072] Figure 1 This is a flowchart illustrating the steps of a Web workflow generation method based on a large language model provided in an embodiment of the present invention.

[0073] Figure 2 This is a structural block diagram of a Web workflow generation device based on a large language model provided in an embodiment of the present invention. Detailed Implementation

[0074] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] As an example, in the process of browser automation, developers need to write scripts to control the browser. Although this is flexible, it requires programming skills and has high debugging and maintenance costs. Alternatively, by recording operations through clicks, corresponding executable scripts or processes can be generated. Although this lowers the technical threshold, the generated code is often not robust enough, making it difficult to handle dynamic pages or load logic and failing to meet the actual needs of users.

[0076] In this invention, user-inputted requirement information and browser context data are obtained. A constraint template for the browser context data is then acquired, and the requirement information and browser context data are input into a large language model for prediction. During prediction, the constraint template constrains the output of the large language model to obtain a workflow plan corresponding to the requirement information. Then, the executable operation context corresponding to the browser context data is obtained, and the executable operation context and workflow plan are input into the large language model for prediction to obtain an executable atomic operation sequence corresponding to the requirement information. This sequence is executed iteratively, and upon completion, it is transformed into an automated task template. Thus, based on the user-input requirement information and the acquired data, an automated browser process can be generated without requiring user coding, effectively lowering the technical barrier to automation. Furthermore, the generated automated task template can automatically adapt to changes such as page layout adjustments, making it more stable. Subsequent similar tasks can be directly called and adjusted, achieving template reuse.

[0077] Optionally, the large language model can be used for requirements understanding, process planning, and operation instruction generation. In the process of requirements understanding and process planning, by inputting the user's natural language description and browser environment data, the large language model can understand the user's real needs through semantic analysis and generate corresponding operation steps by combining webpage structure knowledge. In the process of operation instruction generation, the planned workflow and list of operable elements on the page are processed, and then the corresponding standardized operation instructions are input. Thus, the large language model can create reliable browser automation processes while maintaining the ability to handle complex business scenarios.

[0078] Reference Figure 1 The diagram illustrates a flowchart of a web workflow generation method based on a large language model provided in an embodiment of the present invention, which may specifically include the following steps:

[0079] Step 101: Obtain the user's input request information and the browser context data corresponding to the browser;

[0080] The requirement information can be the task requirements described by the user in natural language, while the browser context data can be a collection of dynamic information in the current web page environment that may affect the execution of automated tasks, which fully describes the browser's operable state at a specific moment.

[0081] Browser context data can include data related to the structure, interaction, navigation, dynamics, and environment. In specific implementations, at the structural level, it can include the DOM tree, i.e., the document object model structure of the current page, such as all interactive elements (e.g., buttons / input boxes / drop-down menus), key content containers (e.g., tables / lists / cards), and element hierarchy and visibility status; at the interaction level, it can include control metadata, such as element type (button / input / select), interaction attributes (disabled / readonly status), unique identifiers (id / class / aria-label), and visual features (position / size / color contrast); at the navigation level, it can include corresponding routing information, such as the current URL and history, available navigation paths (including nested routes), and page function descriptions (mapped through a route configuration table); at the dynamic level, it can include runtime states, such as network request status (AJAX / WebSocket active connections), data loading progress (lazy loading / pagination loading), and front-end framework status (React / Vue component status); at the environment level, it includes browser characteristics, such as window size and scroll position, cookie / local storage status, and content injected by extensions, etc., which are not limited in this invention.

[0082] Step 102: Obtain the constraint template for the browser context data, input the requirement information and the browser context data into the large language model for prediction, and constrain the output of the large language model through the constraint template during the prediction process to obtain the workflow plan corresponding to the requirement information;

[0083] In this embodiment of the invention, the constraint template can be a combination of a set of predefined rules, which can be used to limit the scope and format of the content generated by the large language model to ensure that the output meets business logic and user needs.

[0084] For example, a constraint template could be [{name:'', action:'', target:''}], where name can be a title; action can be an atomic operation type, including click, scroll, double-click, right-click, select, go, input, get text, etc.; and target can be a target description, etc. By obtaining the corresponding constraint template, during the process of inputting requirement information and browser context data into the large language model for prediction, the output of the large language model is constrained by the constraint template, resulting in a workflow plan corresponding to the requirement information. On the one hand, the constraint template constrains the output range and content of the large language model, ensuring that the output conforms to business logic and user needs; on the other hand, the large language model generates the corresponding workflow plan without requiring users to write corresponding code, effectively reducing the technical threshold.

[0085] In some feasible implementations, browser context data includes at least a page routing table. In this case, the requirement information and the page routing table can be input into a large language model for prediction. During the prediction process, the output of the large language model is constrained by a constraint template to obtain a workflow plan corresponding to the requirement information. This ensures that the output range and content of the large language model are constrained by the constraint template, thus ensuring that the output meets the business logic and user requirements. On the other hand, the large language model generates the corresponding workflow plan without requiring users to write corresponding code, effectively reducing the technical threshold.

[0086] The page routing table includes at least one of the following: page path, page title, and at least one subordinate page. Therefore, the page path context corresponding to the page path, page title, and subordinate page can be determined first. Then, the page path context and requirement information are packaged to obtain the first input data. The first input data is then input into the large language model for prediction. During the prediction process, the output of the large language model is constrained by the constraint template to obtain the workflow plan corresponding to the requirement information.

[0087] The page routing table format can be as follows:

[0088]

[0089] In this document, URL can be a page path, name can be a page title, description can be a detailed description of the page's functions, and children can be the page's sub-pages, i.e., the sub-pages that can be accessed from the current page. This invention does not impose any restrictions on these.

[0090] Once the page path context and requirement information are determined, they can be packaged, for example, by... <routes-context>Wrap the page path context, through <task>The system retrieves the package requirements information, obtains the corresponding first input data, and then submits the first input data to the large language model for prediction. The output format of the large language model is constrained by the constraint template [{name:'', action:'', target:''}] to obtain the corresponding workflow plan. This process ensures that the output range and content of the large language model conform to business logic and user needs by constraining the constraint template, and that the large language model generates the corresponding workflow plan without requiring users to write any code, effectively reducing the technical threshold.

[0091] Step 103: Obtain the executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction to obtain the executable atomic operation sequence corresponding to the requirement information;

[0092] After obtaining the workflow plan, the executable operation context corresponding to the browser context data can be further obtained. Then, the executable operation context and the workflow plan are further input into the large language model for prediction to obtain the executable atomic operation sequence corresponding to the requirement information. The corresponding execution process is determined through the executable atomic operation sequence so that the corresponding automated process can be executed according to the executable atomic operation sequence in the future.

[0093] It should be noted that the workflow plan is used to determine the user's intent, the executable context is used to determine how to realize the user's intent (i.e., technical implementation adaptation), and the executable atomic operation sequence can be used to determine the final implementation process (i.e., machine-operable instructions). Thus, by determining the workflow plan, executable operation context, and executable atomic operation sequence, the layered processing maintains a user-friendly way of describing requirements, while ensuring the accuracy of machine execution, effectively ensuring the effectiveness and executability of the automated process.

[0094] In some feasible implementations, browser context data may also include the DOM tree. In this case, during the process of obtaining the executable operation context, the DOM tree can first be reduced to obtain the corresponding target DOM tree, and simultaneously the browser's corresponding operation type can be obtained. Then, the target DOM tree can be reduced according to the operation type to obtain the executable operation context. The operation type must include at least one of the following: click-triggered, double-click-triggered, right-click-triggered, jump-triggered, input-triggered, and text retrieval.

[0095] Optionally, by reducing the DOM tree, the impact of irrelevant data on the generation of automated processes can be effectively reduced, and corresponding data noise can be lowered. Specifically, the DOM tree can be deleted based on preset rules. Starting from the root node, each node in the DOM tree is traversed layer by layer. If the target node being traversed is an operable element or has text content, it is retained. If the target node is not an operable element or does not have text content, it is deleted, its corresponding color attribute is saved, other style attributes are removed, and then a unique identifier is assigned to the retained nodes to obtain the corresponding target DOM tree. Therefore, by reducing the DOM tree, not only can the data to be processed be simplified and processing efficiency improved, but the code structure can also be optimized, making it easier for users to understand the page structure and functionality, and improving code maintainability.

[0096] In some examples, firstly, a counter object `elementCounters` can be defined to record the frequency of each element type. This recording helps prepare for generating unique IDs for the elements later. Then, a `generateUniqueId` function is written, which takes an element as an argument and retrieves the element's tag name, converting it to lowercase. If `elementCounters` does not yet have a count for that tag name, it is initialized to 1. Next, a unique ID in the form of `elementType_SerialNumber` is generated based on the tag name and the count value, and the corresponding element type's count is incremented by 1. The core processing logic is in the `reduceDOMTree` function. When the passed node is an element node, a series of judgments and processes are performed to determine whether the element is an operable element (like a button, input box, etc.). The system checks whether the element is an operable element (link) or has a click event, and also checks if the element has text content. If the element is neither operable nor has text content, it is removed from its parent node.

[0097] Furthermore, for the elements that are retained, attribute processing can be performed. Specifically, the element's text content and calculated color attribute can be obtained first. Then, the element's text content can be set to the obtained text, and the color can be set to the obtained color. Simultaneously, all style attributes of the element except for color will be removed. For example, all style attributes of the element can be separated by semicolons, filtering out style attributes containing "color," and then the filtered style attributes can be recombine and set to the element. Afterward, the `generateUniqueId` function is called to generate a unique ID for the element and assign it to the element's `id` attribute. Finally, the `reduceDOMTree` function is recursively called on the element's child nodes to ensure that the entire DOM tree is processed according to the rules.

[0098] In the main program, the `reduceDOMTree` function is called starting from `document.body`, thus reducing the entire DOM tree. In terms of time complexity, since each node in the DOM tree needs to be traversed once, the time complexity is O(n), where n is the number of nodes in the DOM tree. As for space complexity, because the depth of the recursive call stack depends on the height of the DOM tree, the space complexity is O(h), where h is the height of the DOM tree.

[0099] Step 104: Execute the executable atomic operation sequence repeatedly, and when the executable atomic operation sequence has been completed, convert the executable atomic operation sequence into an automated task template.

[0100] Once the executable atomic operation sequence is determined, it can be executed cyclically. After the executable atomic operation sequence is completed, it is transformed into an automated task template. Based on the user's input requirements and the relevant data obtained, the browser's automated process can be generated. Users do not need to write code, which effectively lowers the technical threshold for automation. At the same time, the generated automated task template can automatically adapt to changes such as page layout adjustments, making the automated task template more stable. Subsequent similar tasks can be directly called and adjusted, realizing template reuse.

[0101] The executable atomic operation sequence includes at least one atomic operation sequence, which includes an execution operation and an execution target corresponding to the execution operation. The execution operation can include clicking, scrolling, double-clicking, right-click triggering, selection, jumping, etc. In the process of executing the executable atomic operation sequence, semantic tags can be obtained first. Semantic tags include at least one of element tags, path tags, behavior tags, and execution target tags. Then, the target path context corresponding to the execution operation and the target DOM element corresponding to the execution target are obtained from the target DOM tree. Next, the target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target to obtain the corresponding second input data. The second input data is then input into the large language model to obtain the operable element corresponding to the target DOM element or the execution path corresponding to the target path context. Finally, the operation function corresponding to the operable element or execution path is obtained, and then the operation function is used to execute the operation corresponding to the operable element or execution path.

[0102] Once the executable atomic operation sequence has been completed, it is transformed into a target sequence. The target sequence consists of an operation function and a target value. The target value includes at least one of the following: a DOM element, a page path context, an operation to be executed, or an execution target. Then, each target sequence and constraint template are persisted to obtain the corresponding automated task template. Based on the user's input requirements and the relevant data obtained, the browser's automated process can be generated. Users do not need to write code, which effectively lowers the technical threshold for automation. At the same time, the generated automated task template can automatically adapt to changes such as page layout adjustments, making the automated task template more stable. Subsequent similar tasks can be directly called and adjusted, realizing template reuse.

[0103] In one example, the process of repeatedly executing a sequence of atomic operations can be described as follows:

[0104] First, use specific semantic tags. <dom-context>Wrap the DOM element;

[0105] <routes-context>Wrap page path context;

[0106] <action>Includes behavior;

[0107] <target>Includes executable targets and input large models.

[0108] Then, the large model returns the operable elements (with ordinal numbers) in the dom-context or the URL path in the routes-context as the actual execution target, and then matches the pre-selected operation function, passes in the target value to complete the corresponding execution operation.

[0109] Optionally, for the above atomic operation sequences, users can select specific atomic operation sequences to delete in order to adjust according to actual needs, thereby improving the flexibility of automated process generation.

[0110] Furthermore, after execution, the atomic operation sequence can be converted into a target sequence of "operation function - target value" and persisted in JSON format. This allows subsequent users to select historically persisted tasks and quickly automate browser processes. For example, a constraint template like [{name:'Click the add button',action:'Click',target:'button_23',params:"},{name:'Enter project name',action:'Enter',target:'input_1',params:'construction project'}] can be persisted, allowing subsequent users to select historically persisted tasks and quickly execute the previously executed task.

[0111] It should be noted that the embodiments of the present invention include, but are not limited to, the examples described above. It is understood that those skilled in the art can make further settings according to actual needs under the guidance of the ideas in the embodiments of the present invention, and the present invention does not limit such settings.

[0112] In this embodiment of the invention, by acquiring user-inputted requirement information and browser context data, a constraint template for the browser context data is obtained. The requirement information and browser context data are then input into a large language model for prediction. During the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the requirement information. Then, the executable operation context corresponding to the browser context data is acquired, and the executable operation context and workflow plan are input into the large language model for prediction to obtain an executable atomic operation sequence corresponding to the requirement information. The executable atomic operation sequence is executed cyclically, and after the executable atomic operation sequence is completed, it is transformed into an automated task template. Thus, based on the user-input requirement information and the acquired relevant data, the browser automation process can be generated. Users do not need to write code, effectively reducing the technical threshold for automation. At the same time, the generated automated task template can automatically adapt to changes such as page layout adjustments, making the automated task template more stable. Subsequent similar tasks can be directly called and adjusted, realizing template reuse.

[0113] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are provided for illustrative purposes:

[0114] In one example, the automatic generation process of web workflows based on large language models can be as follows:

[0115] 1. User input intent, such as (wanting to complete a certain function, opening a certain page to add certain data).

[0116] 2. Prepare browser context data, browser DOM tree, and page routing table:

[0117] - The DOM tree is shrunk based on predefined rules, retaining only text, color, size, and style.

[0118] -The page routing table format is as follows:

[0119]

[0120] 3. <routes-context>Package Page Path Context <task>The user's intent is submitted as input to the large language model.

[0121] 4. Workflow plan for obtaining the output of the large language model.

[0122] 5. Configure the executable operation context and submit it to the large language model in conjunction with the workflow plan.

[0123] 6. Output executable atomic operation sequences: The atomic operation sequence contains an intent and a target. The intent includes click, scroll, double-click, right-click, select, and go; the target is a page element described in the language. Users can select a specific atomic operation sequence to delete.

[0124] 7. Execute atomic operation sequences in a loop: using specific semantic tags <dom-context>Wrap DOM elements <routes-context>Package Page Path Context <action>Includes behavior, <target>It contains an executable target and a large model. The large model returns an operable element (with an ordinal number) in the dom-context or a URL path in the routes-context as the actual execution target, then matches it with a pre-selected operation function, passes the target value, and completes the operation.

[0125] 8. Once the operation sequence is completed, the "operation function-target value sequence" translated into JSON format will be persisted. Users can select historical persistence tasks to quickly execute previous tasks.

[0126] Through the above process, browser automation processes can be generated based on user input and relevant data. Users do not need to write code, which effectively lowers the technical threshold for automation. At the same time, the generated automation task templates can automatically adapt to changes such as page layout adjustments, making the automation task templates more stable. Subsequent similar tasks can be directly called and adjusted, realizing template reuse.

[0127] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0128] Reference< / target> < / action> < / task> < / target> < / action> Figure 2 The diagram illustrates a structural block diagram of a Web workflow generation device based on a large language model provided in an embodiment of the present invention, which may specifically include the following modules:

[0129] The data acquisition module 201 is used to acquire the user's input requirement information and the browser context data corresponding to the browser.

[0130] The workflow prediction module 202 is used to obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and constrain the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information.

[0131] The operation sequence prediction module 203 is used to obtain the executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction to obtain the executable atomic operation sequence corresponding to the requirement information;

[0132] The execution module 204 is used to repeatedly execute the executable atomic operation sequence, and when the executable atomic operation sequence is completed, convert the executable atomic operation sequence into an automated task template.

[0133] In some feasible implementations, the browser context data includes at least a page routing table, and the workflow prediction module 202 is specifically used for:

[0134] The requirement information and the page routing table are input into the large language model for prediction, and the output of the large language model is constrained by the constraint template during the prediction process to obtain the workflow plan corresponding to the requirement information.

[0135] In some feasible implementations, the page routing table includes at least one of page path, page title, and at least one subordinate page, and the workflow prediction module 202 is specifically used for:

[0136] Determine the page path, the page title, and the page path context corresponding to the sub-page;

[0137] The page path context and the requirement information are packaged to obtain the first input data;

[0138] The first input data is input into the large language model for prediction, and the output of the large language model is constrained by the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information.

[0139] In some feasible implementations, the browser context data includes the DOM tree, and the operation sequence prediction module 203 is specifically used for:

[0140] The DOM tree is shrunk to obtain the corresponding target DOM tree;

[0141] Obtain the operation type corresponding to the browser;

[0142] The target DOM tree is shrunk according to the operation type to obtain an executable operation context;

[0143] The operation type includes at least one of the following: click trigger, double-click trigger, right-click trigger, jump trigger, input trigger, and text acquisition.

[0144] In some feasible implementations, the operation sequence prediction module is specifically used for:

[0145] Starting from the root node of the DOM tree, traverse each node of the DOM tree level by level;

[0146] If the target node being traversed is an operable element or has text content, then the target node is retained;

[0147] If the target node being traversed is not the operable element or does not have text content, then delete the target node, save the color attribute corresponding to the target node, and remove other style attributes;

[0148] Assign a unique identifier to the retained nodes to obtain the corresponding target DOM tree.

[0149] In some feasible implementations, the executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the execution module 204 is specifically used for:

[0150] Obtain semantic tags, wherein the semantic tags include at least one of element tags, path tags, behavior tags, and execution target tags;

[0151] Obtain the target path context corresponding to the operation to be performed, and the target DOM element corresponding to the execution target from the target DOM tree;

[0152] The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain the corresponding second input data;

[0153] The second input data is input into the large language model to obtain the operable element corresponding to the target DOM element or the execution path corresponding to the target path context;

[0154] Obtain the operation function corresponding to the operable element or the execution path;

[0155] The operation function is used to perform the operation corresponding to the operable element or the execution path.

[0156] In some feasible implementations, the execution module 204 is specifically used for:

[0157] The atomic operation sequence is transformed into a target sequence, which is composed of the operation function and a target value. The target value includes at least one of the following: DOM element, page path context, operation to be performed, and execution target.

[0158] By performing persistence operations on each of the target sequences and the constraint template, the corresponding automated task template is obtained.

[0159] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0160] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described embodiment of the Web workflow generation method based on a large language model and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0161] This invention also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described embodiment of the Web workflow generation method based on a large language model, achieving the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0162] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0163] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program code.

[0164] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0165] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0167] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0168] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0169] The foregoing has provided a detailed description of a Web workflow generation method and a Web workflow generation device based on a large language model provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.< / task>

Claims

1. A method for generating a web workflow based on a large language model, characterized in that, include: Obtain the user's input request information, as well as the browser context data corresponding to the browser; Obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and constrain the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information. Obtain the executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction to obtain the executable atomic operation sequence corresponding to the requirement information; The executable atomic operation sequence is executed repeatedly, and when the executable atomic operation sequence is completed, the executable atomic operation sequence is converted into an automated task template; The browser context data includes the DOM tree, and obtaining the executable operation context corresponding to the browser context data includes: The DOM tree is shrunk to obtain the corresponding target DOM tree; Obtain the operation type corresponding to the browser; The target DOM tree is shrunk according to the operation type to obtain an executable operation context; The operation type includes at least one of the following: click trigger, double-click trigger, right-click trigger, jump trigger, input trigger, and text acquisition; Wherein, the executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the cyclic execution of the executable atomic operation sequence includes: Obtain semantic tags, wherein the semantic tags include at least one of element tags, path tags, behavior tags, and execution target tags; Obtain the target path context corresponding to the operation to be performed, and the target DOM element corresponding to the execution target from the target DOM tree; The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain the corresponding second input data; The second input data is input into the large language model to obtain the operable element corresponding to the target DOM element or the execution path corresponding to the target path context; Obtain the operation function corresponding to the operable element or the execution path; The operation function is used to perform the operation corresponding to the operable element or the execution path.

2. The method according to claim 1, characterized in that, The browser context data includes at least a page routing table. The process of inputting the requirement information and the browser context data into the large language model for prediction, and constraining the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information, includes: The requirement information and the page routing table are input into the large language model for prediction, and the output of the large language model is constrained by the constraint template during the prediction process to obtain the workflow plan corresponding to the requirement information.

3. The method according to claim 2, characterized in that, The page routing table includes at least one of page path, page title, and at least one sub-page. The step of inputting the requirement information and the page routing table into the large language model for prediction, and constraining the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information, includes: Determine the page path, the page title, and the page path context corresponding to the sub-page; The page path context and the requirement information are packaged to obtain the first input data; The first input data is input into the large language model for prediction, and the output of the large language model is constrained by the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information.

4. The method according to claim 1, characterized in that, The step of reducing the DOM tree to obtain the corresponding target DOM tree includes: Starting from the root node of the DOM tree, traverse each node of the DOM tree level by level; If the target node being traversed is an operable element or has text content, then the target node is retained; If the target node being traversed is not the operable element or does not have text content, then delete the target node, save the color attribute corresponding to the target node, and remove other style attributes; Assign a unique identifier to the retained nodes to obtain the corresponding target DOM tree.

5. The method according to claim 1, characterized in that, The step of converting the executable atomic operation sequence into an automated task template after the executable atomic operation sequence has been executed includes: The atomic operation sequence is transformed into a target sequence, which is composed of the operation function and a target value. The target value includes at least one of the following: DOM element, page path context, operation to be performed, and execution target. By performing persistence operations on each of the target sequences and the constraint template, the corresponding automated task template is obtained.

6. A Web workflow generation device based on a large language model, characterized in that, include: The data acquisition module is used to acquire the user's input requirements and the browser context data corresponding to the browser. The workflow prediction module is used to obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and constrain the output of the large language model through the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information. The operation sequence prediction module is used to obtain the executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction to obtain the executable atomic operation sequence corresponding to the requirement information; An execution module is used to repeatedly execute the executable atomic operation sequence, and when the executable atomic operation sequence has been executed, convert the executable atomic operation sequence into an automated task template; The browser context data includes the DOM tree, and the operation sequence prediction module is specifically used for: The DOM tree is shrunk to obtain the corresponding target DOM tree; Obtain the operation type corresponding to the browser; The target DOM tree is shrunk according to the operation type to obtain an executable operation context; The operation type includes at least one of the following: click trigger, double-click trigger, right-click trigger, jump trigger, input trigger, and text acquisition; Wherein, the executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the execution module is specifically used for: Obtain semantic tags, wherein the semantic tags include at least one of element tags, path tags, behavior tags, and execution target tags; Obtain the target path context corresponding to the operation to be performed, and the target DOM element corresponding to the execution target from the target DOM tree; The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain the corresponding second input data; The second input data is input into the large language model to obtain the operable element corresponding to the target DOM element or the execution path corresponding to the target path context; Obtain the operation function corresponding to the operable element or the execution path; The operation function is used to perform the operation corresponding to the operable element or the execution path.

7. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method as described in any one of claims 1-5.

8. A computer-readable storage medium having instructions stored thereon that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-5.

Images