Web workflow generation method and device based on large language model, equipment and medium

Abstract

The embodiment of the invention provides a Web workflow generation method and device based on a large language model, equipment and a medium, and relates to the technical field of browsers, the method comprises the following steps: obtaining demand information input by a user, and obtaining browser context data corresponding to a browser; obtaining a constraint template for the browser context data, inputting the demand information and the browser context data into the large language model for prediction, and constraining the output of the large language model through the constraint template in the prediction process to obtain a workflow plan corresponding to the demand information; obtaining an executable operation context, inputting the executable operation context and the workflow plan into a large language model for prediction, and obtaining an executable atomic operation sequence corresponding to the demand information; and circularly executing the executable atomic operation sequence, and converting the executable atomic operation sequence into an automatic task template under the condition that the execution of the executable atomic operation sequence is completed.

Description

Technical Field

[0001] The present invention relates to the field of browser technology, and in particular to a method for generating a Web workflow based on a large language model, a device for generating a Web workflow based on a large language model, an electronic device, and a computer-readable storage medium. Background Art

[0002] Browser automation refers to the technology of programmatically controlling web browsers to perform various operations. It is commonly used in scenarios such as testing, data capture, and task automation. During browser automation, developers must write scripts to control the browser. While this offers flexibility, it requires programming skills and is costly to debug and maintain. Alternatively, developers can record operations by clicking on them to generate executable scripts or processes. While this lowers the technical barrier to entry, the generated code is often not robust enough to handle dynamic pages or load logic, failing to meet actual user needs. Summary of the Invention

[0003] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for generating a web workflow based on a large language model, so as to solve or partially solve the problem of high threshold for use of the browser automation process.

[0004] The embodiment of the present invention discloses a method for generating a web workflow based on a large language model, comprising:

[0005] Obtain the user's input requirements and the browser context data corresponding to the browser;

[0006] Obtaining a constraint template for the browser context data, inputting the requirement information and the browser context data into a large language model for prediction, and constraining an output of the large language model by the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information;

[0007] Obtaining an executable operation context corresponding to the browser context data, and inputting the executable operation context and the workflow plan into the large language model for prediction to obtain an executable atomic operation sequence corresponding to the demand information;

[0008] The executable atomic operation sequence is executed cyclically, and when the executable atomic operation sequence is completed, the executable atomic operation sequence is converted into an automation task template.

[0009] In some feasible implementations, the browser context data includes at least a page routing table. Inputting the requirement information and the browser context data into the large language model for prediction, and constraining the output of the large language model using the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information includes:

[0010] The demand information and the page routing table are input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information.

[0011] In some feasible implementations, the page routing table includes at least one of a page path, a page title, and at least one subordinate page. Inputting the demand information and the page routing table into the large language model for prediction, and constraining the output of the large language model using the constraint template during the prediction process to obtain a workflow plan corresponding to the demand information includes:

[0012] Determine the page path, the page title, and the page path context corresponding to the subordinate page;

[0013] Packaging the page path context and the demand information to obtain first input data;

[0014] The first input data is input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information.

[0015] In some feasible implementations, the browser context data includes a DOM tree, and obtaining the executable operation context corresponding to the browser context data includes:

[0016] Performing reduction processing on the DOM tree to obtain a corresponding target DOM tree;

[0017] Obtaining the operation type corresponding to the browser;

[0018] Reduce the target DOM tree according to the operation type to obtain an executable operation context;

[0019] The operation type includes at least one of click triggering, double-click triggering, right-click triggering, jump triggering, input triggering and text acquisition.

[0020] In some feasible implementations, the reducing the DOM tree to obtain the corresponding target DOM tree includes:

[0021] Starting from the root node of the DOM tree, traverse each node of the DOM tree layer by layer;

[0022] If the target node currently being traversed is an operable element or has text content, the target node is retained;

[0023] If the currently traversed target node is not the operable element or does not have text content, the target node is deleted, and the color attribute corresponding to the target node is saved, and other style attributes are removed;

[0024] Assign a corresponding unique identifier to the retained node and obtain the corresponding target DOM tree.

[0025] In some feasible implementations, the executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the looping execution of the executable atomic operation sequence includes:

[0026] Acquire a semantic tag, where the semantic tag includes at least one of an element tag, a path tag, a behavior tag, and an execution target tag;

[0027] Obtaining a target path context corresponding to the execution operation and a target DOM element corresponding to the execution target from the target DOM tree;

[0028] The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain corresponding second input data;

[0029] Inputting the second input data into the large language model to obtain an operable element corresponding to the target DOM element or an execution path corresponding to the target path context;

[0030] Obtaining an operation function corresponding to the operable element or the execution path;

[0031] The operation function is used to perform an operation corresponding to the operable element or the execution path.

[0032] In some feasible implementations, when the executable atomic operation sequence is completed, converting the executable atomic operation sequence into an automated task template includes:

[0033] Convert the atomic operation sequence into a target sequence, where the target sequence consists of the operation function and a target value, where the target value includes at least one of a DOM element, a page path context, an execution operation, and an execution target;

[0034] Each of the target sequences and the constraint template is used to perform a persistence operation to obtain a corresponding automation task template.

[0035] The embodiment of the present invention further discloses a Web workflow generation device based on a large language model, comprising:

[0036] The data acquisition module is used to obtain the required information input by the user and the browser context data corresponding to the browser;

[0037] a workflow prediction module, configured to obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and during the prediction process constrain the output of the large language model using the constraint template to obtain a workflow plan corresponding to the requirement information;

[0038] an operation sequence prediction module, configured to obtain an executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction, thereby obtaining an executable atomic operation sequence corresponding to the requirement information;

[0039] The execution module is used to cyclically execute the executable atomic operation sequence, and when the executable atomic operation sequence is completed, convert the executable atomic operation sequence into an automation task template.

[0040] In some feasible implementations, the browser context data includes at least a page routing table, and the workflow prediction module is specifically configured to:

[0041] The demand information and the page routing table are input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information.

[0042] In some feasible implementations, the page routing table includes at least one of a page path, a page title, and at least one subordinate page, and the workflow prediction module is specifically configured to:

[0043] Determine the page path, the page title, and the page path context corresponding to the subordinate page;

[0044] Packaging the page path context and the demand information to obtain first input data;

[0045] The first input data is input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information.

[0046] In some feasible implementations, the browser context data includes a DOM tree, and the operation sequence prediction module is specifically configured to:

[0047] Performing reduction processing on the DOM tree to obtain a corresponding target DOM tree;

[0048] Obtaining the operation type corresponding to the browser;

[0049] Reduce the target DOM tree according to the operation type to obtain an executable operation context;

[0050] The operation type includes at least one of click triggering, double-click triggering, right-click triggering, jump triggering, input triggering and text acquisition.

[0051] In some feasible implementations, the operation sequence prediction module is specifically configured to:

[0052] Starting from the root node of the DOM tree, traverse each node of the DOM tree layer by layer;

[0053] If the target node currently being traversed is an operable element or has text content, the target node is retained;

[0054] If the currently traversed target node is not the operable element or does not have text content, the target node is deleted, and the color attribute corresponding to the target node is saved, and other style attributes are removed;

[0055] Assign a corresponding unique identifier to the retained node and obtain the corresponding target DOM tree.

[0056] In some feasible implementations, the executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the execution module is specifically configured to:

[0057] Acquire a semantic tag, where the semantic tag includes at least one of an element tag, a path tag, a behavior tag, and an execution target tag;

[0058] Obtaining a target path context corresponding to the execution operation and a target DOM element corresponding to the execution target from the target DOM tree;

[0059] The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain corresponding second input data;

[0060] Inputting the second input data into the large language model to obtain an operable element corresponding to the target DOM element or an execution path corresponding to the target path context;

[0061] Obtaining an operation function corresponding to the operable element or the execution path;

[0062] The operation function is used to perform an operation corresponding to the operable element or the execution path.

[0063] In some feasible implementations, the execution module is specifically configured to:

[0064] Convert the atomic operation sequence into a target sequence, where the target sequence consists of the operation function and a target value, where the target value includes at least one of a DOM element, a page path context, an execution operation, and an execution target;

[0065] Each of the target sequences and the constraint template is used to perform a persistence operation to obtain a corresponding automation task template.

[0066] An embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0067] The memory is used to store computer programs;

[0068] The processor is configured to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0069] An embodiment of the present invention further discloses a computer-readable storage medium having instructions stored thereon. When executed by one or more processors, the processors are enabled to execute the method according to the embodiment of the present invention.

[0070] The embodiments of the present invention include the following advantages:

[0071] In an embodiment of the present invention, by obtaining requirement information input by a user and browser context data corresponding to the browser, then obtaining a constraint template for the browser context data, inputting the requirement information and browser context data into a large language model for prediction, and constraining the output of the large language model through the constraint template during the prediction process, a workflow plan corresponding to the requirement information is obtained, and then an executable operation context corresponding to the browser context data is obtained, and the executable operation context and the workflow plan are input into the large language model for prediction, to obtain an executable atomic operation sequence corresponding to the requirement information, and then the executable atomic operation sequence is executed cyclically. After the executable atomic operation sequence is executed, the executable atomic operation sequence is converted into an automated task template, thereby enabling the generation of a browser automation process based on the requirement information input by the user and the obtained related data. The user does not need to write code, which effectively lowers the technical threshold for automation. At the same time, the generated automated task template can automatically adapt to changes such as page layout adjustment, making the automated task template more stable, and subsequent similar tasks can be directly called and adjusted, thereby achieving template reuse. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 This is a flowchart of a method for generating a Web workflow based on a large language model provided in an embodiment of the present invention;

[0073] Figure 2 This is a structural block diagram of a Web workflow generation device based on a large language model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0075] For example, during browser automation, developers need to write scripts to control the browser. While this provides flexibility, it requires programming skills and has high debugging and maintenance costs. Alternatively, they can generate executable scripts or processes by clicking and recording operations. Although this lowers the technical threshold, the generated code is often not robust enough and has difficulty handling dynamic pages or load logic, failing to meet users' actual needs.

[0076] In this regard, in the present invention, by obtaining the demand information input by the user and the browser context data corresponding to the browser, then obtaining a constraint template for the browser context data, and inputting the demand information and the browser context data into a large language model for prediction, and in the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information, and then obtaining the executable operation context corresponding to the browser context data, and inputting the executable operation context and the workflow plan into the large language model for prediction, obtaining an executable atomic operation sequence corresponding to the demand information, and executing the executable atomic operation sequence in a loop, and when the executable atomic operation sequence is completed, converting the executable atomic operation sequence into an automated task template, thereby realizing the generation of a browser automation process based on the demand information input by the user and the relevant data obtained, the user does not need to write code, which effectively lowers the technical threshold of automation, and at the same time, the generated automated task template can automatically adapt to changes such as page layout adjustment, making the automated task template more stable, and subsequent similar tasks can be directly called and adjusted, realizing template reuse.

[0077] Optionally, the large language model can be used for demand understanding, process planning, and operation instruction generation. In the process of demand understanding and process planning, by inputting the user's natural language description and browser environment data, the large language model can understand the user's real needs through semantic analysis, and generate corresponding operation steps based on the web page structure knowledge; in the process of generating operation instructions, it is processed through the planned workflow and the list of operable elements on the page, and then the corresponding standardized operation instructions are input, so that a reliable browser automation process can be created through the large language model while maintaining the ability to handle complex business scenarios.

[0078] Reference Figure 1 , shows a flowchart of a method for generating a Web workflow based on a large language model provided in an embodiment of the present invention, which may specifically include the following steps:

[0079] Step 101: Obtaining user inputted requirement information and browser context data corresponding to the browser;

[0080] As for demand information, it can be task requirements described by users in natural language, etc., while browser context data can be a collection of all dynamic information in the current web page environment that may affect the execution of automated tasks, which fully describes the operational status of the browser at a specific moment, etc.

[0081] The browser context data may include data related to the structure level, interaction level, navigation level, dynamic level, and environment level. In a specific implementation, at the structural level, it may include a DOM tree, i.e., the document object model structure of the current page, such as all interactive elements (such as buttons / input boxes / drop-down menus, etc.), key content containers (such as tables / lists / cards, etc.), and element hierarchical relationships and visibility states; at the interaction level, it may include control metadata, such as element type (button / input / select, etc.), interaction attributes (disabled / readonly, etc.), unique identifiers (id / class / aria-label, etc.), and visual features (position / size / color contrast); at the navigation level, it may include corresponding routing information, such as the current URL and history, available navigation paths (including nested routes), and page function descriptions (mapped through routing configuration tables); at the dynamic level, it may include runtime status, such as network request status (AJAX / WebSocket active connections), data loading progress (lazy loading / paged loading), and front-end framework status (React / Vue component status); at the environmental level, it may include browser characteristics, such as window size and scroll position, cookie / local storage status, and extension injected content, etc., which are not limited by the present invention.

[0082] Step 102: Obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and constrain the output of the large language model using the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information.

[0083] In an embodiment of the present invention, the constraint template may be a combination of a set of predefined rules, which may be used to limit the scope and format of content generated by a large language model to ensure that the output complies with business logic and user needs.

[0084] For example, a constraint template can be [{name:'', action:'', target:''}], where name can be a title; action can be an atomic operation type, including click, scroll, double-click, right-click, select, go to, input, get text, etc.; target can be a target description, etc., so that by obtaining the corresponding constraint template, when inputting the demand information and browser context data into the large language model for prediction, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information. On the one hand, the output scope and content of the large language model are constrained by the constraint template to ensure that the output conforms to business logic and user needs. On the other hand, the corresponding workflow plan is generated by the large language model without the need for users to write corresponding code, effectively lowering the technical threshold.

[0085] In some feasible implementations, the browser context data includes at least a page routing table. The demand information and the page routing table can then be input into a large language model for prediction. During the prediction process, the output of the large language model can be constrained by a constraint template to obtain a workflow plan corresponding to the demand information. Thus, on the one hand, the output range and content of the large language model can be constrained by the constraint template to ensure that the output complies with business logic and user needs. On the other hand, the corresponding workflow plan is generated by the large language model without the need for users to write corresponding code, effectively lowering the technical threshold.

[0086] Among them, the page routing table includes at least one of the page path, page title and at least one subordinate page. Then, the page path context corresponding to the page path, page title and subordinate page can be determined first, and then the page path context and demand information are packaged to obtain the first input data, and then the first input data is input into the large language model for prediction. In the prediction process, the output of the large language model is constrained by the constraint template to obtain the workflow plan corresponding to the demand information.

[0087] Among them, the page routing table format can be as follows:

[0088]

[0089] Among them, URL can be the page path, name: can be the page title, description: can be a detailed description of the page function, children can be the subordinate pages of the page, that is, the subordinate pages that can be accessed through the current page, etc. The present invention does not limit this.

[0090] After determining the page path context and requirement information, you can first package the page path context and requirement information, for example, you can use <routes-context>Wrap the page path context, via <task>Package demand information is obtained, the corresponding first input data is obtained, and then the first input data is submitted to the large language model for prediction. The output format of the large language model is constrained by the constraint template [{name:'', action:'', target:''}] to obtain the corresponding workflow plan. On the one hand, the output scope and content of the large language model are constrained by the constraint template to ensure that the output meets the business logic and user needs. On the other hand, the corresponding workflow plan is generated by the large language model without the need for users to write corresponding code, which effectively lowers the technical threshold.

[0091] Step 103: Obtain an executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction to obtain an executable atomic operation sequence corresponding to the requirement information;

[0092] After obtaining the workflow plan, we can further obtain the executable operation context corresponding to the browser context data, and then further input the executable operation context and the workflow plan into the large language model for prediction to obtain the executable atomic operation sequence corresponding to the demand information. The corresponding execution process is determined through the executable atomic operation sequence, so that the corresponding automation process can be executed subsequently according to the executable atomic operation sequence.

[0093] It should be noted that the workflow plan is used to determine the user's intention, the executable context is used to determine how to realize the user's intention (i.e., technical implementation adaptation), and the executable atomic operation sequence can be used to determine the final implementation process (i.e., machine-operable instructions). Therefore, by determining the workflow plan, executable operation context, and executable atomic operation sequence, the hierarchical processing maintains a friendly demand description method, while ensuring the accuracy of machine execution, effectively ensuring the effectiveness and executability of the automation process.

[0094] In some feasible implementations, the browser context data may also include a DOM tree. In the process of obtaining the executable operation context, the DOM tree may be first reduced to obtain a corresponding target DOM tree. The browser's corresponding operation type is then obtained. The target DOM tree is then reduced based on the operation type to obtain the executable operation context. The operation type includes at least one of click triggering, double-click triggering, right-click triggering, jump triggering, input triggering, and text acquisition.

[0095] Optionally, by reducing the DOM tree, the impact of irrelevant data on the generation of automated processes can be effectively reduced, reducing the corresponding data noise. The DOM tree can be deleted based on preset rules. Specifically, starting from the root node of the DOM tree, each node of the DOM tree can be traversed layer by layer. If the target node currently traversed is an operable element or has text content, the target node is retained; if the target node currently traversed is not an operable element or does not have text content, the target node is deleted, and the color attribute corresponding to the target node is saved, other style attributes are removed, and then a corresponding unique identifier is assigned to the retained node to obtain the corresponding target DOM tree. By reducing the DOM tree, not only can the data to be processed be simplified and processing efficiency be improved, but the code structure can also be optimized, making it easier for users to understand the page structure and function, and improving the maintainability of the code.

[0096] In some examples, first, you can define a counter object elementCounters, which can record the number of times each element type appears. By recording the corresponding number of times, you can prepare for the subsequent generation of a unique ID for the element. Then, write a generateUniqueId function that can receive an element as a parameter, and obtain the tag name of the element and then convert it to lowercase. If there is no count corresponding to the tag name in elementCounters, it will be initialized to 1. Then, based on the tag name and count value, a unique ID in the form of element type_serial number is generated, and the count value of the corresponding element type is increased by 1. The core processing logic is in the reduceDOMTree function. When the passed-in node is an element node, a series of judgments and processing will be performed to determine whether the element is an operable element (such as a button, input box, Link) or whether it has a click event. On the other hand, it checks whether the element has text content. If it is neither an actionable element nor has text content, the node is removed from its parent node.

[0097] In addition, for the retained elements, attribute processing can be performed. Specifically, the text content and calculated color attribute of the element can be obtained first, and then the text content of the element is set to the obtained text, and the color is set to the obtained color. At the same time, other style attributes of the element except color will be removed. For example, all style attributes of the element can be separated by semicolons, and style attributes containing "color" can be filtered out. The filtered style attributes are then recombined and set to the element. After that, the generateUniqueId function is called to generate a unique ID for the element and assigned to the element's id attribute. Finally, the reduceDOMTree function is recursively called on the element's child nodes to ensure that the entire DOM tree can be processed according to the rules.

[0098] In the main program, the reduceDOMTree function is called starting from document.body to reduce the entire DOM tree. Since each node in the DOM tree must be traversed once, the time complexity is O(n), where n is the number of nodes in the DOM tree. As for space complexity, since the depth of the recursive call stack depends on the height of the DOM tree, the space complexity is O(h), where h is the height of the DOM tree.

[0099] Step 104 : cyclically executing the executable atomic operation sequence, and converting the executable atomic operation sequence into an automation task template when the executable atomic operation sequence is completed.

[0100] After the executable atomic operation sequence is determined, the executable atomic operation sequence can be executed in a loop, and when the executable atomic operation sequence is completed, the executable atomic operation sequence can be converted into an automated task template, so that the browser automation process can be generated according to the user's input demand information and the relevant data obtained. The user does not need to write code, which effectively lowers the technical threshold of automation. At the same time, the generated automated task template can automatically adapt to changes such as page layout adjustment, making the automated task template more stable, and subsequent similar tasks can be directly called and adjusted, realizing template reuse.

[0101] Among them, the executable atomic operation sequence includes at least one atomic operation sequence, and the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation. The execution operation may include clicking, scrolling, double-clicking, right-click triggering, selecting, jumping, etc. Then, in the process of executing the executable atomic operation sequence, the semantic tag can be obtained first, and the semantic tag includes at least one of the element tag, the path tag, the behavior tag and the execution target tag. Then, the target path context corresponding to the execution operation and the target DOM element corresponding to the execution target are obtained from the target DOM tree. Then, the element tag is used to wrap the target DOM element, the path tag is used to wrap the target path context, the behavior tag is used to wrap the execution operation, and the execution target tag is used to wrap the execution target to obtain the corresponding second input data. The second input data is then input into the large language model to obtain the operable element corresponding to the target DOM element or the execution path corresponding to the target path context. Finally, the operation function corresponding to the operable element or the execution path is obtained, and then the operation function is used to execute the operation corresponding to the operable element or the execution path.

[0102] When the executable atomic operation sequence is completed, the atomic operation sequence is converted into a target sequence. The target sequence consists of an operation function and a target value. The target value includes at least one of a DOM element, a page path context, an execution operation, and an execution target. Then, each target sequence and a constraint template are used to perform a persistence operation to obtain the corresponding automated task template. In this way, the browser automation process can be generated based on the user input demand information and the relevant data obtained. The user does not need to write code, which effectively lowers the technical threshold of automation. At the same time, the generated automated task template can automatically adapt to changes such as page layout adjustments, making the automated task template more stable, and subsequent similar tasks can be directly called and adjusted, realizing template reuse.

[0103] In one example, the process of looping through a sequence of atomic operations may be as follows:

[0104] First, use specific semantic tags <dom-context>Wrap DOM element;

[0105] <routes-context>Wrap page path context;

[0106] <action>Contains behavior;

[0107] <target>Contains executable targets, input large models.

[0108] Then, the big model returns the operable element (with a serial number) in the dom-context, or the URL path in the routes-context as the actual execution target, and then matches the pre-selected operation function and passes in the target value to complete the corresponding execution operation.

[0109] Optionally, for the above atomic operation sequences, the user can select a specific atomic operation sequence to delete, so as to adjust it according to actual needs, thereby improving the flexibility of automated process generation.

[0110] Furthermore, after execution, the atomic operation sequence can be converted into a target sequence of "operation function-target value" and persisted in Json format, so that subsequent users can select historically persisted tasks and quickly perform browser automation processes. For example, the constraint template: [{name:'Click the new button', action:'Click', target:'button_23', params:"}, {name:'Enter project name', action:'Enter', target:'input_1', params:'Engineering construction project'}] can be used for persistent storage, so that subsequent users can select historically persisted tasks and quickly execute previous tasks.

[0111] It should be noted that the embodiments of the present invention include but are not limited to the above examples. It is understandable that those skilled in the art can also make settings according to actual needs under the guidance of the ideas of the embodiments of the present invention, and the present invention does not limit this.

[0112] In an embodiment of the present invention, by obtaining requirement information input by a user and browser context data corresponding to the browser, then obtaining a constraint template for the browser context data, inputting the requirement information and browser context data into a large language model for prediction, and constraining the output of the large language model through the constraint template during the prediction process, a workflow plan corresponding to the requirement information is obtained, and then an executable operation context corresponding to the browser context data is obtained, and the executable operation context and the workflow plan are input into the large language model for prediction, to obtain an executable atomic operation sequence corresponding to the requirement information, and then the executable atomic operation sequence is executed cyclically. After the executable atomic operation sequence is executed, the executable atomic operation sequence is converted into an automated task template, thereby enabling the generation of a browser automation process based on the requirement information input by the user and the obtained related data. The user does not need to write code, which effectively lowers the technical threshold for automation. At the same time, the generated automated task template can automatically adapt to changes such as page layout adjustment, making the automated task template more stable, and subsequent similar tasks can be directly called and adjusted, thereby achieving template reuse.

[0113] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are provided for illustrative purposes:

[0114] In one example, the process of automatically generating a web workflow based on a large language model can be as follows:

[0115] 1. User input intentions such as (hoping to complete a certain function, open a certain page and add certain data).

[0116] 2. Prepare browser context data, browser DOM tree, and page routing table:

[0117] -The DOM tree is reduced based on pre-set rules and only the text and color size styles are retained

[0118] -The page routing table format is as follows:

[0119]

[0120] 3. <routes-context>Wrapping page path context <task>The wrapped user intent is submitted as input to the large language model.

[0121] 4. Workflow plan for obtaining the output of the large language model.

[0122] 5. Configure the executable operation context and submit it to the large language model in combination with the workflow plan.

[0123] 6. Output executable atomic operation sequence: The atomic operation sequence contains intention and target. Intention includes click, scroll, double-click, right-click, select, and go; the target is the page element described by the language; users can select a specific atomic operation sequence to delete

[0124] 7. Looping through a sequence of atomic operations: using specific semantic tags <dom-context>Wrapping DOM elements <routes-context>Wrapping page path context <action>Contains behavior, <target>Contains executable targets and large models. The large model returns the operable element (with a serial number) in the dom-context or the URL path in the routes-context as the actual execution target, then matches the pre-selected operation function and passes in the target value to complete the operation.

[0125] 8. After the operation sequence is executed, the "operation function-target value sequence" translated into it will be persisted in JSON format. Users can quickly execute previous tasks by selecting historical persistence tasks.

[0126] Through the above process, the browser automation process can be generated based on the user's input demand information and the relevant data obtained. Users do not need to write code, which effectively lowers the technical threshold of automation. At the same time, the generated automation task template can automatically adapt to changes such as page layout adjustments, making the automation task template more stable, and subsequent similar tasks can be directly called and adjusted, realizing template reuse.

[0127] It should be noted that for the sake of simplicity, the method embodiments are described as a series of actions. However, those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0128] Reference< / target> < / action> < / task> < / target> < / action> Figure 2 , shows a structural block diagram of a Web workflow generation device based on a large language model provided in an embodiment of the present invention, which may specifically include the following modules:

[0129] The data acquisition module 201 is used to acquire the user's input requirement information and the browser context data corresponding to the browser;

[0130] The workflow prediction module 202 is configured to obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and constrain the output of the large language model using the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information;

[0131] An operation sequence prediction module 203 is configured to obtain an executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction to obtain an executable atomic operation sequence corresponding to the requirement information;

[0132] The execution module 204 is configured to execute the executable atomic operation sequence cyclically, and convert the executable atomic operation sequence into an automation task template when the executable atomic operation sequence is completed.

[0133] In some feasible implementations, the browser context data includes at least a page routing table, and the workflow prediction module 202 is specifically configured to:

[0134] The demand information and the page routing table are input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information.

[0135] In some feasible implementations, the page routing table includes at least one of a page path, a page title, and at least one subordinate page, and the workflow prediction module 202 is specifically configured to:

[0136] Determine the page path, the page title, and the page path context corresponding to the subordinate page;

[0137] Packaging the page path context and the demand information to obtain first input data;

[0138] The first input data is input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information.

[0139] In some feasible implementations, the browser context data includes a DOM tree, and the operation sequence prediction module 203 is specifically configured to:

[0140] Performing reduction processing on the DOM tree to obtain a corresponding target DOM tree;

[0141] Obtaining the operation type corresponding to the browser;

[0142] Reduce the target DOM tree according to the operation type to obtain an executable operation context;

[0143] The operation type includes at least one of click triggering, double-click triggering, right-click triggering, jump triggering, input triggering and text acquisition.

[0144] In some feasible implementations, the operation sequence prediction module is specifically configured to:

[0145] Starting from the root node of the DOM tree, traverse each node of the DOM tree layer by layer;

[0146] If the target node currently being traversed is an operable element or has text content, the target node is retained;

[0147] If the currently traversed target node is not the operable element or does not have text content, the target node is deleted, and the color attribute corresponding to the target node is saved, and other style attributes are removed;

[0148] Assign a corresponding unique identifier to the retained node and obtain the corresponding target DOM tree.

[0149] In some feasible implementations, the executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the execution module 204 is specifically configured to:

[0150] Acquire a semantic tag, where the semantic tag includes at least one of an element tag, a path tag, a behavior tag, and an execution target tag;

[0151] Obtaining a target path context corresponding to the execution operation and a target DOM element corresponding to the execution target from the target DOM tree;

[0152] The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain corresponding second input data;

[0153] Inputting the second input data into the large language model to obtain an operable element corresponding to the target DOM element or an execution path corresponding to the target path context;

[0154] Obtaining an operation function corresponding to the operable element or the execution path;

[0155] The operation function is used to perform an operation corresponding to the operable element or the execution path.

[0156] In some feasible implementations, the execution module 204 is specifically configured to:

[0157] Convert the atomic operation sequence into a target sequence, where the target sequence consists of the operation function and a target value, where the target value includes at least one of a DOM element, a page path context, an execution operation, and an execution target;

[0158] Each of the target sequences and the constraint template is used to perform a persistence operation to obtain a corresponding automation task template.

[0159] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0160] In addition, an embodiment of the present invention further provides an electronic device, comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the various processes of the above-mentioned embodiment of the method for generating a Web workflow based on a large language model are implemented, and the same technical effects can be achieved. To avoid repetition, these processes will not be described here.

[0161] The present invention also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the various processes of the aforementioned embodiment of the method for generating a web workflow based on a large language model, achieving the same technical effects. To avoid repetition, the details are omitted here. The computer-readable storage medium may be, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0162] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0163] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, apparatuses, or computer program products. Therefore, the embodiments of the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program code.

[0164] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0165] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0166] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0167] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0168] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0169] The above is a detailed introduction to a Web workflow generation method based on a large language model and a Web workflow generation device based on a large language model provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.< / task>

Claims

1. A method for generating a Web workflow based on a large language model, characterized in that: include: Obtain the user's input requirements and the browser context data corresponding to the browser; Obtaining a constraint template for the browser context data, inputting the requirement information and the browser context data into a large language model for prediction, and constraining an output of the large language model by the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information; Obtaining an executable operation context corresponding to the browser context data, and inputting the executable operation context and the workflow plan into the large language model for prediction to obtain an executable atomic operation sequence corresponding to the demand information; The executable atomic operation sequence is executed cyclically, and when the executable atomic operation sequence is completed, the executable atomic operation sequence is converted into an automation task template.

2. The method according to claim 1, characterized in that The browser context data includes at least a page routing table. Inputting the requirement information and the browser context data into the large language model for prediction, and constraining the output of the large language model using the constraint template during the prediction process to obtain a workflow plan corresponding to the requirement information includes: The demand information and the page routing table are input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information.

3. The method according to claim 2, characterized in that The page routing table includes at least one of a page path, a page title, and at least one subordinate page. The requirement information and the page routing table are input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the requirement information, including: Determine the page path, the page title, and the page path context corresponding to the subordinate page; Packaging the page path context and the demand information to obtain first input data; The first input data is input into the large language model for prediction, and during the prediction process, the output of the large language model is constrained by the constraint template to obtain a workflow plan corresponding to the demand information.

4. The method according to any one of claims 1 to 3, characterized in that The browser context data includes a DOM tree, and obtaining the executable operation context corresponding to the browser context data includes: Performing reduction processing on the DOM tree to obtain a corresponding target DOM tree; Obtaining the operation type corresponding to the browser; Reduce the target DOM tree according to the operation type to obtain an executable operation context; The operation type includes at least one of click triggering, double-click triggering, right-click triggering, jump triggering, input triggering and text acquisition.

5. The method according to claim 4, characterized in that The reducing process of the DOM tree to obtain a corresponding target DOM tree includes: Starting from the root node of the DOM tree, traverse each node of the DOM tree layer by layer; If the target node currently being traversed is an operable element or has text content, the target node is retained; If the currently traversed target node is not the operable element or does not have text content, the target node is deleted, and the color attribute corresponding to the target node is saved, and other style attributes are removed; Assign a corresponding unique identifier to the retained node and obtain the corresponding target DOM tree.

6. The method according to claim 4, characterized in that The executable atomic operation sequence includes at least one atomic operation sequence, the atomic operation sequence includes an execution operation and an execution target corresponding to the execution operation, and the looping execution of the executable atomic operation sequence includes: Acquire a semantic tag, where the semantic tag includes at least one of an element tag, a path tag, a behavior tag, and an execution target tag; Obtaining a target path context corresponding to the execution operation and a target DOM element corresponding to the execution target from the target DOM tree; The target DOM element is wrapped with the element tag, the target path context is wrapped with the path tag, the execution operation is wrapped with the behavior tag, and the execution target is wrapped with the execution target tag to obtain corresponding second input data; Inputting the second input data into the large language model to obtain an operable element corresponding to the target DOM element or an execution path corresponding to the target path context; Acquire an operation function corresponding to the operable element or the execution path; The operation function is used to perform an operation corresponding to the operable element or the execution path.

7. The method according to claim 6, characterized in that When the executable atomic operation sequence is completed, converting the executable atomic operation sequence into an automated task template includes: Convert the atomic operation sequence into a target sequence, where the target sequence consists of the operation function and a target value, where the target value includes at least one of a DOM element, a page path context, an execution operation, and an execution target; Each of the target sequences and the constraint template is used to perform a persistence operation to obtain a corresponding automation task template.

8. A Web workflow generation device based on a large language model, characterized in that: include: The data acquisition module is used to obtain the required information input by the user and the browser context data corresponding to the browser; a workflow prediction module, configured to obtain a constraint template for the browser context data, input the requirement information and the browser context data into a large language model for prediction, and during the prediction process constrain the output of the large language model using the constraint template to obtain a workflow plan corresponding to the requirement information; an operation sequence prediction module, configured to obtain an executable operation context corresponding to the browser context data, and input the executable operation context and the workflow plan into the large language model for prediction, thereby obtaining an executable atomic operation sequence corresponding to the requirement information; The execution module is used to cyclically execute the executable atomic operation sequence, and when the executable atomic operation sequence is completed, convert the executable atomic operation sequence into an automation task template.

9. An electronic device, characterized in that: comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs; The processor is configured to implement the method according to any one of claims 1 to 7 when executing a program stored in the memory.

10. A computer-readable storage medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 7.

Images