Webpage process automation method and device, equipment, storage medium and program product

Abstract

The invention discloses a webpage process automation method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the webpage process automation method comprises the following steps: receiving a business demand description input by a user; step disassembling is carried out on the business demand description, a webpage operation process is obtained, and the webpage operation process comprises a plurality of operation step instructions; and based on a pre-acquired webpage document object model and the plurality of operation step instructions, performing webpage element positioning through a large model to obtain a webpage element path, and executing the webpage operation process through the webpage element path. According to the method and the device, webpage process automation based on the large model and the webpage document object model is realized, the problem of low webpage element positioning efficiency of an existing webpage process automation method is solved, and the webpage element positioning efficiency of webpage process automation is improved.

Description

Technical field

[0001] The application relates to the field of artificial intelligence technology, and in particular to a web page process automation method, device, device, storage medium and program product. Background technology

[0002] RPA (Robotic Process Automation) technology imitates users' operations on computers and automates the work of programs and systems, which can free users from monotonous and repetitive work and improve work efficiency.

[0003] The existing web page process automation method will strengthen the learning of the agent to identify web page components and locate page elements, which requires a lot of time to label data and a lot of resources to train the model, giving the agent too many page elements, which increases the difficulty of page element positioning, resulting in low page element positioning efficiency.

[0004] The above content is only used to assist in understanding the technical solutions of the present application and does not mean that the above content is prior art. Contents of invention

[0005] The main purpose of this application is to provide a web page process automation method, device, equipment, storage medium and program product, aiming to solve the technical problem of low efficiency in positioning web page elements of existing web page process automation methods.

[0006] To achieve the above-mentioned object, the present application proposes a method for automation of web page processes, which includes:

[0007] Receive user input description of business requirements;

[0008] The business requirements description is broken down step by step to obtain a web page operation process, which includes several operation step instructions;

[0009] Based on the pre-acquisitioned web page document object model and the several operation step instructions, the web page element positioning is performed through the large model, and the web page element path is obtained, and the web page operation process is performed through the web page element path.

[0010] In one embodiment, the steps of disassembling the business requirements description and obtaining the web page operation process include:

[0011] Obtain full component information;

[0012] Based on the business requirement description and full component information, a first disassembly prompt word is generated;

[0013] The first disassembly prompt word is input into the big model, and the big model is disassembled step by step, and the web page operation process is output.

[0014] In one embodiment, the full amount of component information includes one or more of component name, component parameters, component relationships, and component function description, the operation step instructions include one or more of step names, step instructions parameters, and step function description, the operation step instructions are used to operate web page elements of a front-end web page, and the web page elements include one or more of a text box, a button, a drop-down box, and a check box.

[0015] In one embodiment, the step of positioning web page elements through a large model and obtaining web page element paths based on the pre-acquisitioned web page document object model and the several operation step instructions further includes:

[0016] Based on the preset verification rules, the several operating step instructions are checked;

[0017] When the verification of the several operation step instructions fails, an operation semantic vector is generated based on the operation step instructions that fail to pass the verification;

[0018] According to the operation semantic vector, the target component data is obtained through the preset component vector database;

[0019] Based on the target component data and the first disassembly prompt word, a second disassembly prompt word is generated to input the second disassembly prompt word into the big model, and the step-by-step disassembly is performed through the big model, and the web page operation process is re-outputed.

[0020] In one embodiment, the step of obtaining web page element paths based on the pre-acquisitioned web page document object model and the several operation step instructions is to locate web page elements through the preset large model, including:

[0021] Based on the web page document object model and several operation step instructions, a positioning prompt word is generated;

[0022] The positioning prompt word is input into the big model, the web page element is positioned through the big model, and the web page element path is output.

[0023] In one embodiment, the step of inputting the positioning prompt word into the big model, locating the web element through the big model, and outputting the web element path includes:

[0024] Enter the positioning prompt word into the big model for the following processing:

[0025] According to the positioning prompt word, determine the web page element to be operated;

[0026] Read the node corresponding to the web page element in the web page document object model, and set the node corresponding to the web page element as the current node;

[0027] Recursively call the parent node of the current node in the web document object model, and set the parent node of the current node as the current node until the current node does not have a parent node, obtain the node path of the recursive call;

[0028] According to the node path, the web page element path is output.

[0029] In addition, to achieve the above-mentioned object, the application also proposes a web page process automation device, which includes:

[0030] Receive module, used to receive a description of the business requirements input by the user;

[0031] The disassembly module is used to disassemble the business requirements description step by step to obtain a web page operation process, which includes several operation step instructions;

[0032] The positioning module is used to locate web page elements through a large model based on the pre-acquisitioned web page document object model and the several operation step instructions, obtain the web page element path, and execute the web page operation process through the web page element path.

[0033] In addition, to achieve the above-mentioned object, the present application also proposes a web page process automation device, which includes a memory, a processor, and a computer program stored on the memory and can be run on the processor, the computer program configured to implement the steps of the web page process automation method as described above.

[0034] In addition, to achieve the above-mentioned object, the present application also proposes a storage medium, which is a computer readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of implementing the web page process automation method as described above.

[0035] In addition, to achieve the above-mentioned object, the present application also provides a computer program product, which comprises a computer program, and when executed by the processor, implements the steps of the web page process automation method as described above.

[0036] This application provides a web page process automation method, first receiving a business requirement description input by the user to determine the requirements for web page process automation; performing step-disassembly of the business requirement description to obtain a web page operation process containing several operation step instructions to convert user needs into multiple step instructions that can be executed by the web page; according to the pre-acquisitioned web page document object model and several operation step instructions obtained by step disassembly, the web page element path is obtained by using large model positioning, and the web page operation process is automatically executed through the web page element path to efficiently locate web page elements. The web page process automation based on the large model and web document object model is realized, and the problem of low efficiency of web page element positioning in the existing web page process automation method is solved, and the web page element positioning efficiency of web page process automation is improved. Attached description of the drawings

[0037] The accompanying drawings here are incorporated into and form part of this specification, show embodiments that comply with the present application, and together with the present application, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings required for the embodiments or the prior art description will be briefly introduced below. It will be obvious that for those skilled in the art, other accompanying drawings can also be obtained based on these accompanying drawings without giving creative labor.

[0039] Figure 1 A flow diagram provided for the first embodiment of the web page process automation method of this application;

[0040] Figure 2 A flow diagram provided for the second embodiment of the web page process automation method of this application;

[0041] Figure 3 A structural diagram provided for the third embodiment of the web page process automation method of this application;

[0042] Figure 4 A flow diagram provided for the third embodiment of the web page process automation method of this application;

[0043] Figure 5 A flow diagram of business requirements disassembly provided for the third embodiment of the web page process automation method of this application;

[0044] Figure 6 A flow diagram for the search-enhanced generation provided for the third embodiment of the web page process automation method of this application;

[0045] Figure 7 A flow diagram of web page element positioning provided for the third embodiment of the web page process automation method of this application;

[0046] Figure 8 A flow diagram of web page process execution provided for the third embodiment of the application web page process automation method;

[0047] Figure 9 It is a schematic diagram of the module structure of the web page process automation device in the embodiment of this application;

[0048] Figure 10 It is a schematic diagram of the equipment structure of the hardware operating environment involved in the web page process automation method in the embodiment of the application.

[0049] The implementation, functional features and advantages of the present application will be further explained in connection with the embodiments and refer to the accompanying drawings. Specific implementation methods

[0050] It should be understood that the specific embodiments described herein are for the purpose of explaining the technical solutions of the present application only and are not intended to limit the present application.

[0051] In order to better understand the technical solutions of the present application, the following will be explained in detail in connection with the accompanying drawings of the specification and specific embodiments.

[0052] The main solution of the embodiment of this application is: receiving a business requirement description input by a user; disassembly the business requirement description step by step to obtain a web page operation process, the web page operation process includes several operation step instructions; based on the pre-acquisitioned web page document object model and the several operation step instructions, the web page element positioning is performed through the large model, the web page element path is obtained, and the web page operation process is executed through the web page element path.

[0053] Since in the prior art, training reinforcement learning models is usually used to identify UI (User Interface) components and web page elements, thereby realizing web page process automation. In order to allow reinforcement learning agents to identify front-end component categories (i.e. buttons, text boxes, drop-down boxes, etc.) corresponding to various web page elements, learn to align instruction key-value pairs with operable leaf nodes and realize automatic actions, during the training stage, it is necessary to collect data sets described by different web pages about UI components such as buttons, drop-down boxes, check boxes, etc., and label these data sets. This method requires a lot of time to label data and a large number of resources to train the model.

[0054] Secondly, the way of giving all manipulated nodes (leaf nodes) in the web document object model to the agent will lead to too many page elements given to the agent, which will increase the difficulty of page elements positioning. When parsing natural instructions, we get command key-value pairs, rather than task execution steps. The granularity and dimension of the decomposition are not fine enough, and they are not consistent with the existing web page process automation design steps, which is not conducive to reusing existing RPA components and user understanding.

[0055] Technical terms involved in this application:

[0056] RPA (Robotic Process Automation) is a digital technology that automates the work of programs and systems by imitating users' operations on computers.

[0057] Large model: refers to machine learning models with large-scale parameters and complex computing structures. These models are usually built from deep neural networks and have billions or even hundreds of billions of parameters that can handle more complex tasks and data.

[0058] Page element recognition technology: technology to automatically identify and understand various elements on web pages or application interfaces. By combining HTML parsing, element positioning, data scraping and cleaning, various complex automation tasks can be realized, such as web crawlers, UI automation testing, data extraction, etc.

[0059] Plug-in: is a program written in accordance with certain standards of application program interfaces. It can only run under the system platform specified by the program (may support multiple platforms at the same time), and cannot run separately from the specified platform.

[0060] DOM (DocumentObjectModel) is a programming interface for representing and manipulating HTML or XML documents. It represents a document as a structured tree structure, with each node representing an object in the document, such as an element, attribute or text.

[0061] Based on the semantic understanding and reasoning capabilities of the big model, this application can automatically complete the disassembly of user business needs, convert the natural language input by the user into several executable steps, reduce the threshold for RPA process orchestration, reduce the time of process orchestration, and be able to reuse existing RPA components to complete the execution of operation steps, which is basically consistent with the existing RPA design steps, which is conducive to reusing existing RPA components and users' understanding. In addition, as input, a simple page element is positioned from the given page element and page DOM structure, and then the page element and operation step instructions are given to the executor engine for execution. For different web page interfaces or in different application scenarios, the page elements will not be affected by factors such as web page layout, color, and style. Page elements can be positioned efficiently and accurately.

[0062] It should be noted that the executing body of this embodiment may be a computing service device with data processing, network communication and program operation functions, such as a tablet computer, a personal computer, a mobile phone, or an electronic device, a web page process automation device, etc. that can realize the above functions. The following describes this embodiment and the following embodiments using the web page process automation equipment as an example.

[0063] Based on this, the embodiment of the present application provides a method of automation of web page processes, refer to Figure 1 , Figure 1 This is a flow diagram of the first embodiment of the web page process automation method of this application.

[0064] In this embodiment, the method of automation of web page flow includes steps S10 to S30:

[0065] Step S10, receive a description of the service requirement input by the user;

[0066] It should be noted that the business requirements description refers to the user's specific business requirements description for automation tasks, and is the starting point of automation process design. Users can enter through web plug-ins, user graphical interfaces, command line input, or upload requirements documents, etc. The business requirements description may include the task content, the landing page, and the expected results that the user wants to perform.

[0067] It can be understood that in order to clarify user needs and provide a foundation for subsequent automation process design, it can receive a description of the specific business needs of the user for the automation process of the web page, so as to understand the specific tasks that the user wants to automate and ensure that the automated process can meet the actual business needs of the user.

[0068] Step S20, the business requirements description is disassembled to obtain a web page operation process, which includes several operation step instructions;

[0069] It should be noted that the web page operation process refers to a collection of steps for automated operations in the web page, which is used to guide the execution of the automated process. Operation step instructions refer to specific operation instructions, which are used to guide automation tools to perform specific actions on web pages, such as "click the form button", "enter data", "click the submit button", etc.

[0070] It can be understood that in order to convert the user's high-level requirements into specific operational steps that can be executed by the automation tool, the business requirements description entered by the user is broken down into a web page operation process containing multiple executable operational step instructions, providing clear operational guidance for the automation tool and ensuring the executability of the process.

[0071] Step S30: Based on the pre-acquisitioned web page document object model and the several operation step instructions, web page elements are positioned through the large model, and the web page element path is obtained, and the web page operation process is performed through the web page element path.

[0072] It should be noted that the web page document object model refers to a tree-like representation of the web page structure, which is used to locate and manipulate web page elements. A complete web page document object model can include web page elements, comments, js, styles and other contents of the entire web page. The web page element path points to the specific location of a web page element in the DOM tree of the web page document object model. These paths can accurately interact with elements on the web page, thereby guiding the automation tool to find and operate specific web page elements in the web page, which describes how to reach the path from the root node of the web page document object model to a specific web page element. For example, use a CSS selector (such as #loginButton) or XPath (such as / html / body / div / button[1]) to locate elements.

[0073] In addition, it should be noted that when web page elements cannot be positioned directly through the DOM tree or text description, elements such as buttons, input boxes and other elements in the page can be found through image recognition technology, and elements can be further accurately positioned based on the visual content of the web page.

[0074] It can be understood that in order to realize automated web interaction, the web page's document object model and operation step instructions are used to locate the web page element path, operate the web page elements through the web page element path, and automatically execute the web page operation process. Through large-scale model technology and web page document object model technology, the system can accurately identify the target elements in complex web pages and perform the web page process automation operations.

[0075] In a feasible embodiment, the steps of disassembling the business requirements description and obtaining the web page operation process include:

[0076] Step S201, obtain the full amount of component information;

[0077] It should be noted that full component information refers to all possible operable component information. Components usually include interactive elements such as buttons, text boxes, links, drop-down boxes, check boxes, and their properties (such as IDs, class names, locations, label texts, etc.), which can be obtained by collecting different component library data.

[0078] It can be understood that there are differences in components used in different web pages. In order to ensure that the system can fully understand the components of different web pages and provide rich operation information for subsequent steps to disassemble them, collect all web page component information, so that when disassemble them, the web page elements of the corresponding component operations can be identified based on the full amount of component information and generate an operation process.

[0079] Step S202 generates a first disassembly prompt word based on the business requirement description and full component information;

[0080] It should be noted that the disassembly prompt word is a descriptive problem or instruction, designed to guide the big model to disassemble the steps described by user requirements. Transform users' business needs into task prompt words that the machine can understand and disassemble. For example, when the user enters "I want to log in", the system may generate a teardown prompt word: "The user wants to fill in the user name and password on the login page, click the login button."

[0081] It can be understood that in order to combine user needs and web component information, the disassembly prompt word guides the big model to disassembly business needs as a specific operation step, and combines the business requirement description provided by the user with the full component information obtained from it to generate a disassembly prompt word that guides the big model to output the answer, so as to transform the abstract business needs into operation prompts directly associated with web components, providing detailed guidance for subsequent automated disassembly.

[0082] In step S203, the first disassembly prompt word is input into the large model, and the large model is used to disassemble it stepwise to output the web page operation process.

[0083] It should be noted that the big model can be a trained deep learning model that can understand the specific meaning of the task from the input prompt words and generate a series of operation steps.

[0084] It can be understood that the first disassembly prompt word is used as input, and it is further processed and analyzed using the reasoning ability of the big model, and intelligently disassembles the user's needs into a web page operation process, ensuring the accuracy and efficiency of the web page operation process execution.

[0085] In this embodiment, based on the business needs input by the user, the natural language processing capability of the big model extracts key actions, goals and context information in the user's business needs, understands the specific operations corresponding to each step in the business needs and their related elements, and can transform the natural language needs input by the user into operations that can be performed by the machine, improving the level of automation. With the language understanding and logical reasoning capabilities of the big model, it can not only process static web pages, but also adapt to page structure changes and flexibly respond to different types of web pages.

[0086] In a feasible embodiment, the full amount of component information includes one or more of component name, component parameters, component relationships, and component function description, the operation step instructions include one or more of step names, step instructions parameters, and step function description, the operation step instructions are used to operate web page elements of a front-end web page, and the web page elements include one or more of a text box, a button, a drop-down box, and a check box.

[0087] It should be noted that the component name refers to the basic identifier of the web page element, usually the name of the HTML tag. For example, the button corresponds to <button>, the text box corresponds to <input> , the drop-down box corresponds to <select>, the check box corresponds to <inputtype="checkbox">, etc. Component parameters refer to the specific properties of a component that determine the behavior and display of the component. Component relationships refer to the association or relative position between components and other components, and are important for determining the operation steps and order. For example, a button may be located in a form, or a text box is part of a login form. A component function description refers to a description of the function or function of the component. For example, a "Submit" button's function description may be "Submit Form Data", and a text box may be "Enter Username" or "Enter Password".

[0088] The step name describes the function or purpose of the step, such as: "Enter a user name", "Click the login button", etc. The step instruction parameters give specific operating parameters. For example, the "Input Username" step may contain a parameter "username", that is, the component name corresponding to the input box; the "Click Button" step may contain the parameter "submit_button", that is, the component name of the button. The step function description describes the function of the operation step in detail. For example, the function description of "Enter a user name" can be "Enter a valid user name in the user name text box", and the function description of "Click the login button" can be "Click the login button to submit a form".

[0089] In a feasible embodiment, the pre-acquisitioned web page document object model and the several operation step instructions are based on the pre-acquisitioned web page document object model and the several operation step instructions, and before the step of obtaining the web page element path is:

[0090] Step S301, and the several operation step instructions are verified based on the pre-set verification rules;

[0091] It should be noted that the rules pre-set for determining whether the operation step instructions are valid by the verification rules are only valid for the web page operation process, which can include syntax, semantic and logical requirements, such as format verification, syntax verification, semantic verification and operation sequence verification, etc. Syntax verification refers to checking whether the structure of the operation step instructions meets the requirements, such as whether the format and parameters of the instructions are complete. Semantic verification refers to checking whether the logic of the operation step instructions is correct, such as whether there are valid components or whether there are impossible operations (for example, clicking an invisible button or entering invalid data, etc.). Operation sequence verification refers to ensuring that the order of operation steps is reasonable and avoiding logical incoherence of previous and subsequent steps.

[0092] It can be understood that in order to ensure that the instructions of the operation steps are valid and executable, several operation step instructions output by the large model are checked to ensure that these operation step instructions comply with preset verification rules, and thus ensure that these operation step instructions are valid and executable, avoid incorrect instructions entering the subsequent process, ensure the correctness and consistency of the instructions, and improve the reliability of the overall system.

[0093] Step S302, when the verification of the several operation step instructions fails, an operation semantic vector is generated based on the operation step instructions that fail to pass;

[0094] It can be understood that when certain operation step instructions fail to pass, it is necessary to understand the potential problems in these operation step instructions, conduct semantic analysis of the operation step instructions that fail to pass, and generate corresponding operation semantic vectors. The operation semantic vector not only reflects the content of the operation step, but may also cover potential error information or ambiguity.

[0095] Step S303, according to the operation semantic vector, search through the preset component vector database to obtain the target component data;

[0096] It should be noted that the attributes of each web page component (such as ID, component name, type, location, etc.) can be encoded and converted into vector form, and the operation semantic vector reflects the function, type, relationship with other elements, etc. of the component. A component vector database is a vectorized database containing all operable components in a web page. Component data (such as location, type, state, etc.) are stored in vector form.

[0097] It can be understood that after converting the operation steps into semantic vectors, the system can use these vectors to search in a pre-established component vector database to find the target component data related to the target operation, so that subsequent operations can be accurately performed on the target component, ensuring that the identified components are components expected by the user, thereby avoiding misoperation.

[0098] Exemplary, assuming that the corresponding component of a step instruction is "click the login button", but due to page layout changes, the button does not exist or is not visible, the semantics of this step can be converted into a vector and marked as "button is not available" or "target element not found", the model may match a component related to "login" based on the semantic vector.

[0099] Step S304, based on the target component data and the first disassembly prompt word, a second disassembly prompt word is generated to input the second disassembly prompt word into the big model, and the step disassembly is performed through the big model, and the web page operation process is re-outputed.

[0100] It can be understood that based on the target component data and the first disassembly prompt word generated for the first time, a more optimized second disassembly prompt word is generated, and the generated second disassembly prompt word is input into the big model, and the steps of user business requirements are redisassembled, and finally a more accurate and more accurate web page operation process is output.

[0101] In this embodiment, through the combination of large models and search enhancement generation technology, the operation step instructions are checked, searched and optimized to ensure the accurate execution of the operation steps, and ensure that automated operations can adapt to various complex web page structures and elements changes according to the real-time situation of the web page, thereby improving the intelligence level of web page process automation.

[0102] This embodiment provides a web page process automation method, first receiving the business requirement description input by the user to determine the requirements for web page process automation; performing step disassembly of the business requirement description, obtaining a web page operation process containing several operation step instructions to convert user needs into multiple step instructions that can be executed by the web page; according to the pre-acquisitioned web page document object model and several operation step instructions obtained by step disassembly, use large model positioning to obtain the web page element path, and automatically perform the web page operation process through the web page element path to perform efficient web page element positioning. The web page process automation based on the large model and web document object model is realized, and the problem of low efficiency of web page element positioning in the existing web page process automation method is solved, and the web page element positioning efficiency of web page process automation is improved.

[0103] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above-mentioned first embodiment can be referred to the above introduction and will not be described in detail later. On this basis, please refer to FIG. 2 , which is a flow diagram of the second embodiment of the web page process automation method of the present application.

[0104] In this embodiment, the step of obtaining web page element positioning through the preset large model based on the pre-acquisitioned web page document object model and the several operation step instructions to obtain the web page element path, including:

[0105] Step S305, based on the web page document object model and the several operation step instructions, generate a positioning prompt word for inputting web page element positioning in the big model, so that the element can be quickly positioned in the web page when the web page process is executed.

[0107] Step S306, input the positioning prompt word into the big model, and position the web page element through the big model, and output the web page element path.

[0108] It should be noted that the positioning prompt word will be input into the big model, and the web element positioning is used to use the natural language understanding and reasoning ability of the big model, and the location of the web element is inferred from the positioning prompt word, and finally output the path of the web element corresponding to these operation step instructions in the web document object model.

[0109] In a feasible embodiment, the step of inputting the positioning prompt word into the large model, positioning the web page element through the large model, and outputting the web page element path includes:

[0110] Entering the positioning prompt word into the large model for the following processing:

[0111] Step S3061, determines the web page element that needs to be operated according to the positioning prompt word;

[0112] It can be understood that after inputting the positioning prompt word into the large model, the big model analyzes the input positioning prompt word, understands its semantics, and determines the web page element that needs to be operated. For example, "clicking the login button" will point to the button element with a login function on the page.

[0113] Step S3062, read the node corresponding to the web page element in the web page document object model, and set the node corresponding to the web page element as the current node;

[0114] It should be noted that once the target element is recognized, the big model will find the node of the element from the DOM tree structure of the web page document object model, and the node contains all the information of the element, such as label type, attributes, child nodes, etc.

[0115] It can be understood that the node of the web page element that needs to be operated in the web page document object model, and set it as the current node, use it as the starting point of recursive traversal, and prepare for the subsequent parent node to recursively find the node path of the web page element.

[0116] Step S3063, recursively call the parent node of the current node in the web page document object model, and set the parent node of the current node as the current node until the current node does not have a parent node, and obtain the node path of the recursive call;

[0117] It should be noted that the node path uniquely identifies the location of the web page element in the DOM tree. The node path called recursively eventually forms a complete path by taking the current node's tag and its properties as part of the path in each recursion.

[0118] It can be understood that the current node is used as the starting point of recursive search, and recursively traverses the DOM tree from the node to the root node of the DOM tree, and obtains the parent node of the current node until it reaches the root node of the DOM tree, stops the recursive call, and builds the complete path of the continuous parent-child node relationship of the target element in the DOM from the node of the target element to the root node, which can accurately track and construct the node path of the web page element corresponding to different complex web pages.

[0119] Step S3064, according to the node path, the web page element path is output.

[0120] It should be noted that the web page element path can be a hierarchical structure description of the DOM node, such as html>body>div>form>button[id="submit"], which can be output in the form of a string or in a structured format (such as JSON).

[0121] It can be understood that after obtaining the path from the target element node to the root node through a recursive method, the path is converted into a readable web element path and output, and the target element can be uniquely identified, so that users or automation tools can use this path to accurately find the target element executed by the web operation process.

[0122] Example, assuming that the setting bit prompt word is "click the submit button", the big model recognizes that the "submit button" corresponds to an element, and determines the target operation based on the prompt word. The big model then reads the element node from the DOM and starts recursively looking up its parent node until it reaches the root node. Assume that the path is like: html>body>div>form>button[id="submit"], the path can accurately locate the "submit button" in the page and output it as the final web page element path.

[0123] In this embodiment, in combination with the reasoning ability of the big model, the DOM structure is traversed by recursive call method, and the path from the target element to the root node is constructed, thereby realizing the precise positioning of the web page element. It is possible to accurately identify and locate the web page element according to the positioning prompt word, and build a complete DOM path of the web page element.

[0124] In this embodiment, by combining natural language processing, large-model reasoning capabilities and web page DOM structure, efficient and flexible web page element positioning is achieved, and positioning prompt words can be automatically generated from operation step instructions, and the path of web page elements is inferred through the big model, ultimately realizing automated web page operations and testing, improving the accuracy, robustness and adaptability of the automation system.

[0125] Based on the first embodiment and / or the second embodiment of this application, in the third embodiment of this application, the same or similar contents as the above-mentioned first embodiment and the second embodiment can be referred to the above introduction and will not be described in detail later.

[0126] Example, in order to help understand the implementation flow of the web page process automation method obtained after the above-mentioned first and second embodiments, please refer to Figure 3. Figure 3 is an architectural schematic diagram of the third embodiment of the web page process automation method of the application. Specifically, it can include three layers:

[0127] The first layer is a display layer, mainly interacting with users through a web page plug-in;

[0128] The second layer is a business logic layer, mainly including three parts: business disassembly, tool calls, and task execution;

[0129] The third layer is a large model layer, mainly including large model, Agent, and vector database.

[0130] As shown in Figure 4, Figure 4 is a flow diagram of the third embodiment of the web page process automation method of this application. The user-oriented presentation layer is implemented in the form of a web page plug-in, which mainly includes the following steps:

[0131] 1. Obtain the description of the natural language business requirements received by the web page plug-in

[0132] After the web page plug-in is started, a text input box will be displayed on the front-end web page. The user can enter natural language through the text input box to describe the business requirements that need to be implemented. For example: the natural language entered by the user is: "Baidu search for Huawei Mate 60 mobile phone". Natural language text entered through the web plug-in will be passed to the business logic layer to disassemble the module processing.

[0133] 2. Disassembly the business requirements into operation step instructions

[0134] As shown in Figure 5, Figure 5 is a schematic flow diagram of the business requirements disassembly provided by the third embodiment of the web page process automation method of this application. The main purpose is to semantic understanding of the input natural language business needs through a large language model and implement step disassembly, thereby outputting operation step instructions. The output operation step instructions need to pass various verification checks, and the final disassembly is not completed until the operation step instructions corresponding to the business requirements fully meet the conditions. The specific processing process includes:

[0135] (1) Intent verification

[0136] After receiving the description of natural language business requirements entered by the user through the web plug-in, the intention identification screening is first necessary, and the business needs that violate laws and regulations, social ethics, etc. that do not meet the requirements are directly withdrawn from the processing, and corresponding prompt information is given to the user to enable the user to re-enter the legal business needs.

[0137] (2) Large model generation operation step instructions

[0138] The big model generates operation step instructions according to the prompt word generation operation step instructions, and the key information contained in the prompt word is: full component information, historical generation memory information. The full component information includes component name, parameters required for component execution, relationships between components, functional descriptions of components, etc.; historical memory information is mainly prompt information that the operation step instructions generated by the historical generation do not meet the requirements, indicating the problem of the operation step instructions generated before the large model, so that the large model can generate results that meet the requirements next time.

[0139] Among them, the number of repetitions of the operation step instructions can be realized through configuration, and it will be generated cycled within the configuration times until all conditions are met, and the operation step instructions that meet all conditions are output. The generated operation step instructions are output in json format and include information such as step name, step parameters, step function description, etc.

[0140] In addition, the accuracy of the business requirements steps of the large model dismantling are closely related to the temperature parameters of the large model. The temperature is the sampling temperature of the large model, and the value range is [0.0, 1.0]. The larger the value, the higher the diversity of the large model answers. In this embodiment, different values ​​can be taken for temperature, and the number of times the business needs splits meet the conditions can be counted to determine the value of temperature, so as to improve the accuracy of step disassembly.

[0141] The disassembly of business needs is automatically completed using a large model. The disassembly output is the operation step instructions. The granularity of the disassembly is consistent with the granularity of the existing RPA components, which can better reuse the existing RPA components, and at the same time, users can easily understand the generated operation step instructions.

[0142] (3) Verify the operation step instructions for the output of the large model

[0143] In the prompt word for the large model, there are detailed descriptions of the restriction conditions for the output of the model, so that the large model can output the operation step instructions according to the given requirements. However, sometimes the results of the output of the large model do not fully meet the needs, so the output results of the large model need to be verified. Only when the verification is completely passed is an effective operation step instructions. The main verification includes the following aspects:

[0144] (i) The output operation step instructions are in json format, and the fields in json also meet the preset field requirements;

[0145] (ii) The verification operation step instructions first need to verify whether the step name is in a known component set, and the step name and the use component name must be clearly stated in the prompt word; secondly, whether the output step instruction parameters meet the requirements of the component, and the required types of parameters cannot be missing, otherwise this operation step cannot be performed correctly; finally, whether the relationship between components meets the requirements, such as: two components cannot appear continuously, etc.

[0146] Wherein, when the component name verification fails, the correct component name will be output through RAG (Retrieval Augmented Generation) technology, and then the big model will be given the command to regenerate the operation step. Retrieval enhancement generation searches relevant information through its own vertical database, and then merges it into a prompt template to answer the big model generation problem. It is an effective solution to solve the limitations, hallucinations, data security and other problems of general basic big model knowledge. The component data of each RPA system is its own vertical domain data, so the big model can be well provided with this data through RAG technology.

[0147] As shown in Figure 6, Figure 6 is a schematic diagram of the process of search enhancement generation provided by the third embodiment of the application web page process automation method. The main processing flow of search enhancement generation includes:

[0148] First, the component data is converted into a semantic vector through the Embedding model and stored in the vector database; when the user or the system fails to pass the component name verification, it constructs a question (i.e., problem), and also converts it into a semantic vector using the Embedding model (i.e., embedded model) into a semantic vector, and searches in the vector database, and submits the retrieved first top_n data and prompt words to the big model. The big model finally gives the correct component name and operation step instructions, where top_n can be configured by itself.

[0149] If the verification is not passed, the error prompt information will be clearly given or the positive data prompt will be provided, and the input will be given to the big model again. In this way, the big model can better generate operation step instructions that meet the requirements and realize business needs in this way.

[0150] (4) Output operation step instructions

[0151] Pass the operation step instructions that meet the requirements generated by the big model to the web page element acquisition module, and persist the user's input and generated results to the database. If the user enters the same business requirements next time, he can first obtain it from the database to reduce the number of calls to the big model, and also improve the efficiency of disassembly of operation steps instructions.

[0152] Taking "Baidu Search Huawei Mate 60 Mobile Phone" as an example, the step disassembly process is explained:

[0153] a. After the user enters "Baidu Search Huawei Mate 60 Mobile Phone" in the interface, he first performs intention recognition on this statement. The recognition result is compliant, and then enters the step disassembly logic. Through the prompt word project, let the big model output the disassembly steps in the prescribed format.

[0154] b. After obtaining the disassembly step of the large model output, verify the component name, component parameters, component relationships, etc. in the step. If the verification fails, an error message will be generated, and then the error message, historical memory, etc. will be input into the big model together through the prompt words. In this way, a decomposition step that meets the requirements is generated. The following is an example of a generated operation step instruction:

[0155]

[0156] Based on the big model and prompt word engineering technology, its technical feature is to verify the results generated by the big model, add prompt information that does not meet the requirements to the prompt word, so that the big model has context memory, and cycle to generate operation step instructions until the requirements meet the requirements. This will automatically complete the disassembly, lower the threshold for RPA process orchestration and reduce the time for process orchestration.

[0157] 3. Positioning web page elements

[0158] In the previous step, the operation step instructions that meet the requirements have been generated. For each operation step instructions, if you want to automatically execute the web page elements that need to be positioned for operation step instructions, you must first. In this embodiment, the web page elements corresponding to each operation step instruction are simple web page elements, such as text boxes, buttons, drop-down boxes and check boxes in the web page.

[0159] Usually, a web page element can be uniquely positioned through the XPath path in a web page. This embodiment is to use the semantic understanding and reasoning ability of the big model to find the web page element corresponding to the operation step instructions, and then obtain the XPath path of this web page element through recursive call, thereby easily realizing web page element positioning.

[0160] As shown in Figure 7, Figure 7 is a schematic flow diagram of web page element positioning provided by the third embodiment of the application web page process automation method. The specific flow of web page element positioning includes:

[0161] (1) Obtaining the document object model of the web page

[0162] Through the URL address of the web page, the document object model of the web page can be obtained. For example: The document object model of the current web page can be obtained through Selenium or browser plug-in technologies.

[0163] (2) Document object model for processing web pages

[0164] After the original web page document object model obtained through the URL address, many of the contents have no effect on positioning web page elements. Remove these contents first, such as comments, js, styles, etc. in the document object model. The processed web object model is put into the cache, and the next time it needs to be used, it is directly obtained from the cache, which speeds up the efficiency of obtaining the web document object model.

[0165] (3) The big model outputs the web page elements to be operated

[0166] The big model locates the web page elements to be operated according to the prompt word. The prompt word mainly includes the processed web page document object model, given web page elements, historical error information, etc. Among them, the processed web document object model still retains the complete web elements and their structure. For obtaining a given web page element, you first need to determine the type of the given web page element. The type of web page element is pre-configured according to the operation step instructions. For example, the operation step instructions are to click the submit button, and the submit button is mainly passed.< / select> <input type＝"submit"> and< / button> <button type＝"submit">Two forms appear. If there are other forms, they can also be added through configuration. The specific web page elements can be obtained in the web page document object model through the type of web page element. The historical error message contains error information that previously locates web elements. It is used as a context memory of the big model, telling the big model where there is a problem with the result generated by the big model, so that the big model can output the correct result next time.

[0167] The big model only needs to locate the web elements that need to be operated in a given web element based on the complete document object model and error prompt information through semantic understanding and reasoning capabilities. The number of times the positioning web page element loop can be realized through configuration. The web page element is positioned through the big model loop within the number of times the configuration is not finally completed until the corresponding web page element is found and the loop is exited. Only then can the web page element that meets the criteria be found.

[0168] (4) Verify the output results

[0169] The operation object corresponding to each operation step instruction is determined. Only in the given web page element provided to the big model, the web page element output by the big model must be in the given web page element. If the web page element output by the big model is not in the given web page element, the corresponding error message is given, and the big model is asked to locate the web page element again until the web page element that meets the criteria is found.

[0170] (5) Get the XPath of the web page element

[0171] Through the specific web page elements located in the previous step and the complete web page document object model, the XPath path corresponding to the web page element can be obtained through recursive calls. The document object model is represented as a tree structure called a DOM tree. Now that there are specific web page elements, it can obtain its parent node. By judging whether the parent node exists to end the recursive call, the complete XPath path of the web page element can be obtained through this recursive call method.

[0172] Exemplary, taking "Baidu Search Huawei Mate 60 Mobile Phone" as an example, the previous step has generated operation step instructions and executed each step: first obtain the source code of the current web page through the browser plug-in, and then remove the source code of the web page, such as js, css, annotations, etc. Then, the processed web page source code and the web page elements that need to be operated in the current step are given to the big model. Through the prompt word project, the big model outputs the exact web page elements that need to be operated in this step. Finally, the XPath path of this web page element is obtained through the complete web page source code and recursive call of the web page element. By positioning the web page element, you can call the component to perform the current step.

[0173] Through the powerful semantic understanding and reasoning capabilities of the big model, by inputting web document model objects, given web page elements, and historical error prompt information, in a way similar to multiple rounds of dialogue, the web page elements can be accurately and efficiently positioned, and the various operation step instructions are automatically completed through the executor engine.

[0174] 4. Automatically execute operation steps and instructions

[0175] The automatic execution module of the operation step instruction is responsible for the automatic execution of the operation step instruction, which is mainly completed by calling the corresponding components through the executor engine. Components are the logical units that can be repeatedly executed encapsulated in RPA. Components that operate web page elements need to pass on the operation web page elements, parameters, and operation step instructions.

[0176] Exemplary, such as Figure 8 Shown, Figure 8 The process diagram of the web page process execution provided for the third embodiment of the application web page process automation method, the specific process of the web page process execution includes:

[0177] First, check the input parameters, verify whether the parameters are legal and whether there are any necessary parameters. After the parameters are checked, the corresponding component will be called to execute. After the execution is completed, the execution result will be returned to the next step. Finally, through the operation step instructions of natural language disassembly and the XPath path of the web page element, the corresponding component can be automatically called to complete the operation steps. The execution of each operation step instruction is the same execution logic. If all operation step instructions are executed in a loop, the user's business needs can be successfully completed, thereby realizing the user's business needs.

[0178] In this embodiment, the business requirements described in natural language are automatically converted into specific operation step instructions, and then the corresponding operation actions are automatically executed according to the operation step instructions. Make the RPA process construction process more automated and intelligent, reducing the workload of manual orchestration process; at the same time, using the semantic understanding and reasoning capabilities of the large language model, accurately find the web page elements that need to be operated by the operation step instructions, and obtain the XPath path of the web page elements, thereby achieving automatic completion of the operation step instructions. Moreover, this method will not be affected by factors such as page style and layout, and has good generalization capabilities.

[0179] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the method of automation of the web page process of this application. It is within the scope of the protection of this application based on this technical concept.

[0180] This application also provides a web page process automation device, please refer to Figure 9 , the web page process automation device includes:

[0181] The receiving module 10 is used to receive a description of the service requirements input by the user;

[0182] The disassembly module 20 is used to disassemble the business requirement description step by step to obtain a web page operation process, which includes several operation step instructions;

[0183] The positioning module 30 is used to locate web page elements through a large model based on the pre-acquisitioned web page document object model and the several operation step instructions, obtain a web page element path, and execute the web page operation process through the web page element path.

[0184] Optionally, the disassembly module 20 is also used for:

[0185] Obtain full component information;

[0186] Based on the business requirement description and full component information, a first disassembly prompt word is generated;

[0187] The first disassembly prompt word is input into the big model, and the big model is disassembled step by step, and the web page operation process is output.

[0188] Optionally, the full amount of component information includes one or more of component name, component parameters, component relationships and component function description, the operation step instructions include one or more of step names, step instructions parameters, and step function description, the operation step instructions are used to operate web page elements of a front-end web page, and the web page elements include one or more of a text box, a button, a drop-down box, and a check box.

[0189] Optionally, the disassembly module 20 is also used for:

[0190] Based on the preset verification rules, the several operating step instructions are checked;

[0191] When the verification of the several operation step instructions fails, an operation semantic vector is generated based on the operation step instructions that fail to pass the verification;

[0192] According to the operation semantic vector, the target component data is obtained through the preset component vector database;

[0193] Based on the target component data and the first disassembly prompt word, a second disassembly prompt word is generated to input the second disassembly prompt word into the big model, and the step-by-step disassembly is performed through the big model, and the web page operation process is re-outputed.

[0194] Optionally, the positioning module 30 is also used for:

[0195] Based on the web page document object model and several operation step instructions, a positioning prompt word is generated;

[0196] The positioning prompt word is input into the big model, the web page element is positioned through the big model, and the web page element path is output.

[0197] Optionally, the positioning module 30 is also used for:

[0198] Enter the positioning prompt word into the big model for the following processing:

[0199] According to the positioning prompt word, determine the web page element to be operated;

[0200] Read the node corresponding to the web page element in the web page document object model, and set the node corresponding to the web page element as the current node;

[0201] Recursively call the parent node of the current node in the web document object model, and set the parent node of the current node as the current node until the current node does not have a parent node, obtain the node path of the recursive call;

[0202] According to the node path, the web page element path is output.

[0203] The web page process automation device provided in this application can solve the technical problem of low efficiency in web page element positioning using the web page process automation method in the above-mentioned embodiment. Compared with the prior art, the beneficial effects of the web page process automation device provided in this application are the same as the beneficial effects of the web page process automation method provided in the above-mentioned embodiment, and other technical features in the web page process automation device are the same as those disclosed in the above-mentioned embodiment methods, and will not be described here.

[0204] The present application provides a web page process automation device, which includes: at least one processor; and a memory in communication connected with at least one processor; wherein the memory stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor to enable the at least one processor to execute the web page process automation method in the first embodiment described above.

[0205] Reference below Figure 10 , which shows a structural diagram of a web page process automation device suitable for implementing an embodiment of the present application. The web page process automation device in the embodiment of the application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), in-vehicle terminals (such as in-vehicle navigation terminals), and fixed terminals such as digital TVs, desktop computers, and the like. Figure 10 The shown web page process automation device is only an example and should not limit any limitations on the functions and scope of use of the embodiments of the present application.

[0206] like Figure 10 As shown, the web page flow automation device may include a processing device 1001 (eg, a central processor, a graphics processor, etc.) that may perform various appropriate actions and processing according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from the storage device 1003 into a random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the web page process automation device are also stored. The processing device 1001 , the ROM 1002 , and the RAM 1004 are connected to each other through the bus 1005 . The input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to an I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, and the like; and a communication device 1009. Communication device 1009 may allow web page process automation devices to communicate wirelessly or wired with other devices to exchange data. Although web page flow automation devices with various systems are shown in the figures, it should be understood that implementation or possession of all the shown systems is not required. More or fewer systems may be implemented alternatively or have.

[0207] In particular, according to an embodiment disclosed in the present application, the process described above with reference to the flowchart may be implemented as a computer software program. For example, embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium including program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001 , the above-mentioned functions defined in the method of the disclosed embodiment of the present application are performed.

[0208] The web page process automation equipment provided in this application can solve the technical problem of low efficiency in web page element positioning of existing web page process automation methods using the web page process automation method. Compared with the prior art, the beneficial effects of the web page process automation device provided in this application are the same as the beneficial effects of the web page process automation method provided in the above-mentioned embodiment, and other technical features in the web page process automation device are the same as those disclosed in the previous embodiment method, and will not be described here.

[0209] It should be understood that the various parts disclosed in this application may be implemented in hardware, software, firmware, or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics may be combined in suitable ways in any one or more embodiments or examples.

[0210] The above is only the specific implementation of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art who is familiar with the art can easily think of changes or substitutions within the technical scope disclosed in this application, and should be covered within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the said claims.

[0211] The present application provides a computer readable storage medium with computer readable program instructions (ie, computer programs) stored thereon, which are used to execute the web page flow automation method in the above-mentioned embodiment.

[0212] The computer readable storage medium provided in this application may be, for example, a USB disk, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM: Random Access Memory), read-only memory (ROM: Read Only Memory), erasable programmable read-only memory (EPROM: Erasable Programmable Read Only Memory or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM: CD-Read Only Memory), optical memory devices, magnetic memory devices, or any suitable combination of the above. In this embodiment, the computer readable storage medium may be any tangible medium containing or storing a program that may be used or used in combination with an instruction execution system, system, or device. Program code contained on the computer readable storage medium may be transmitted in any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency) and the like, or any suitable combination of the above.

[0213] The above-mentioned computer-readable storage medium may be contained in the web page process automation device; it may also exist separately and not assembled into the web page process automation device.

[0214] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the web page process automation device, the web page process automation device: receives a business requirement description input by the user; disassembles the business requirement description stepwise to obtain a web page operation process, the web page operation process includes several operation step instructions; based on the pre-acquisitioned web page document object model and the several operation step instructions, the web page element positioning is performed through the large model, and the web page element path is obtained, and the web page operation process is executed through the web page element path.

[0215] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or combinations thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computer, partly on the user computer, as a standalone software package, partly on the user computer, partly on the remote computer, or entirely on the remote computer or server. In cases involving remote computers, the remote computer may be connected to a user computer through any kind of network—including a local area network (LAN: Local Area Network) or Wide Area Network (WAN: Wide Area Network), or may be connected to an external computer (eg, using an Internet service provider to connect over the Internet).

[0216] The flowcharts and block diagrams in the drawings illustrate possible architectures, functions and operations of systems, methods and computer program products in accordance with various embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a portion of a module, program block or code that contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in a different order than those marked in the accompanying drawings. For example, two consecutively represented boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functionality involved. It is also noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented with a dedicated hardware-based system that performs prescribed functions or operations, or may be implemented with a combination of dedicated hardware and computer instructions.

[0217] Description The modules involved in the embodiments of the present application can be implemented in software or hardware. Among them, the name of the module does not constitute a limitation to the unit itself in some cases.

[0218] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (ie, computer programs) for executing the above-mentioned web page process automation method, which can solve the technical problem of low efficiency in positioning web page elements of the existing web page process automation method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the web page process automation method provided in the above-mentioned embodiments and will not be described here.

[0219] The present application also provides a computer program product, including a computer program, which implements the steps of an automated method of web page flow as described above when executed by a processor.

[0220] The computer program products provided by this application can solve the technical problem of inefficient positioning of web page elements in existing web page process automation methods. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the web page process automation method provided in the above-mentioned embodiments, and will not be described here.

[0221] The above are only partial embodiments of this application and do not therefore limit the scope of the patent of this application. Any equivalent structural change made using the specification and drawings of this application under the technical concept of this application, or directly / indirectly applied in other related technical fields, is included within the scope of the patent protection of this application.< / button>

Claims

1. A web page process automation method, characterized in that: The method comprises: Receive business requirement description input by users; Decomposing the business requirement description into steps to obtain a webpage operation process, wherein the webpage operation process includes a plurality of operation step instructions; Based on the pre-acquired web page document object model and the several operation step instructions, the web page elements are located through the large model to obtain the web page element path, and the web page operation process is executed through the web page element path.

2. The method according to claim 1, characterized in that: The step of decomposing the business requirement description into steps to obtain the webpage operation process includes: Get all component information; Based on the business requirement description and the full component information, generate a first disassembly prompt word; The first disassembly prompt word is input into the large model, the steps are disassembled through the large model, and the web page operation process is output.

3. The method according to claim 2, characterized in that: The full component information includes one or more of component name, component parameters, component relationship and component function description; the operation step instructions include one or more of step name, step instruction parameters and step function description; the operation step instructions are used to operate the web page elements of the front-end web page; the web page elements include one or more of text boxes, buttons, drop-down boxes and check boxes.

4. The method according to claim 2, characterized in that: Before the step of locating the web page element through the large model based on the pre-acquired web page document object model and the several operation step instructions to obtain the web page element path, the step further includes: Based on preset verification rules, verify the several operation step instructions; When the plurality of operation step instructions fail verification, generating an operation semantic vector based on the operation step instructions that fail verification; According to the operational semantic vector, searching is performed through a preset component vector database to obtain target component data; Based on the target component data and the first disassembly prompt word, a second disassembly prompt word is generated, and the second disassembly prompt word is input into the large model, and the steps are disassembled through the large model to re-output the web page operation process.

5. The method according to claim 1, characterized in that: The step of locating the web page element through a preset large model based on the pre-acquired web page document object model and the plurality of operation step instructions to obtain the web page element path includes: Based on the webpage document object model and a number of operation step instructions, generating a positioning prompt word; The positioning prompt word is input into the large model, the web page element is positioned through the large model, and the web page element path is output.

6. The method according to claim 5, characterized in that The step of inputting the positioning prompt word into the large model, locating the web page element through the large model, and outputting the web page element path includes: The positioning prompt word is input into the large model and processed as follows: Determine the web page element that needs to be operated according to the positioning prompt word; Reading a node corresponding to the web page element in the web page document object model, and setting the node corresponding to the web page element as a current node; Recursively call the parent node of the current node in the webpage document object model, and set the parent node of the current node as the current node, until the current node has no parent node, to obtain a node path of the recursive call; According to the node path, a web page element path is output.

7. A web page process automation device, characterized in that: The device comprises: A receiving module, used to receive a business requirement description input by a user; A disassembly module, used to disassemble the business requirement description into steps to obtain a webpage operation process, wherein the webpage operation process includes a plurality of operation step instructions; The positioning module is used to locate the web page elements through the large model based on the pre-acquired web page document object model and the several operation step instructions, obtain the web page element path, and execute the web page operation process through the web page element path.

8. A web page process automation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the webpage process automation method according to any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the web page process automation method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the web page flow automation method according to any one of claims 1 to 6 are implemented.

Images