Interactive RPA webpage flow coding system and method

Abstract

The invention discloses an interactive RPA webpage flow coding system and method, and the system comprises a decision model which is used for executing behaviors according to all current information decisions so as to meet user demands; all the current information comprises a user demand, recorded operation step information and a current RPA flow code; the data merging tool is used for determining operation steps related to user requirements and webpage element numbers from the newly recorded operation step information, screening the newly recorded operation step information and storing the newly recorded operation step information into an operation step database; the data diagnosis tool is used for selecting data needing to be diagnosed from the operation step database and the newly recorded operation step information, and performing diagnosis to obtain a diagnosis result; and the automatic recording tool is used for analyzing the user demand and the RPA flow code, selecting a starting url according to the operation webpage recorded by the operation step database, and automatically recording the webpage operation flow related to the user demand.

Description

Technical Field

[0001] The present application relates to an interactive RPA web page process coding system and method, and belongs to the field of software technology. Background Art

[0002] In the process automation design of RPA (Robotic Process Automation), users can plan continuous web page operations according to their own needs. Each operation is an interactive action between the user and the web page UI (User Interface), such as clicking and inputting. After the web page automation process design is completed, the RPA program will automatically execute the page operation according to the user's design steps. Through the process design and automatic execution of RPA, users can greatly reduce repetitive and tedious manual page operations.

[0003] In essence, RPA process automation software will convert the user's planned operation process into a series of codes related to web page operations based on inherent rules; each web page operation code is bound to a web page interaction interface. These interfaces can be developed based on JavaScript or directly call functions of third-party libraries such as Playwright and Selenium, and the automation process is to repeatedly execute these web page operation codes.

[0004] In recent years, multimodal large language models (MLLM), such as GPT4o and Claude3.5, have been increasingly used by RPA vendors to develop agents related to web page operations due to their powerful text understanding and image recognition capabilities. MLLM agents can automatically plan page operation processes that meet user requirements based on user prompts and input web page images, and perform web page operations on behalf of users.

[0005] However, the more powerful the MLLM service, the higher the service fee. If every automation execution relies on the MLLM service, RPA users cannot afford the high service fees. Even if you deploy your own trained MLLM, you still need to consider the high expenses in computing power and service deployment.

[0006] Therefore, it is a more economical solution to use MLLM to plan the operation process only in the RPA design phase and execute the code related to the operation process in the automation execution phase. Summary of the invention

[0007] Coding has always been one of the most powerful features of MLLM. Many intelligent agents, such as "Cursor", use MLLM's coding capabilities to provide users with services that allow them to code according to their needs and continuously improve the code. These coding agents can replace general programmers.

[0008] In view of at least one of the above technical problems, the present application provides an interactive RPA web page process coding system and method, which can convert the user's web page process into RPA code based on MLLM, and continuously modify and improve the code according to user needs.

[0009] Technical solution: To solve the above technical problems, the technical solution adopted in this application is:

[0010] In a first aspect, the present application provides an interactive RPA webpage process coding system, the system comprising:

[0011] A decision model is used to: use MLLM to make decisions and execute actions based on all current information to meet user needs; wherein all current information includes user needs, recorded operation step information, current RPA process code, execution error of the last required RPA code, and execution result of the last action; wherein the actions include: MLLM text generation behavior and the behavior of scheduling external tools; behavior; MLLM text generation behavior includes "generate or modify RPA process code", "answer questions", and "result feedback"; the behavior of scheduling external tools includes "data merging", "data diagnosis", and "automatic recording";

[0012] The data merging tool is used to: determine the operation steps related to the user's needs and the web page element numbers of these operation steps from the newly recorded operation step information by using MLLM; filter the newly recorded operation step information according to the web page element numbers and store it in the operation step database;

[0013] Data diagnostic tool, used to: select the data to be diagnosed from the operation step database and the newly recorded operation step information using MLLM according to user needs, and diagnose and obtain the diagnosis results by combining user needs with the description of the current RPA process code;

[0014] The automatic recording tool is used to: use MLLM to analyze user needs and RPA process codes, and select a starting URL based on the operation web page recorded in the operation step database; automatically record the web page operation process related to the user needs based on the user needs and the starting URL to obtain the newly recorded operation step information.

[0015] In some embodiments, the operation step information of each operation includes operation steps, operation carrier, operation type, operation web page, web page element type, web page element class name, web page element XPath, web page element visible text, surrounding text of the web page element, web page element label and web page element number.

[0016] In some embodiments, the data diagnostic tool specifically includes:

[0017] The data selection module is used to: obtain the operation step database and the newly recorded operation step information, use MLLM to select the data to be diagnosed according to user needs, execute the code generated by the selected data, and capture the data printed in the code to obtain the captured data;

[0018] The diagnostic data module is used to: combine user requirements with the description of the current RPA process code, and use MLLM to diagnose the captured data to obtain diagnostic results.

[0019] In some embodiments, the automatic recording tool specifically includes:

[0020] Select the URL module, which is used to: use MLLM to analyze user needs and RPA process codes, and select the starting URL based on the operation webpage (webpage URL) recorded in the operation step database;

[0021] The web page operation recording intelligent agent is used to automatically record the web page operation process related to the user's needs according to the user's needs and the starting URL, and obtain the newly recorded operation step information.

[0022] In some embodiments, the prompt words of the decision model are divided into system prompt words and user prompt words. The system prompt words specify the function of the decision model, and the user prompt words provide user needs and the results of the previous behavior; the decision model will output the decision behavior and corresponding results based on each round of user prompt words.

[0023] Furthermore, the system prompt words include system tasks, behavior lists, coding requirements, system rules and output formats;

[0024] The system tasks explain that the decision model needs to continuously take actions to write RPA process codes or answer user questions; the list of actions includes: "Generate or modify RPA process code", "Answer questions", "Result feedback", "Data merging", "Data diagnosis", and "Automatic recording"; among them, the actions of "Generate or modify RPA process code", "Answer questions", and "Result feedback" are executed using the MLLM text generation capability; "Data merging", "Data diagnosis", and "Automatic recording" respectively call the data merging tool, data diagnosis tool, and automatic recording tool to execute;

[0025] The coding requirement describes the operation function interface required by the process code;

[0026] System rules establish the connection between behaviors, specifically: 1) data must be merged after generating or modifying the RPA process code, 2) the recording file provided by the first user requirement is used to generate the initial RPA process code, 3) if no recording file is provided in subsequent user requirements and the code needs to be modified, automatic recording is called;

[0027] The output format specifies the output specifications for each round of behavioral decisions, including decision behavior, decision reasons, RPA process code, and responses. If the behavior is "generate or modify RPA process code", the generated code will be directly indicated in the "RPA process code". The response is the answer to a specific question or a prompt for the next step.

[0028] Furthermore, before each decision is made, a user prompt will be generated, which will organize all current information, including user needs, recorded operation step information, current RPA process code, execution error of the last required RPA code, and execution result of the last behavior; user prompts prompt the decision model to make correct decisions based on all current information.

[0029] Furthermore, the RPA web page process encoding system has a clearing function, which can clear all operation step information and MLLM user prompts of the current encoding process.

[0030] In a second aspect, the present application provides an interactive RPA webpage process encoding method, the method comprising:

[0031] S1. Obtain the first user demand, which includes an operation process recording file; the decision model uses the text generation capability of the multimodal large language model MLLM to generate an initial RPA process code according to the first user demand and the operation step information in the operation process recording file;

[0032] S2. Obtain subsequent user requirements, which may or may not include operation process recording files;

[0033] If the user requirements include or do not include the operation process recording file, the decision model schedules the "data diagnostic tool" to check whether the current operation step information meets the user requirements; if the current operation step information does not meet the user requirements, the decision model calls the "automatic recording tool" to record the supplementary operation step information until the operation step information meets the user requirements; then the text generation capability of the multimodal large language model MLLM is used to call the "generate or modify RPA process code" behavior to generate or modify the RPA process code according to the code requirements and operation step information in the user requirements; if there are RPA process problems in the user requirements, the "answer question" behavior is called to answer the user requirement questions; if the analysis shows that the user requirements have been met, the "result feedback" behavior is called to terminate the decision process of the current user requirements;

[0034] If the user requirements include operation process recording files, the decision model schedules the "Data Diagnostic Tool" to check whether the current operation step information meets the user requirements; if the current operation step information meets the user requirements, the text generation capability of the multimodal large language model MLLM is directly used to call the "Generate or Modify RPA Process Code" behavior to generate or modify the RPA process code according to the code requirements and operation step information in the user requirements; if there are RPA process problems in the user requirements, the "Answer Questions" behavior is called to answer the user's requirements; if the analysis shows that the user requirements have been met, the "Result Feedback" behavior is called to terminate the decision-making process of the current user requirements.

[0035] In some embodiments, the interactive RPA web page process coding method further includes: in response to the completion of the "generate or modify RPA process code" behavior, the decision model calls the "data merge tool" to save the used operation step information to the operation step database.

[0036] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the second aspect.

[0037] In a fourth aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the second aspect when executing the computer program.

[0038] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which implements the steps of the method described in the second aspect when executed by a processor.

[0039] Beneficial effects: The interactive RPA web process coding system and method provided by the present application have the following advantages: including, based on the decision model and external tools built on MLLM, the user's web process can be converted into RPA code, and the code can be continuously modified and improved according to user needs, and the recording of supplementary data can be automatically called when the data does not meet user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is a schematic diagram of an interactive RPA webpage process coding method application according to an embodiment of the present application;

[0041] Figure 2 This is a schematic diagram of the principle of a data merging tool according to an embodiment of the present application;

[0042] Figure 3 This is a schematic diagram of the principle of a data diagnostic tool according to an embodiment of the present application;

[0043] Figure 4This is a schematic diagram of the principle of an automatic recording tool according to an embodiment of the present application;

[0044] Figure 5 A schematic diagram of an interactive RPA webpage process encoding system according to an embodiment of the present application;

[0045] Figure 6 A schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0046] The present application is further described below in conjunction with the accompanying drawings and embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and cannot be used to limit the protection scope of the present application.

[0047] Example 1: Figure 5 As shown, an interactive RPA webpage process coding system includes:

[0048] A decision model is used to: use MLLM to make decisions and execute actions based on all current information to meet user needs; wherein all current information includes user needs, recorded operation step information, current RPA process code, execution error of the last required RPA code, and execution result of the last action; wherein the actions include: MLLM text generation behavior and the behavior of scheduling external tools; behavior; MLLM text generation behavior includes "generate or modify RPA process code", "answer questions", and "result feedback"; the behavior of scheduling external tools includes "data merging", "data diagnosis", and "automatic recording";

[0049] The data merging tool is used to: determine the operation steps related to the user's needs and the web page element numbers of these operation steps from the newly recorded operation step information by using MLLM; filter the newly recorded operation step information according to the web page element numbers and store it in the operation step database;

[0050] Data diagnostic tool, used to: select the data to be diagnosed from the operation step database and the newly recorded operation step information using MLLM according to user needs, and diagnose and obtain the diagnosis results by combining user needs with the description of the current RPA process code;

[0051] The automatic recording tool is used to: use MLLM to analyze user needs and RPA process codes, and select a starting URL based on the operation web page (web page URL) recorded in the operation step database; automatically record the web page operation process related to the user needs based on the user needs and the starting URL to obtain the newly recorded operation step information.

[0052] It should be noted that the recording format of the webpage operation process used by the system can be obtained through the MLLM webpage exploration agent or through manual recording by the user. The recording technology and method are not the focus of this solution. The recording format of each operation in the webpage operation process is shown in Table 1. The operation record consists of a series of metadata, which characterizes an operation from the perspectives of operation mode and element attributes.

[0053] Table 1 Data format of recording operation

[0054]

[0055] The operation step information of each operation includes operation steps, operation carrier, operation type, operation web page (web page url), web page element type, web page element class name, web page element XPath, web page element visible text, surrounding text of web page element, web page element label and web page element number.

[0056] The important information in the operation mode is the operation type and the operation webpage. This information represents the specific operation on the specific webpage, which helps to understand the role of MLLM in the entire process for a single operation in the subsequent steps. Element attributes are native content of HTML, and webpage element information is recorded from multiple angles. In the automation stage, because there are some unknown dynamic changes in the HTML of the webpage, such as changes in the class attribute, it is impossible to use a single element information for matching; using as much element information as possible for fuzzy matching can resist the impact of dynamic changes. However, when coding the RPA process, it is impossible to write all element information into the operation code interface, so the webpage element number is used to represent the specific webpage element.

[0057] The web process coding system consists of a decision model and multiple external tools. The decision behavior of the decision model will dispatch these external tools to obtain auxiliary information outside the decision process. External tools can be developed based on MLLM or directly call other MLLM agents. The following is a detailed introduction to three external tools.

[0058] Each time the process coding service is used, a temporary database will be created, which will continuously store the operation step information recorded by the user. Figure 2As shown, the data merging tool uses MLLM to determine the operation steps related to user needs and the web page element numbers of these operation steps from the newly recorded operation step information; select the newly recorded operation step information according to the web page element number and store it in the operation step database; the operation step information of each operation includes: operation step, operation carrier, operation type, operation web page (web page url), web page element type, web page element class name, web page element XPath, web page element visible text, web page element surrounding text, web page element label and web page element number. Through this tool, the database will merge the original operation information with the newly recorded operation information.

[0059] The data diagnostic tool is divided into two modules: the data selection module and the data diagnostic module (such as Figure 3 As shown in Figure 2, each module is developed based on MLLM.

[0060] In some embodiments, the data diagnostic tool specifically includes:

[0061] The data selection module is used to: obtain the operation step database and the newly recorded operation step information, use MLLM to select the data to be diagnosed according to user needs, execute the code generated by the selected data, and capture the data printed in the code to obtain the captured data;

[0062] The diagnostic data module is used to: combine user needs with the description of the current RPA process code, and use MLLM to diagnose the captured data to obtain diagnostic results, such as "whether the current data meets user needs".

[0063] like Figure 4 As shown, the automatic recording tool is composed of a URL selection module and a web page operation recording agent; the web page operation recording agent will explore and record web page operations according to needs, which is not specifically described here. In some embodiments, the automatic recording tool specifically includes:

[0064] Select the URL module, which is used to: use MLLM to analyze user needs and RPA process codes, and select the starting URL based on the operation webpage (webpage URL) recorded in the operation step database;

[0065] The web page operation recording intelligent agent is used to automatically record the web page operation process (one to two steps of operation) related to the user's needs according to the user's needs and the starting URL, and obtain the newly recorded operation step information.

[0066] The recording format and external tools are the basic preparations for the successful operation of the decision model. The main tool of the decision model is to make decisions and execute actions to meet user needs based on user needs and recorded operations ( Figure 1). The execution result of each action will affect the next decision, and some actions require calling external tools.

[0067] In an RPA coding service, users will interact with the system multiple times, and each interaction will raise a requirement; as the code changes, these requirements will also change. Generally speaking, in the first requirement, the user will provide an operation process recording file and ask the system to generate the initial RPA process code based on the file. In this solution, users can manually enter the file path, or directly specify the recording file name in the requirement, and the system will locate the specific recording file based on the file name.

[0068] After the initial RPA process code is generated, subsequent user requirements tend to modify the code, such as adding or deleting operations. Subsequent requirement descriptions can provide new recording files or not provide recording files. User requirements and the operation step information in the recording files are input into the decision model as different text blocks.

[0069] The decision model continuously makes decisions to meet user needs. Some of these actions can be the text generation capabilities of MLLM itself (Table 2), or external tools can be dispatched to generate results.

[0070] MLLM text generation behaviors include generating or modifying RPA process codes, answering questions, and providing result feedback. The decision model will generate corresponding results while deciding these behaviors. For example, the decision behavior "generate or modify RPA process code" will be output together with the corresponding Python code.

[0071] The decision model will call "Generate or modify RPA process code" to generate the corresponding code according to the user's code requirements. The generated or modified code will be saved on the local server for subsequent behavior calls. If the user's requirements are related to RPA process questions, the decision model will call the "Answer question" behavior. When the decision model believes that the user's requirements have been met, it calls "Result Feedback" and the decision process of the current requirements will also terminate.

[0072] Table 2 Behavior list

[0073]

[0074] The behaviors of scheduling external tools include data merging, data diagnosis, and automatic recording. Different from text generation behaviors, the decision model only determines the behaviors, and the execution results of the behaviors need to be obtained by calling external tools.

[0075] Whenever "Generate or modify RPA process code" is completed, the decision model will call "Data Merge" to save the operation step information used. If the user does not provide a recording file, the decision model will schedule "Data Diagnosis" to check whether the current operation step information meets the user's needs; if the information does not meet the requirements, the decision model will call "Automatic Recording" to supplement the information.

[0076] The prompt words of the decision model are divided into system prompt words and user prompt words. The system prompt words specify the function of the decision model, and the user prompt words provide user needs and the results of the previous behavior. The decision model will output the decision behavior and the corresponding results based on the user prompt words in each round.

[0077] The system prompt words include system tasks, behavior lists, coding requirements, system rules and output formats;

[0078] The system tasks explain that the decision model needs to continuously take actions to write RPA process codes or answer user questions; the list of actions includes: "Generate or modify RPA process code", "Answer questions", "Result feedback", "Data merging", "Data diagnosis", and "Automatic recording"; among them, the actions of "Generate or modify RPA process code", "Answer questions", and "Result feedback" are executed using the MLLM text generation capability; "Data merging", "Data diagnosis", and "Automatic recording" respectively call the data merging tool, data diagnosis tool, and automatic recording tool to execute;

[0079] The coding requirements describe the operation function interfaces required by the process code, as shown in Table 3. These interfaces are regular web page operations. In addition, the coding requirements also specify some code rules, such as the need to comment the operation function code and use element numbers to encapsulate the operation function.

[0080] Table 3 Some operation function interfaces

[0081] Operation Type Function interface Open URL Webtool.openURL(init_url) Mouse clicks Webtool.mouseClick(id) Setting the text Webtool.setValue(id,value) Get Text Webtool.getValue(id) Whether the element exists Webtool.exist(XPath) 。。。 。。。

[0082] System rules establish the connection between behaviors, specifically: 1) data must be merged after generating or modifying the RPA process code, 2) the recording file provided by the first user requirement is used to generate the initial RPA process code, 3) if no recording file is provided in subsequent user requirements and the code needs to be modified, automatic recording is called;

[0083] The output format specifies the output specifications for each round of behavioral decisions, including decision behavior, decision reasons, RPA process code, and response. If the behavior is "generate or modify RPA process code", the generated code will be directly indicated in the "RPA process code". The response is the answer to a specific question or the next step. It is worth noting that because the behavior list is short, this solution does not use the function nesting function in the MLLM service to represent the behavior function and output format, which are directly written in the prompt word.

[0084] Before each decision is made, a user prompt is generated, which organizes all current information, including user requirements, recorded operation steps, current RPA process code, execution error of the last required RPA code, and execution result of the last behavior. The user prompt prompts the decision model to make correct decisions based on all current information, such as modifying the current RPA process code according to the result of the last "automatic recording" behavior.

[0085] The process coding system of this solution allows multiple interactions with users. Each time the user interacts, they can make demands or provide recording files, allowing the system to generate the initial code from the beginning and continuously modify the code later. Each interaction and each behavior under the interaction will prompt the improvement of the RPA process code, and the operation process information is also constantly accumulated. The prompt words of the decision model will stack with the number of interactions, which also enables MLLM to have an overview of all information of the entire process coding process. The RPA web page process coding system has a clearing function, which can clear all operation step information and MLLM user prompt words of the current coding process, allowing users to start another RPA coding service.

[0086] Example 2: Based on Example 1, this example provides an interactive RPA webpage process encoding method, such as Figure 1 As shown, including:

[0087] S1. Obtain the first user demand, which includes an operation process recording file; the decision model uses the text generation capability of the multimodal large language model MLLM to generate an initial RPA process code according to the first user demand and the operation step information in the operation process recording file;

[0088] S2. Obtain subsequent user requirements, which may or may not include operation process recording files;

[0089] If the user requirements include or do not include the operation process recording file, the decision model schedules the "data diagnostic tool" to check whether the current operation step information meets the user requirements; if the current operation step information does not meet the user requirements, the decision model calls the "automatic recording tool" to record the supplementary operation step information until the operation step information meets the user requirements; then the text generation capability of the multimodal large language model MLLM is used to call the "generate or modify RPA process code" behavior to generate or modify the RPA process code according to the code requirements and operation step information in the user requirements; if there are RPA process problems in the user requirements, the "answer question" behavior is called to answer the user requirement questions; if the analysis shows that the user requirements have been met, the "result feedback" behavior is called to terminate the decision process of the current user requirements;

[0090] If the user requirements include operation process recording files, the decision model schedules the "data diagnostic tool" to check whether the current operation step information meets the user requirements; if the current operation step information meets the user requirements, the text generation capability of the multimodal large language model MLLM is directly used to call the "generate or modify RPA process code" behavior according to the code requirements and operation step information in the user requirements to generate or modify the RPA process code; if there are RPA process problems in the user requirements, the "answer question" behavior is called to answer the user's demand questions; if the analysis shows that the user requirements have been met, the "result feedback" behavior is called to terminate the decision process of the current user requirements;

[0091] In response to the completion of the "generate or modify RPA process code" behavior, the decision model calls the "data merge tool" to save the used operation step information to the operation step database.

[0092] In summary, the interactive RPA webpage process coding system and method provided by the embodiment of the present application allows multiple interactions with users. Each time the user interacts, he can make demands or provide recording files, allowing the system to generate the initial code from the beginning to the subsequent continuous modification of the code. Each interaction and each behavior under the interaction will promote the improvement of the RPA process code, and the operation process information is also constantly accumulated. The prompt words of the decision model will stack with the number of interactions, which also enables MLLM to have an overview of all information of the entire process coding process.

[0093] In addition, the system provides a clearing function, which clears all operation process information and MLLM user prompts of the current coding process, allowing users to start another RPA coding service.

[0094] Embodiment 3: Based on Embodiment 2, this embodiment provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in Embodiment 2 are implemented.

[0095] Embodiment 4: Based on embodiment 2, as Figure 6 As shown, this embodiment provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in Example 2 when executing the computer program.

[0096] Embodiment 5: Based on Embodiment 2, this embodiment proposes a hardware system that applies an interactive RPA webpage process encoding method in Embodiment 2. The method is applied on the server side.

[0097] The terminal may be an electronic device with network access capability, for example, a desktop computer, a tablet computer, a laptop computer, a smart phone, a digital assistant, a shopping guide terminal, a television, etc.

[0098] The server can be an electronic device with certain computing and processing capabilities. For example, the server can be a server of a distributed system, or a system with multiple processors, memories, network communication modules, etc. operating in collaboration. The server can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. The server can also be a server cluster formed by several servers. Alternatively, with the development of science and technology, the server can also be a new technical means that can realize the corresponding functions of the implementation method of the specification. For example, it can be a new form of "server" based on quantum computing.

[0099] Example 6: Based on Example 2, this example also provides a computer program product comprising instructions, which, when executed by a computer, enables the computer to execute an interactive RPA web page process encoding method in Example 2.

[0100] It should be understood that the specific examples in this article are only intended to help those skilled in the art better understand the implementation methods of this specification, rather than limiting the scope of this application.

[0101] It can be understood that in the various implementations of this specification, the size of the sequence number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the implementation methods of this specification.

[0102] It can be understood that the various embodiments described in this specification can be implemented individually or in combination, and the embodiments of this specification are not limited to this.

[0103] Unless otherwise specified, all technical and scientific terms used in the embodiments of this specification have the same meaning as those generally understood by those skilled in the art of the technical field of this specification. The terms used in this specification are only for the purpose of describing specific embodiments and are not intended to limit the scope of this specification. The term "and / or" used in this specification includes any and all combinations of one or more related listed items. The singular forms of "a", "above", and "the" used in the embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0104] It can be understood that the processor of the embodiment of this specification can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method implementation can be completed by the hardware integrated logic circuit or software instructions in the processor. The above processor can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiment of this specification can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of this specification can be directly embodied as a hardware decoding processor to perform, or the hardware and software modules in the decoding processor are combined and executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0105] It is understood that the memory in the embodiments of this specification may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (programmable ROM, PROM), an erasable programmable read-only memory (erasablePROM, EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0106] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this specification.

[0107] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method implementation methods and will not be repeated here.

[0108] In the several embodiments provided in this specification, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0109] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0110] In addition, each functional unit in each embodiment of the present specification may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0111] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this specification, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of this specification. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0112] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An interactive RPA webpage process coding system, characterized in that: include: Decision model, used to: use MLLM to make decisions based on all current information and perform actions to meet user needs; The current information includes user requirements, recorded operation steps, current RPA process code, execution error of the last required RPA code, and execution result of the last behavior; the behaviors include: MLLM text generation behavior and external tool scheduling behavior; MLLM text generation behavior includes "generate or modify RPA process code", "answer questions", and "result feedback"; external tool scheduling behavior includes "data merging", "data diagnosis", and "automatic recording"; The data merging tool is used to: determine the operation steps related to the user's needs and the web page element numbers of these operation steps from the newly recorded operation step information by using MLLM; filter the newly recorded operation step information according to the web page element numbers and store it in the operation step database; Data diagnostic tool, used to: select the data to be diagnosed from the operation step database and the newly recorded operation step information using MLLM according to user needs, and diagnose and obtain the diagnosis results by combining user needs with the description of the current RPA process code; The automatic recording tool is used to: use MLLM to analyze user needs and RPA process codes, and select a starting URL based on the operation web page recorded in the operation step database; automatically record the web page operation process related to the user needs based on the user needs and the starting URL to obtain the newly recorded operation step information.

2. The interactive RPA webpage process coding system according to claim 1, characterized in that: The operation step information of each operation includes the operation step, operation carrier, operation type, operation web page, web page element type, web page element class name, web page element XPath, web page element visible text, web page element surrounding text, web page element label and web page element number.

3. The interactive RPA webpage process coding system according to claim 1, characterized in that: The data diagnostic tool specifically includes: The data selection module is used to: obtain the operation step database and the newly recorded operation step information, use MLLM to select the data to be diagnosed according to user needs, execute the code generated by the selected data, and capture the data printed in the code to obtain the captured data; The diagnostic data module is used to: combine user requirements with the description of the current RPA process code, and use MLLM to diagnose the captured data to obtain diagnostic results.

4. The interactive RPA webpage process coding system according to claim 1, characterized in that: The automatic recording tool specifically includes: Select the URL module, which is used to: use MLLM to analyze user needs and RPA process codes, and select the starting URL based on the operation webpage recorded in the operation step database; The web page operation recording intelligent agent is used to automatically record the web page operation process related to the user's needs according to the user's needs and the starting URL, and obtain the newly recorded operation step information.

5. The interactive RPA webpage process coding system according to claim 1, characterized in that: The prompt words of the decision model are divided into system prompt words and user prompt words. The system prompt words specify the function of the decision model, and the user prompt words provide user needs and the results of the previous behavior; the decision model will output the decision behavior and corresponding results based on the user prompt words in each round.

6. The interactive RPA webpage process coding system according to claim 5, characterized in that: The system prompt words include system tasks, behavior lists, coding requirements, system rules and output formats; The system tasks explain that the decision model needs to continuously take actions to write RPA process codes or answer user questions; the action list includes: "Generate or modify RPA process code", "Answer questions", "Result feedback", "Data merging", "Data diagnosis", and "Automatic recording" actions; among them, "Generate or modify RPA process code", "Answer questions", and "Result feedback" actions are executed using the MLLM text generation capability; "Data merging", "Data diagnosis", and "Automatic recording" respectively call the data merging tool, data diagnosis tool, and automatic recording tool to execute; The coding requirement describes the operation function interface required by the process code; System rules establish the connection between behaviors, specifically: 1) data must be merged after generating or modifying the RPA process code, 2) the recording file provided by the first user requirement is used to generate the initial RPA process code, 3) if no recording file is provided in the subsequent user requirements and the code needs to be modified, automatic recording is called; The output format specifies the output specifications for each round of behavioral decisions, including decision behavior, decision reasons, RPA process code, and response. If the behavior is "generate or modify RPA process code", the generated code will be directly indicated in the "RPA process code". The response is the answer to a specific question or a prompt for the next step.

7. The interactive RPA webpage process coding system according to claim 5, characterized in that: Before each decision is made, a user prompt is generated, which organizes all current information, including user requirements, recorded operation steps, current RPA process code, execution error of the last required RPA code, and execution result of the last behavior; user prompts prompt the decision model to make correct decisions based on all current information.

8. The interactive RPA webpage process coding system according to claim 1, characterized in that: The RPA web page process coding system has a clearing function, which can clear all operation step information and MLLM user prompts of the current coding process.

9. An interactive RPA web page process coding method, characterized in that: The method comprises: S1. Obtain the first user demand, which includes an operation process recording file; the decision model uses the text generation capability of the multimodal large language model MLLM to generate an initial RPA process code according to the first user demand and the operation step information in the operation process recording file; S2. Obtain subsequent user requirements, which may or may not include operation process recording files; If the user requirements include or do not include the operation process recording file, the decision model schedules the "Data Diagnosis Tool" to check whether the current operation step information meets the user requirements; if the current operation step information does not meet the user requirements, the decision model calls the "Automatic Recording Tool" to record the supplementary operation step information until the operation step information meets the user requirements; then the text generation capability of the multimodal large language model MLLM is used to call the "Generate or Modify RPA Process Code" behavior to generate or modify the RPA process code according to the code requirements and operation step information in the user requirements; if there are RPA process problems in the user requirements, the "Answer Question" behavior is called to answer the user's demand questions; if the analysis shows that the user requirements have been met, the "Result Feedback" behavior is called to terminate the decision process of the current user requirements; If the user requirements include operation process recording files, the decision model schedules the "Data Diagnostic Tool" to check whether the current operation step information meets the user requirements; if the current operation step information meets the user requirements, the text generation capability of the multimodal large language model MLLM is directly used to call the "Generate or modify RPA process code" behavior to generate or modify the RPA process code according to the code requirements and operation step information in the user requirements; if there are RPA process problems in the user requirements, the "Answer Questions" behavior is called to answer the user's requirements questions; if the analysis shows that the user requirements have been met, the "Result Feedback" behavior is called to terminate the decision process of the current user requirements.

10. The interactive RPA webpage process encoding method according to claim 9, characterized in that: Also includes: In response to the completion of the "generate or modify RPA process code" behavior, the decision model calls the "data merge tool" to save the used operation step information to the operation step database.

Images