Method for realizing automatic generation of RPA process through reinforcement learning

Abstract

The invention discloses a method for realizing automatic generation of an RPA process through reinforcement learning, and the method comprises the following steps: 1, training a multi-modal large model, and enabling the multi-modal large model to have the task process decomposition capability based on page screenshot and task description; 2, training a process supervision reward model to enable the process supervision reward model to judge whether the next generation step is correct or not according to the page screenshot, the task description and the context; 3, performing reinforcement learning training on the multi-modal large model by using a process supervision reward model to obtain a task process decomposition model; and step 4, generating detailed RPA steps based on the trained task process decomposition model, then calling a general large model API to convert each RPA step into an actual RPA instruction, and forming a complete RPA process by the RPA instructions. According to the method, the illusion generated by the model in the process generation is greatly reduced, the task process decomposition capability of the model is improved, and the RPA instruction selection generation accuracy is enhanced.

Description

Technical Field

[0001] The present invention relates to a method for realizing automatic generation of RPA processes through reinforcement learning, and belongs to the technical field of RPA processes. Background Art

[0002] At present, the automatic generation scheme of RPA process generally follows the following two steps: Task process decomposition: According to the task description, the task is decomposed into multiple specific steps that can be implemented by RPA, and each step corresponds to an RPA instruction. RPA instruction selection generation: The above specific step description is converted into executable RPA instructions, including the selection of RPA actions and operation objects. For task process decomposition, there are three solutions: 1. Manual generation: Use manual methods to first decompose the task description into step descriptions that can be executed by RPA. However, this solution still has the problems of being time-consuming and requiring high personnel capabilities. 2. Large language model generation: Through unsupervised learning and supervised learning methods, fine-tune the open source large language model to generate the decomposed process steps according to the task description. This solution only uses unsupervised learning and supervised learning methods, and it is difficult to align the generated step description with the executable RPA instructions; because the large language model cannot fully understand the specific business information, such as which clickable objects exist on the web page, it may generate non-existent / unrealizable RPA steps, that is, hallucinations; due to the inevitable hallucination problems of the large language model itself, as well as the hallucination problems caused by insufficient information mentioned above, in the generation of long processes, the overall accuracy is exponentially affected, and there will be a situation where one step is wrong and all steps are wrong. 3. Multimodal model feedback: When the process contains non-existent operation objects, feedback is provided in the RPA instruction selection generation step, so that the multimodal model is used for page description, and then the large language model regenerates the process steps based on the page description. This solution relies solely on textual page descriptions, which cannot fully express the information of the page, so it is difficult to ensure that the regenerated process steps are executable. Summary of the invention

[0003] The purpose of the present invention is to provide a method for realizing automatic generation of RPA processes by using reinforcement learning. The present invention greatly reduces the illusions generated by the model in process generation, improves the ability of the model to decompose task processes, and enhances the accuracy of RPA instruction selection and generation.

[0004] The technical solution of the present invention is as follows:

[0005] A method for realizing automatic generation of RPA processes by using reinforcement learning includes the following steps:

[0006] Step 1: Train the multimodal large model so that it has the ability to decompose the task flow based on page screenshots and task descriptions;

[0007] Step 2: Train the process supervision reward model so that the process supervision reward model can judge whether the next generation step is correct based on the page screenshot, task description and context;

[0008] Step 3: Use the process supervision reward model to perform reinforcement learning training on the multimodal large model to obtain a task process decomposition model;

[0009] Step 4: Generate detailed RPA steps based on the trained task process decomposition model, and then call the general large model API to convert each RPA step into actual RPA instructions, which form a complete RPA process.

[0010] In the above-mentioned method of realizing automatic generation of RPA processes through reinforcement learning, in step 1, the training of the multimodal large model includes supervised training and unsupervised training; the supervised training is used to enable the multimodal large model to understand RPA tasks and web page screenshots; the unsupervised training is used to enable the multimodal large model to output task steps according to RPA tasks and web page screenshots.

[0011] The aforementioned method of implementing automatic generation of RPA processes through reinforcement learning, wherein the training data for training the multimodal large model includes documents of the RPA platform, open source datasets of descriptions of common web pages, task descriptions of existing RPA applications and text descriptions of each step, and screenshots of page changes during the operation of the RPA application.

[0012] The aforementioned method of implementing automatic generation of RPA processes through reinforcement learning, the formula of the process supervision reward model is expressed as follows:

[0013] Rθ=(s,a);

[0014] In the formula, s represents the current state of the task description, webpage screenshot and context, a represents the next generated step, and R θ Represents the PRM score of a task process. The PRM score is defined as the product of the correctness probability of each step.

[0015] The aforementioned reinforcement learning method for automatically generating RPA processes uses a general large model API to supervise the training data of the reward model in the training process, generates a large number of detailed task steps, and then predicts each generated task step based on the task description, web page screenshots and context through manual annotation. The prediction results are divided into three categories: correct, neutral and wrong.

[0016] In the above-mentioned method of realizing automatic generation of RPA processes through reinforcement learning, in step 3, the reinforcement learning training uses the PPO strategy optimization algorithm, and the goal of the PPO strategy optimization algorithm is to maximize the return It is expressed as:

[0017]

[0018] Among them, E is the expectation, τ represents a strategy The generated trajectory, R θ is a process supervision reward model; t represents the index of the time step, representing a moment when an observation is obtained from the environment, an action is taken, and a reward is obtained.

[0019] In the aforementioned method of realizing automatic generation of RPA processes through reinforcement learning, in step 4, the function of the general large model API to convert each RPA step into actual RPA instructions is to build an intelligent agent by writing prompts and perform conversion with the intelligent agent.

[0020] The aforementioned reinforcement learning method for automatically generating RPA processes, the process of transformation of the agent is to describe in detail to the agent which actions and operation objects in RPA are available for operation; then let the agent select specific RPA actions and operation objects according to the detailed step description; at the same time, let the agent see all previously generated RPA instructions. If the previous instructions cannot be naturally connected to the current step, let the agent generate multiple RPA instructions so that the process can run completely.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. The multimodal large model used in the present invention can greatly reduce the illusions produced by the model in process generation by inputting more complete information.

[0023] 2. In the step of task process decomposition, the present invention adds the use of reinforcement learning to train the multimodal large model, further improving the ability of the model to decompose the task process.

[0024] 3. The present invention uses a process supervision reward model for reinforcement learning. For multi-step generation tasks, compared with the result supervision reward model, it can make the multimodal large model more aware of the causes of hallucinations, thereby further reducing the hallucinations of the model.

[0025] 4. In the instruction generation stage, the present invention uses a general large model API and builds an intelligent agent by writing prompts, which reduces the training cost and enhances the accuracy of RPA instruction selection and generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a flowchart for training a large multimodal model;

[0027] Figure 2 It is a flow chart of the agent generating RPA instructions;

[0028] Figure 3is a flow chart of manually providing a webpage screenshot in an embodiment;

[0029] Figure 4 It is a flow chart of automatically obtaining a screenshot of a web page in an embodiment. DETAILED DESCRIPTION

[0030] The present invention is further described below in conjunction with the accompanying drawings and embodiments, but they are not intended to limit the present invention.

[0031] Embodiment: A method for realizing automatic generation of RPA processes through reinforcement learning, such as Figure 1 As shown, the following steps are included:

[0032] Step 1: Train the multimodal large model so that it has the ability to decompose the task flow based on page screenshots and task descriptions;

[0033] In this step, the training goal of the multimodal large model is to enable it to fully understand the RPA tasks and web page screenshots, and to train its ability to generate detailed task steps based on task descriptions and web page screenshots.

[0034] The training tasks of the multimodal large model include supervised training and unsupervised training; wherein supervised training is used to enable the multimodal large model to understand RPA tasks and web page screenshots; unsupervised training is used to enable the multimodal large model to output task steps according to RPA tasks and web page screenshots. The training data in the training tasks include RPA understanding task data, web page understanding task data, and task process decomposition task data.

[0035] Among them, RPA understands task data as the documentation of the RPA platform, including what tasks RPA can achieve, what actions RPA includes, what functions the distribution represents, etc.

[0036] The web page understanding task data is an open source dataset of common web page descriptions.

[0037] Task process decomposition task data is to reversely construct the task description of each application and the text description of each step from a large number of existing RPA applications. In addition, during the operation process, screenshots are taken for each page change to obtain screenshots of the pages that will appear in the process.

[0038] In this step, the model training of the multimodal large model includes selecting an open source multimodal large model and first performing unsupervised training on its language module, that is, generating text through the autoregressive method. Then, a detailed description of the task and screenshots of the pages that will appear in the process are input, and the training model outputs detailed task steps, each of which corresponds to a specific RPA instruction.

[0039] Step 2: Train the process supervision reward model so that the process supervision reward model can judge whether the next generation step is correct based on the page screenshot, task description and context;

[0040] In this step, the training data acquisition process of the process supervision reward model uses the general large model API to generate a large number of detailed task steps. Then, through manual annotation, based on the task description, web page screenshots and context, the score of each step is predicted, which is divided into three categories: correct, neutral and wrong.

[0041] In this step, the model training form of the process-supervised reward model (PRM) is to predict the correctness of each step after the last token of each step; this prediction takes the form of a single token, and the log-likelihood of these target tokens is maximized during the training process. Therefore, PRM can be trained in the standard language model pipeline without any special adjustments. To determine the step-level prediction at test time, it is only necessary to perform a PRM forward pass on the entire solution. The PRM score of a process is defined as the probability that each step is correct under PRM, that is, the product of the correctness probabilities of each step. Based on the above steps, the formula of the process-supervised reward model given in this embodiment is expressed as follows:

[0042] Rθ=(s,a);

[0043] In the formula, s represents the current state of the task description, webpage screenshot and context, a represents the next generated step, and R θ Represents the PRM score of a task process. The PRM score is defined as the product of the correctness probability of each step.

[0044] Step 3: Use the process supervision reward model to perform reinforcement learning training on the multimodal large model to obtain a task process decomposition model;

[0045] In this step, the training goal of reinforcement learning training is to make the generation results of the multimodal large model more in line with the preferences of the process supervision reward model based on the above process supervision reward model, that is, to make each step of its generation more correct.

[0046] In this step, the reinforcement learning model training uses the PPO algorithm to perform reinforcement learning on the multimodal large model. PPO (Proximal Policy Optimization) is a stable and efficient policy optimization algorithm that is widely used in reinforcement learning tasks. PPO limits the amplitude of each policy update to prevent drastic changes in the policy and ensure the stability of the learning process. The goal of PPO policy optimization is to maximize the reward J(φ), which can be expressed as:

[0047]

[0048] Among them, E is the expectation, τ represents a strategy The generated trajectory, R θ is a process supervision reward model; t represents the index of the time step, representing a moment when an observation is obtained from the environment, an action is taken, and a reward is obtained.

[0049] Step 4: Generate detailed RPA steps based on the trained task process decomposition model, and then call the general large model API to convert each RPA step into actual RPA instructions. The RPA instructions form a complete RPA process. The process of this step is as follows: Figure 2 shown.

[0050] In this step, the conversion of RPA instructions is to select the currently powerful general large model API and build an intelligent agent by writing prompts. This task is relatively simple, and because only the model is allowed to make choices, the hallucination problem is relatively small.

[0051] In this step, the transformation process of the agent is to describe in detail to the agent which actions and operation objects are available in RPA; then let the agent select specific RPA actions and operation objects according to the detailed step description; at the same time, let the agent see all the previously generated RPA instructions. If the previous instructions cannot be naturally connected to the current step, let the agent generate multiple RPA instructions so that the process can run completely.

[0052] Based on the method in this embodiment, two different application examples of automatic generation of RPA processes are given according to different ways of obtaining web page screenshots.

[0053] 1. Manually provide webpage screenshots

[0054] The process of this step is as follows Figure 3 As shown in the figure, a task description and all the page screenshots that may be involved in the task are provided manually and handed over to the task process decomposition model trained above. The task process decomposition model decomposes the task into detailed RPA steps. Then, the intelligent agent built through the general large model API selects specific RPA actions and operation objects, and finally forms a complete RPA process.

[0055] 2. Automatically obtain web page screenshots

[0056] The process of this step is as follows Figure 4 As shown, since there is no ready-made page screenshot, you need to execute the instructions step by step to determine whether the page has changed. The specific steps are as follows:

[0057] s1: The initial input content is the task description and a screenshot of the current page. If there is no initial page, the general large model API can provide an initial URL based on the task description and open the URL;

[0058] s2: The trained task flow decomposition model is used to gradually decompose the task into detailed RPA steps, pausing after each step is generated;

[0059] s3: After each step of the task process decomposition model is generated, the agent built by the general large model API is synchronously called to select a specific RPA action and operation object to form an executable RPA instruction;

[0060] s4: Execute the RPA instructions generated in the previous step to determine whether the page has changed. If the page has not changed, continue to generate RPA steps using the task flow decomposition model. If the page has changed, re-input the screenshot of the new page into the task flow decomposition model, and use the previously generated steps as the previous input to continue generating subsequent steps.

[0061] s5: Until all steps are generated to form a complete RPA process.

[0062] In summary, the innovative features of the present invention are as follows:

[0063] 1. Traditional RPA process automatic generation solutions have no image input when performing task process decomposition, which makes the large model unable to determine whether the content contained in the process exists on the web page, thereby generating unfeasible process steps. Based on the page description fed back by the multimodal model, the task process is re-decomposed by the large language model. Not only does its page description fail to fully reflect all page information, but it also needs to be regenerated multiple times, which is inefficient. The present invention innovatively uses a multimodal large model to decompose the task process. By directly inputting web page screenshots, the multimodal large model can fully understand the page logic, thereby more efficiently generating more accurate process steps.

[0064] 2. When performing task process decomposition, the traditional RPA process automatic generation solution does not use reinforcement learning methods to align the output process steps with RPA instructions. This may result in the inability to convert the step descriptions in the process into executable RPA instructions, ultimately rendering the process invalid. However, the present invention incorporates reinforcement learning methods during the training process, improving the ability of multimodal large models to decompose task processes.

[0065] 3. The reward model in traditional reinforcement learning is based on result supervision. In the generation of multi-step process tasks, the impact of errors grows exponentially. Therefore, only using a reward model with result supervision cannot allow the model to locate the specific location of the error, which easily leads to the problem of one wrong step leading to all wrong steps. The present invention innovatively uses a process supervision reward model, which enables the multimodal large model to effectively learn how the generation of each step of the process conforms to the business logic and aligns with the RPA instructions.

[0066] 4. In the instruction generation stage, traditional RPA processes are fine-tuned or reinforced based on open source big models / self-developed models. This not only brings high training costs, but also limits the perspective to the generation of a single step. The present invention innovatively uses the general big model API to build an intelligent agent, which can accurately convert step descriptions into executable RPA instructions, and also allows the intelligent agent to see all previously generated RPA instructions; if the previous instructions cannot be naturally connected to the current step, the intelligent agent can generate multiple RPA instructions so that the process can run completely.

[0067] 5. Since the present invention innovatively utilizes multimodal information, web page screenshots are required as input. In addition to conventional manual screenshots, the present invention also proposes a solution for automatically obtaining web page screenshots and automatically generating RPA processes, further reducing labor costs.

Claims

1. A method for realizing automatic generation of RPA processes by using reinforcement learning, characterized in that: The steps include: Step 1: Train the multimodal large model so that it has the ability to decompose the task flow based on page screenshots and task descriptions; Step 2: Train the process supervision reward model so that the process supervision reward model can judge whether the next generation step is correct based on the page screenshot, task description and context; Step 3: Use the process supervision reward model to perform reinforcement learning training on the multimodal large model to obtain a task process decomposition model; Step 4: Generate detailed RPA steps based on the trained task process decomposition model, and then call the general large model API to convert each RPA step into actual RPA instructions, which form a complete RPA process.

2. The method for realizing automatic generation of RPA processes by reinforcement learning according to claim 1, characterized in that: In step 1, the training of the multimodal large model includes supervised training and unsupervised training; the supervised training is used to enable the multimodal large model to understand RPA tasks and web page screenshots; the unsupervised training is used to enable the multimodal large model to output task steps according to RPA tasks and web page screenshots.

3. The method for realizing automatic generation of RPA processes by reinforcement learning according to claim 2, characterized in that: The training data for training the multimodal large model includes documents of the RPA platform, open source datasets of common web page descriptions, task descriptions of existing RPA applications and text descriptions of each step, and screenshots of page changes during the runtime of the RPA application.

4. The method for realizing automatic generation of RPA processes by reinforcement learning according to claim 1, characterized in that: The process supervision reward model is formulated as follows: Rθ=(s,a); In the formula, s represents the current state of the task description, webpage screenshot and context, a represents the next generated step, and R θ Represents the PRM score of a task process. The PRM score is defined as the product of the correctness probability of each step.

5. The method for realizing automatic generation of RPA processes by reinforcement learning according to claim 4 is characterized in that: The training data of the supervised reward model in the training process uses a general large model API to generate a large number of detailed task steps, and then predicts each generated task step based on the task description, web page screenshots and context through manual annotation. The prediction results are divided into three categories: correct, neutral and wrong.

6. The method for realizing automatic generation of RPA processes by reinforcement learning according to claim 1, characterized in that: In step 3, the reinforcement learning training uses the PPO strategy optimization algorithm, the goal of which is to maximize the return It is expressed as: Among them, E is the expectation, τ represents a strategy The generated trajectory, R θ is a process supervision reward model; t represents the index of the time step, representing a moment when an observation is obtained from the environment, an action is taken, and a reward is obtained.

7. The method for realizing automatic generation of RPA processes by reinforcement learning according to claim 1, characterized in that: In step 4, the general large model API converts each RPA step into actual RPA instructions by writing prompts to build an agent and perform conversion with the agent.

8. The method for realizing automatic generation of RPA processes by reinforcement learning according to claim 7, characterized in that: The transformation process of the agent is to describe in detail to the agent which actions and operation objects are available in RPA; then let the agent select specific RPA actions and operation objects according to the detailed step description; at the same time, let the agent see all the previously generated RPA instructions. If the previous instructions cannot be naturally connected to the current step, let the agent generate multiple RPA instructions so that the process can run completely.

Images