A model training method, device, system and readable storage medium

Abstract

A model training method, device, system and readable storage medium are applied to the field of reinforcement learning technology. The method includes: judging whether the reinforcement learning model has converged; if the reinforcement learning model does not converge, randomly selecting a software environment and generating a random number; judging whether the number of machines is greater than a preset threshold; if not, randomly selecting a piece of training data; performing offline reinforcement learning on the training data; if so, performing online reinforcement learning on the software environment; after completing reinforcement learning, judging again whether the reinforcement learning model has converged; if the reinforcement learning model does not converge, randomly selecting a software environment again and generating a random number; if the reinforcement learning model converges, then ending the training. Therefore, the present application is based on a reinforcement learning algorithm training model, so that the model fully learns known fault-tolerant knowledge and can automatically handle similar errors when they occur, greatly improving the fault tolerance of the process.

Description

Technical Field

[0001] The present application relates to the field of reinforcement learning technology, and in particular to a model training generation method, device, system and readable storage medium. Background Art

[0002] Computer software operations in industrial production are often repetitive and regular. To reduce the effort and cost of repetitive tasks, RPA (Robotic Process Automation) technology has emerged. RPA automates mouse and keyboard operations according to specific rules to complete specific tasks. The entire process simulates human operation, with the identification and manipulation of software components being a key component.

[0003] In actual production, software components may change due to updates or exceptions, and operations may not be executed as planned due to accidents. Therefore, fault tolerance is an important part of RPA and the key to the stable operation of automated processes.

[0004] However, the operating elements in the software require the development of targeted element positioning methods. Once the elements are changed or updated, they need to be re-adapted, resulting in poor adaptability to element changes and low fault tolerance. Summary of the Invention

[0005] The present application provides a model training method, device, system and readable storage medium, which can improve the adaptability and fault tolerance of element changes.

[0006] This application discloses the following technical solutions:

[0007] In a first aspect, the present application discloses a model training generation method, the method comprising:

[0008] Determine whether the reinforcement learning model has converged;

[0009] If the reinforcement learning model does not converge, randomly selecting a software environment and generating a random number;

[0010] If the reinforcement learning model converges, the training ends;

[0011] Determine whether the random number is greater than a preset threshold;

[0012] If not, randomly select a piece of training data, the training data coming from the software environment; perform offline reinforcement learning on the training data;

[0013] If so, performing online reinforcement learning on the software environment;

[0014] After the reinforcement learning is completed, the operation of randomly selecting a software environment and generating a random number is performed again.

[0015] Optionally, determining whether the reinforcement learning model has converged includes:

[0016] Based on the DQN and DDQN combined with imitation learning algorithms, check whether the reinforcement learning model has converged.

[0017] Optionally, the method for generating the training data includes:

[0018] Determine whether there are unrecorded errors in the RPA process and the corresponding fault tolerance process;

[0019] If there are unrecorded errors and corresponding fault-tolerant processes, the simulation generates the first error;

[0020] Determine whether the RPA process can complete the fault-tolerant processing process;

[0021] If so, record the fault-tolerant processing flow;

[0022] If not, perform manual intervention; based on the manual intervention process, record the fault-tolerant processing process;

[0023] After recording the fault-tolerant processing process, re-determine whether there are any unrecorded errors and corresponding fault-tolerant processes in the RPA process;

[0024] If there are unrecorded errors and corresponding fault-tolerant processes, the simulation generates a second error;

[0025] If there are no unrecorded errors and corresponding fault-tolerant processes, the normal processing process is recorded;

[0026] Based on the normal processing flow, training data is generated.

[0027] Optionally, the fault-tolerant processing process includes: each step of the operation in the fault-tolerant processing process, the corresponding environment of the operation, and the environmental status before and after the operation.

[0028] In a second aspect, the present application discloses a model training generation device, the device comprising: a first judgment module, a selection generation module, a second judgment module, a selection module, an offline reinforcement learning module, and an online reinforcement learning module;

[0029] The first judgment module is used to judge whether the reinforcement learning model has converged; if the reinforcement learning model has converged, the training is terminated;

[0030] The selection generation module is used to randomly select a software environment and generate a random number if the reinforcement learning model does not converge;

[0031] The second judging module is configured to judge whether the random number is greater than a preset threshold;

[0032] The selection module is configured to randomly select a piece of training data if the random number is not greater than a preset threshold range, the training data being from the software environment;

[0033] The offline reinforcement learning module is used to perform offline reinforcement learning on the training data;

[0034] The online reinforcement learning module is configured to perform online reinforcement learning on the software environment if the random number is greater than a preset threshold range;

[0035] The selection generation module is further configured to, after completing reinforcement learning, again execute the operation of randomly selecting a software environment and generating a random number.

[0036] Optionally, the first judgment module is specifically configured to:

[0037] Based on the DQN and DDQN combined with imitation learning algorithms, check whether the reinforcement learning model has converged.

[0038] Optionally, the training data generating device includes: a first judgment module, a simulation module, a second judgment module, a first recording module, a manual intervention module, a second recording module, and a generation module;

[0039] The first judgment module is used to judge whether there are unrecorded errors and corresponding fault-tolerant processes in the RPA process;

[0040] The simulation module is configured to simulate the generation of a first error if there is an unrecorded error and a corresponding fault-tolerant process;

[0041] The second judgment module judges whether the RPA process can complete the fault tolerance processing process;

[0042] The first recording module is configured to record the fault-tolerant processing process if the RPA process can complete the fault-tolerant processing process;

[0043] The manual intervention module is used to perform manual intervention if the RPA process cannot complete the fault-tolerant processing process;

[0044] The first recording module is further configured to record the fault-tolerant processing process based on the manual intervention process;

[0045] The first judgment module is further configured to, after recording the fault-tolerant processing process, re-judgment whether there are any unrecorded errors and corresponding fault-tolerant processes in the RPA process;

[0046] The simulation module is further configured to simulate the generation of a second error if there is an unrecorded error and a corresponding fault-tolerant process;

[0047] The second recording module is configured to record the normal processing process if there are no unrecorded errors and corresponding fault-tolerant processes;

[0048] The generating module is used to generate training data based on the normal processing flow.

[0049] Optionally, the fault-tolerant processing process includes: each step of the operation in the fault-tolerant processing process, the corresponding environment of the operation, and the environmental status before and after the operation.

[0050] In a third aspect, the present application discloses a model training generation device, comprising: a memory and a processor;

[0051] The memory is used to store programs;

[0052] The processor is configured to execute the program to implement the various steps of the method described in the first aspect.

[0053] In a fourth aspect, the present application discloses a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0054] Compared with the existing technology, this application has the following beneficial effects:

[0055] This application leverages the fault-tolerance strategies of existing RPA processes, incorporates a small amount of manual assistance, and trains models based on reinforcement learning algorithms. This allows the model to fully learn known fault-tolerance knowledge and automatically handle similar errors when they occur, greatly improving the process's fault tolerance. Furthermore, this application trains processes with the same functionality, using reinforcement learning algorithms, in different software environments. This enables a single model to recognize multiple elements in multiple styles and forms, allowing it to flexibly respond to system changes and significantly improving element location fault tolerance. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0057] Figure 1 A flow chart of a model training method provided in an embodiment of the present application;

[0058] Figure 2A diagram of different layout styles of a login component provided in an embodiment of the present application;

[0059] Figure 3 A schematic diagram of a component usage process method provided in an embodiment of the present application;

[0060] Figure 4 A flow chart of a training data generation method provided in an embodiment of the present application;

[0061] Figure 5 A schematic diagram of a model training device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following is an introduction to the technical terms involved in this application.

[0063] RPA technology, or Robotic Process Automation, uses specific robotic software to simulate human operations on computers, automatically executing process tasks according to rules. This technology replaces repetitive, regular, and stable manual operations with robotic operations. To enable RPA to assist humans in their work, it's necessary to identify the manual processes so that RPA robots can execute them, thereby replacing manual labor.

[0064] Fault tolerance refers to a product's ability to withstand erroneous operations, specifically the probability of errors occurring during a product operation and the probability of them being resolved. A fault-tolerant process automatically handles and avoids abnormal errors that occur during RPA execution, such as click failures, input failures, and pop-up windows, ensuring the process continues normally. Taking "login" as an example, the RPA process involves a machine automatically completing the following sequence of operations on behalf of a human: 1. Click the username input box and enter the username; 2. [Fault-tolerant process] verifies that the username is entered successfully; if it fails, the process returns to step 1 and repeats; 3. Click the password input box and enter the password; 4. [Fault-tolerant process] verifies that the password is entered successfully; if it fails, the process returns to step 3 and repeats; 5. Click the login button. Steps 2 and 4 are fault-tolerant processes, part of the RPA process, designed to prevent text box errors due to input errors. The RPA process for the login task can be free of these two fault-tolerant sub-processes, but if an input failure exception occurs, the login will fail, and the RPA process will become stuck and unable to continue. However, there are many abnormal situations, and it is very costly to design many fault-tolerant processes for each step, such as text box input operations.

[0065] As mentioned earlier, in actual production using RPA technology, software components may change due to updates or anomalies, and operations may not execute as planned due to unexpected events. Therefore, fault tolerance is a crucial component of RPA and key to the stable operation of automated processes. However, specific methods must be developed to locate operational elements in the software. Any changes or updates to these elements require re-adaptation, resulting in poor adaptability to changes and low fault tolerance.

[0066] To address the above-mentioned shortcomings, this application provides a model training method, device, system, and readable storage medium. This method utilizes the fault-tolerant strategies in existing RPA processes, incorporates a small amount of manual assistance, and trains the model based on a reinforcement learning algorithm. This allows the model to fully learn known fault-tolerant knowledge and automatically handle similar errors when they occur, greatly improving the fault tolerance of the process. Furthermore, this application trains processes with the same function under different software environments based on a reinforcement learning algorithm, enabling a single model to possess the ability to recognize multiple styles and forms of elements, flexibly respond to system changes, and significantly improve the fault tolerance of element positioning.

[0067] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0068] See also Figure 1 , which is a flow chart of a model training method provided in an embodiment of the present application. The method includes:

[0069] S101: Check whether the reinforcement learning model has converged.

[0070] Reinforcement learning is a general term for a class of algorithms. In one embodiment, algorithms such as DQN and DDQN combined with imitation learning can be used as the basic algorithms of this embodiment, and this application does not limit them.

[0071] According to the above training algorithm, check whether the reinforcement learning model has converged. When the reinforcement learning model is being trained, each round of training will output an indicator reawrd, that is, an indicator return. When the average indicator return of the last n rounds is greater than a certain threshold, it can be determined that the reinforcement learning model has converged. This application does not limit the specific number of n. Among them, the threshold is a value set in advance by the reinforcement learning model, which can be 0.9, or 0.8, etc., and this application does not limit it. If the reinforcement learning model checked converges, the training is terminated and S107 is executed directly; if the reinforcement learning model does not converge, the training process continues and S102 is executed.

[0072] S102: Randomly select a software environment for reinforcement learning.

[0073] Software environment refers to a graphical software operating environment, which contains several object elements such as buttons and text boxes that can be used for software operations. Software operations refer to operations such as mouse, keyboard, and internal parameter changes on the graphical software interface. Figure 2 , this figure shows different layout styles of a login component provided in an embodiment of the present application. The login interface is a typical software environment, including an operating system software environment (windows file browser), a browser software environment (chrome), etc.

[0074] The same operational flow occurs across different software environments. In one implementation, the sequence of mouse and keyboard operations involved in "logging in" constitutes one operational flow. These different software environments form a set E. The training algorithm or training process randomly selects one environment, e, from these environments as the interactive training environment for this reinforcement learning session.

[0075] Through interactive learning of the same functional elements in each round and in different environments, the model can recognize different styles or code structures of the same element, thereby obtaining element positioning fault tolerance. Among them, the element positioning fault tolerance is specifically that when there are one or more operable elements in the software page, RPA needs to locate them first when operating. It can be based on image features, that is, the UI style presented by the element, or based on code features, that is, the implementation code of the element. This application does not limit it. The image features and code features of the operable elements of the same function may be different in different software environments. Updates to the software environment will also cause changes in the above features. RPA needs to be able to adapt to these differences and changes and still recognize the operable elements of the same function.

[0076] In one embodiment, see Figure 2 , this figure is a diagram of different layout styles of a login component provided in an embodiment of the present application. Layout style refers to the visual presentation method of the software environment and is part of the software environment. Taking the login process as a scenario, each subsystem within the company, external mailbox, etc. has its own login page. The styles of these login pages may be different, but the login process is nothing more than entering the user name, entering the password, and clicking the login button. RPA can randomly select a system's login page as the training environment for this round and perform reinforcement learning.

[0077] S103: Generate a random number between 0 and 1, and determine whether the random number is greater than a preset threshold range.

[0078] A preset threshold range controls the ratio of online to offline reinforcement learning. Online reinforcement learning refers to conventional reinforcement learning conducted online. The model uses a trial-and-error approach to explore and identify the operational procedures for completing a task. Each time, a software operation, A, is selected and executed within the software environment to verify its validity and necessity. Because the range of software operations is vast and the correct operational procedures can be lengthy, online learning is often slow and may not be able to find the correct operational procedures for completing the task. Offline reinforcement learning involves manually demonstrating the correct operational procedures to generate training data. The model then learns these correct operational procedures, avoiding the high trial costs associated with online reinforcement learning. However, this training data is often limited and cannot cover all scenarios. Therefore, it is often used in conjunction with online reinforcement learning.

[0079] The reinforcement learning model generates a random number between 0 and 1 and determines whether the random number is greater than a preset threshold value θ. If the random number is greater than the preset threshold value θ, online reinforcement learning is used and S106 is executed; otherwise, S104 is executed.

[0080] S104: Randomly select a piece of training data.

[0081] A piece of training data is randomly selected from the software environment e. The training data includes each operation a, the software environment e corresponding to the operation, and the environment states s and s' before and after the operation.

[0082] S105: Offline reinforcement learning method training.

[0083] Based on the training data in S104, offline reinforcement learning is performed. Offline reinforcement learning does not interact with the environment and relies solely on offline training data, hence the name "offline." For each piece of training data (s, a), the training algorithm inputs the environment state s. The current model predicts the software operation a' based on s. The training algorithm then calculates the difference between a and a' as the loss, and backpropagates the model parameters. This ensures that each predicted result a' increasingly approximates the training data a, thereby fully learning the process's fault-tolerance characteristics from the training data. The algorithm learns the fault-tolerance knowledge contained in the training data, which records the fault-tolerance handling process of existing RPA processes and the partial manual intervention process to resolve errors.

[0084] After the offline reinforcement learning method training is completed, S101 is executed to re-determine whether the model has converged. If not, S102 is continued. If so, the training is completed and S107 is executed.

[0085] S106: Online reinforcement learning method training.

[0086] Online reinforcement learning is performed by interacting with the software environment e. Online reinforcement learning requires interaction with the environment, hence the name "online." In the case of online training, the training algorithm obtains the current software environment's environmental state s and predicts the software operation a based on s. A is executed in the software environment, generating a new environmental state s'. The above steps are repeated until the task is completed (reward = 1) or the timeout is terminated (reward = -1). The model parameters are updated through backpropagation using the reward as the loss. When the reward is 1, it indicates that the current sequence of operations is valid, and the model will be more inclined to predict this result. The essence of online learning is to repeatedly explore the environment. There is no standard answer, and you have to find the correct operation path yourself, so you can explore more possibilities.

[0087] The online reinforcement learning model independently tries and fails, learns how to complete tasks, can discover and adapt to more error situations, and forms a complementary relationship with offline reinforcement learning.

[0088] After the online reinforcement learning method training is completed, S101 is executed to re-determine whether the model has converged. If not, S102 is continued. If so, the training is completed and S107 is executed.

[0089] S107: Training ends.

[0090] If the reinforcement learning model is converged, the training is terminated.

[0091] The adaptive fault-tolerant components generated based on the "component training process" can be directly applied to RPA process development, simplifying the multi-step operation process covered by the component into a single component call. Figure 3 This figure is a schematic diagram of a component usage process method provided by an embodiment of the present application. Operations 1, 2, and 3 in the original process can be replaced with component 1, eliminating the need for additional fault-tolerant processes. This significantly reduces development costs and improves the stability of RPA implementation.

[0092] This application utilizes the fault-tolerant strategies in existing RPA processes, integrates a small amount of manual assistance, and trains models based on reinforcement learning algorithms, so that the model can fully learn known fault-tolerant knowledge and automatically handle similar errors when they occur, greatly improving the fault tolerance of the process. In addition, this application is based on reinforcement learning algorithms for processes with the same function, and is trained in different software environments, so that a single model has the ability to recognize multiple styles and forms of elements, can flexibly respond to system changes, and greatly improves the fault tolerance of element positioning. Furthermore, this application does not require targeted rule configuration, and can automatically generate training data based on existing RPA processes, making the component generation cost low. In addition, the adaptive fault-tolerant components generated by this application can directly replace several existing processes, making it more convenient and efficient to use.

[0093] See also Figure 4 , which is a flow chart of a training data generation method provided in an embodiment of the present application. The method includes:

[0094] S201: Check whether there are unrecorded errors and corresponding fault-tolerant processes in all fault-tolerant modules in the RPA process.

[0095] After simulating an existing RPA process, check all fault-tolerant modules within the RPA process for undocumented errors and corresponding fault-tolerant processes. An RPA process is an automated process that simulates human operations and operates specific software. A fault-tolerant process automatically handles and avoids abnormal errors that occur during RPA execution, such as click failures, input failures, and newly added pop-up windows, allowing the process to continue normally.

[0096] If there are unrecorded errors and corresponding fault-tolerant processes, then S202 is executed; if there are no unrecorded errors and corresponding fault-tolerant processes, then S206 is executed.

[0097] In some embodiments, in a login scenario, possible errors include entering a username or password without focusing on the input box, and partially missing an input.

[0098] S202: Simulation generates an error.

[0099] If there are undocumented fault-tolerant processes, we first manually simulate the corresponding errors in the software environment. The software environment refers to a graphical software operating environment, which contains several objects that can perform software operations, such as buttons and text boxes. Software operations refer to operations on the graphical software interface using the mouse, keyboard, or changing internal parameters.

[0100] S203: Determine whether the RPA can handle the error.

[0101] After simulating an error, a human determines whether the RPA can handle it. Normally, the process RPA can handle the error and mitigate it. If so, proceed to S205. However, in some cases, the RPA cannot handle the error, requiring human intervention. In this case, proceed to S204.

[0102] In some embodiments, the text box input fails and the RPA can handle the error, that is, the RPA enters the correct content in the text box.

[0103] S204: Perform manual intervention.

[0104] Manual intervention is required to resolve abnormal errors that RPA cannot handle.

[0105] In some embodiments, the text box input fails and RPA cannot handle the error, which requires manual processing, that is, manual input of the correct content in the text box.

[0106] S205: Record the fault-tolerance processing process.

[0107] Whether implementing RPA fault tolerance or manual intervention, monitoring the software environment's operations and state is crucial. This ensures that each operation a, the corresponding environment e, and the environmental states s and s' before and after the operation are recorded. The environmental state refers to the current state of the RPA-implemented software, including visual information on the software interface and internal software parameters.

[0108] In some embodiments, each operation a, the corresponding environment e, and the environmental states s and s' before and after the operation may be recorded in a storage unit, or in other units or modules, which is not limited in this application.

[0109] S206: Generate training data.

[0110] When there are no unrecorded fault-tolerant processes, that is, when no errors occur, the RPA process executes all steps normally to complete the task, recording each step a of the normal process, the corresponding environment e, and the environmental states s and s' before and after the operation, thus generating training data.

[0111] This application utilizes the fault-tolerant strategies in existing RPA processes, integrates a small amount of manual assistance, and trains models based on reinforcement learning algorithms, so that the model can fully learn known fault-tolerant knowledge and automatically handle similar errors when they occur, greatly improving the fault tolerance of the process. In addition, this application is based on reinforcement learning algorithms for processes with the same function, and is trained in different software environments, so that a single model has the ability to recognize multiple styles and forms of elements, can flexibly respond to system changes, and greatly improves the fault tolerance of element positioning. Furthermore, this application does not require targeted rule configuration, and can automatically generate training data based on existing RPA processes, making the component generation cost low. In addition, the adaptive fault-tolerant components generated by this application can directly replace several existing processes, making it more convenient and efficient to use.

[0112] See also Figure 5 , which is a schematic diagram of a model training device provided in an embodiment of the present application. The device 500 includes: a first judgment module 501, a selection generation module 502, a second judgment module 503, a selection module 504, an offline reinforcement learning module 505, and an online reinforcement learning module 506;

[0113] The first judgment module 501 is used to judge whether the reinforcement learning model has converged; if the reinforcement learning model has converged, the training is terminated;

[0114] A selection generation module 502 is configured to randomly select a software environment and generate a random number if the reinforcement learning model does not converge;

[0115] The second judgment module 503 is used to judge whether the random number is greater than a preset threshold;

[0116] A selection module 504 is configured to randomly select a piece of training data if the random number is not greater than a preset threshold range, the training data being from a software environment;

[0117] Offline reinforcement learning module 505, used for performing offline reinforcement learning on training data;

[0118] An online reinforcement learning module 506 is configured to perform online reinforcement learning on the software environment if the random number is greater than a preset threshold range;

[0119] The selection generation module 502 is further configured to re-execute the operation of randomly selecting a software environment and generating a random number after completing the reinforcement learning.

[0120] This application utilizes the fault-tolerant strategies in existing RPA processes, integrates a small amount of manual assistance, and trains models based on reinforcement learning algorithms, so that the model can fully learn known fault-tolerant knowledge and automatically handle similar errors when they occur, greatly improving the fault tolerance of the process. In addition, this application is based on reinforcement learning algorithms for processes with the same function, and is trained in different software environments, so that a single model has the ability to recognize multiple styles and forms of elements, can flexibly respond to system changes, and greatly improves the fault tolerance of element positioning. Furthermore, this application does not require targeted rule configuration, and can automatically generate training data based on existing RPA processes, making the component generation cost low. In addition, the adaptive fault-tolerant components generated by this application can directly replace several existing processes, making it more convenient and efficient to use.

[0121] The embodiments of the present application also provide corresponding generation devices and computer storage media for implementing the solutions provided in the embodiments of the present application.

[0122] The device includes a memory and a processor, the memory is used to store instructions or codes, and the processor is used to execute instructions or codes so that the device executes a lighting control method of any embodiment of the present application.

[0123] The computer storage medium stores code. When the code is executed, the device executing the code implements the method described in any embodiment of the present application.

[0124] The "first" and "second" (if any) in the names mentioned in the embodiments of this application are only used as name identifiers and do not mean the first or second in order.

[0125] Through the description of the above embodiments, it can be known that those skilled in the art can clearly understand that all or part of the steps in the above embodiment methods can be implemented by means of software plus a general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a storage medium, such as a read-only memory (ROM) / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a router) to execute the methods described in each embodiment or certain parts of the embodiments of the present application.

[0126] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0127] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A model training generation method, characterized in that: The method comprises: Determine whether the reinforcement learning model has converged; If the reinforcement learning model does not converge, randomly selecting a software environment and generating a random number; If the reinforcement learning model converges, the training ends; Determine whether the random number is greater than a preset threshold; If not, randomly select a piece of training data, the training data coming from the software environment; perform offline reinforcement learning on the training data; If so, performing online reinforcement learning on the software environment; After completing the reinforcement learning, the operation of randomly selecting a software environment and generating a random number is performed again; The method for generating training data includes: Determine whether there are unrecorded errors in the RPA process and the corresponding fault tolerance process; If there are unrecorded errors and corresponding fault-tolerant processes, the simulation generates the first error; Determine whether the RPA process can complete the fault-tolerant processing process; If so, record the fault-tolerant processing flow; If not, perform manual intervention; based on the manual intervention process, record the fault-tolerant processing process; After recording the fault-tolerant processing process, re-determine whether there are any unrecorded errors and corresponding fault-tolerant processes in the RPA process; If there are unrecorded errors and corresponding fault-tolerant processes, the simulation generates a second error; If there are no unrecorded errors and corresponding fault-tolerant processes, the normal processing process is recorded; Based on the normal processing flow, training data is generated.

2. The method according to claim 1, characterized in that Determining whether the reinforcement learning model has converged includes: Based on the DQN and DDQN combined with imitation learning algorithms, check whether the reinforcement learning model has converged.

3. The method according to claim 1, characterized in that The fault-tolerant processing flow includes: each step of the operation in the fault-tolerant processing process, the corresponding environment of the operation, and the environmental status before and after the operation.

4. A model training generation device, characterized in that: The device includes: a first judgment module, a selection generation module, a second judgment module, a selection module, an offline reinforcement learning module, and an online reinforcement learning module; The first judgment module is used to judge whether the reinforcement learning model has converged; if the reinforcement learning model has converged, the training is terminated; The selection generation module is used to randomly select a software environment and generate a random number if the reinforcement learning model does not converge; The second judging module is configured to judge whether the random number is greater than a preset threshold; The selection module is configured to randomly select a piece of training data if the random number is not greater than a preset threshold range, the training data being from the software environment; The offline reinforcement learning module is used to perform offline reinforcement learning on the training data; The online reinforcement learning module is configured to perform online reinforcement learning on the software environment if the random number is greater than a preset threshold range; The selection generation module is further configured to, after completing reinforcement learning, again execute the operation of randomly selecting a software environment and generating a random number; The training data generation device includes: a first judgment unit, a simulation module, a second judgment unit, a first recording module, a manual intervention module, a second recording module, and a generation module; The first judgment unit is used to judge whether there are unrecorded errors and corresponding fault-tolerant processes in the RPA process; The simulation module is configured to simulate the generation of a first error if there is an unrecorded error and a corresponding fault-tolerant process; The second judgment unit judges whether the RPA process can complete the fault tolerance processing process; The first recording module is configured to record the fault-tolerant processing process if the RPA process can complete the fault-tolerant processing process; The manual intervention module is used to perform manual intervention if the RPA process cannot complete the fault-tolerant processing process; The first recording module is further configured to record the fault-tolerant processing process based on the manual intervention process; The first judgment unit is further configured to, after recording the fault-tolerant processing process, re-judgment whether there are any unrecorded errors and corresponding fault-tolerant processes in the RPA process; The simulation module is further configured to simulate the generation of a second error if there is an unrecorded error and a corresponding fault-tolerant process; The second recording module is configured to record the normal processing process if there are no unrecorded errors and corresponding fault-tolerant processes; The generating module is used to generate training data based on the normal processing flow.

5. The device according to claim 4, characterized in that The first judgment module is specifically configured to: Based on the DQN and DDQN combined with imitation learning algorithms, check whether the reinforcement learning model has converged.

6. The device according to claim 4, characterized in that The fault-tolerant processing flow includes: each step of the operation in the fault-tolerant processing process, the corresponding environment of the operation, and the environmental status before and after the operation.

7. A model training and generation device, characterized in that: include: memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the method according to any one of claims 1 to 3.

8. A computer storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the method according to any one of claims 1 to 3 is implemented.

Images