RPA process generation method and device based on large model, medium and product

Abstract

The embodiment of the invention provides an RPA process generation method and device based on a large model, a medium and a product. The method comprises the following steps: receiving business demand information in a natural language form input by a user; based on a preset large model and a business capability knowledge base, analyzing the business demand information, and determining a business operation sequence matched with the business scene; based on a robot process automation (RPA) component knowledge base and a preset multi-modal visual understanding model, analyzing the service operation sequence into a single-step executable atomic operation sequence; and according to the atomic operation sequence and the large model, generating an executable code of the target RPA component. According to the scheme, a user is supported to input business process requirements in a natural language form, the requirements are automatically converted into the business process operation sequence based on a large model, and the corresponding RPA code is generated and executed, so that the understanding cost of the user is greatly reduced, the execution efficiency of the business process is improved, and the labor cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically to a large-model-based RPA process generation method, device, medium, and product. Background Art

[0002] In the government and enterprise sectors, automation tools are often used to improve business process efficiency, handling repetitive and rule-based tasks. Robotic Process Automation (RPA) technology is a typical example. Traditional RPA technology uses robots with predefined rules to simulate human actions to complete business processes, such as filling out reports and sending and receiving emails. Process generation typically involves users dragging and dropping RPA components and entering parameters and variables to create business processes. This allows for low-code or even no-code processes, but requires users to master the use of RPA components, posing an additional learning cost for non-experts. This creates a technical barrier to entry and operational complexity, and it also suffers from poor flexibility and scalability, making it difficult to adapt to rapidly changing business needs within an enterprise. With the development of artificial intelligence (AI), particularly breakthroughs in deep learning, researchers are attempting to combine traditional RPA with AI technologies, such as large language models, to further improve business process execution efficiency. While existing traditional RPA designers enable low-code or no-code design, technicians must manually drag and drop components to update RPA processes, requiring considerable background knowledge and a technical barrier to entry. Every time a new business is faced, the RPA process needs to be manually designed. The repetitive and tedious manual design reduces the efficiency of business process generation and makes it difficult to adapt to the company's frequently changing business.

[0003] There are also solutions in the existing technology that combine artificial intelligence technology, but most of these technical solutions are limited to generating business processes based on large language models. They use a static process generation method and cannot monitor the execution effect of each step of the process. They cannot achieve adaptive dynamic response, that is, they cannot dynamically update the business status based on the execution result of the previous step, and then accurately call the RPA component in the next step. They only call the large language model to understand user business needs and generate RPA processes based only on natural language processing (NLP) technology. They do not utilize existing multimodal data information and have high requirements on the capabilities of the large model itself. Summary of the Invention

[0004] At least one embodiment of the present application provides a large-model-based RPA process generation method, device, medium, and product to address the problems in the prior art of traditional RPA technology, such as poor flexibility and scalability, high technical barriers, and the fact that existing RPA products combined with artificial intelligence technology do not consider dynamic supervision of business process execution effects.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for generating an RPA process based on a large model, including:

[0007] Receive business requirement information in natural language input from users;

[0008] Based on the preset big model and business capability knowledge base, the business demand information is analyzed to determine the business operation sequence adapted to the business scenario;

[0009] Based on the RPA component knowledge base and the preset multimodal visual understanding model, the business operation sequence is parsed into a single-step executable atomic operation sequence;

[0010] Generate executable code of the target RPA component according to the atomic operation sequence and the large model.

[0011] Optionally, based on a preset large model and business capability knowledge base, the business requirement information is parsed to determine a business operation sequence that is adapted to the business scenario, including:

[0012] Based on the preset big model and business capability knowledge base, the business requirement information in natural language form is converted into structured information;

[0013] The structured information is subjected to semantic analysis, context understanding, and task decomposition to determine a business operation sequence adapted to the business scenario corresponding to the business requirement information; each operation sequence in the business operation sequence includes one or more RPA atomic operations.

[0014] Optionally, based on the Robotic Process Automation (RPA) component knowledge base and a preset multimodal visual understanding model, the business operation sequence is parsed into a single-step executable atomic operation sequence, including:

[0015] Using the collected data information of the business scenario, fine-tuning the initial multimodal visual understanding model to determine the multimodal visual understanding model suitable for the business scenario;

[0016] Obtain the RPA component knowledge base and general knowledge base corresponding to the business scenario;

[0017] Based on the RPA component knowledge base, the general knowledge base, and the large model, a parsed single-step executable target operation sequence is obtained;

[0018] Inputting the business operation sequence and the current screen image screenshot into the multimodal visual understanding model to obtain the interactive elements in the current screen image screenshot;

[0019] The atomic operation sequence is determined according to the interactive element and the target operation sequence.

[0020] Optionally, determining the atomic operation sequence according to the interactive element and the target operation sequence includes:

[0021] Based on the keyword features in the business operation sequence, matching a target interactive element from the interactive element; the target interactive element includes semantic information of the interactive element and corresponding position coordinates;

[0022] Update the preset global variable table based on the target interactive element to determine the target global variable table;

[0023] Based on the target global variable table, the target operation sequence is filled in and the atomic operation sequence is determined.

[0024] Optionally, generating executable code for a target RPA component based on the atomic operation sequence and the large model includes:

[0025] Based on the prompt word completion algorithm of the large model, the atomic operation sequence is completed into expression information that can be understood by the target RPA component;

[0026] Generate executable code of the target RPA component based on the expression information and the large model.

[0027] Optionally, after generating the executable code of the target RPA component according to the atomic operation sequence and the large model, the method further includes:

[0028] Extracting environment-dependent parameters from the executable code;

[0029] Scan the system attributes of the target execution environment according to the environment dependency parameters and generate an environment matching evaluation report;

[0030] When it is detected in the environment matching evaluation report that the environment does not match, the large model is triggered to reconstruct the environment-dependent parameters in the executable code.

[0031] Optionally, after generating the executable code of the target RPA component according to the atomic operation sequence and the large model, the method further includes:

[0032] In the case of performing cross-platform operations, identifying characteristic parameters of the target execution environment;

[0033] Based on the characteristic parameters, converting the executable code into RPA code adapted to the target execution environment;

[0034] The RPA code is executed, and an environment change is detected during the execution process. When an environment change is detected, a generation strategy of the RPA code is automatically adjusted.

[0035] In a second aspect, an embodiment of the present application provides an RPA process generation device based on a large model, including:

[0036] A receiving module is used to receive business requirement information in natural language input by the user;

[0037] A first determination module is used to analyze the business demand information based on a preset large model and a business capability knowledge base, and determine a business operation sequence adapted to the business scenario;

[0038] A first processing module is configured to parse the business operation sequence into a single-step executable atomic operation sequence based on a robotic process automation (RPA) component knowledge base and a preset multimodal visual understanding model;

[0039] The second processing module is used to generate executable code of the target RPA component according to the atomic operation sequence and the large model.

[0040] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the steps of the method described in any one of the first aspects are implemented.

[0041] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in any one of the first aspects.

[0042] Compared with the existing technology, the RPA process generation method, device, medium and product based on the big model provided in the embodiments of the present application use the big model to parse natural language business requirements and match the business knowledge base to generate an operation sequence, combine the RPA component knowledge base and the multimodal visual model to decompose it into atomic operations, and then generate executable code from the big model. It lowers the technical threshold from the source through natural language interaction, uses modular atomic operations to improve expansion flexibility, and uses the multimodal visual model to realize dynamic environmental perception and execution effect supervision of the operation process, effectively solving the problems of poor flexibility, weak scalability and high technical threshold of traditional RPA, as well as the problem of lack of dynamic supervision when artificial intelligence and RPA are combined. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0044] Figure 1 This is one of the flowcharts of the RPA process generation method based on the large model provided in the embodiment of the present application;

[0045] Figure 2 This is the second flowchart of the RPA process generation method based on the large model provided in the embodiment of the present application;

[0046] Figure 3 A flowchart of the parsing service operation sequence provided in an embodiment of the present application;

[0047] Figure 4 A flowchart of the execution business process provided in the embodiment of the present application;

[0048] Figure 5 This is a structural diagram of the RPA process generation device based on a large model provided in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The terms "first", "second", etc. in this application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same type, and do not limit the number of objects, for example, the first object can be one or more. In addition, "or" in this application represents at least one of the connected objects. For example, "A or B" covers three options, namely, Option 1: including A but not including B; Option 2: including B but not including A; Option 3: including both A and B. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0050] The term "indication" in this application can be either a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, the operation to be performed, or the requested result, etc. in the instruction sent; an indirect indication can be understood as the receiver determining the corresponding information based on the instruction sent by the sender, or making a judgment and determining the operation to be performed or the requested result, etc. based on the judgment result.

[0051] As described in the background technology, there are two main defects in the existing technology: (1) Traditional RPA technology has poor flexibility and scalability, and has a certain technical threshold. Although the RPA designer used has achieved low-code or even code-free design, in the face of frequently changing business processes in the enterprise, whenever a new business is encountered, technical personnel need to update the RPA process, and manually drag and drop RPA components during the update process. On the one hand, this traditional RPA process generation method requires technical personnel to have considerable background knowledge and has a certain technical threshold; on the other hand, every time a new business is faced, the corresponding RPA process needs to be manually designed. This repetitive and tedious manual design greatly reduces the efficiency of business process generation and is difficult to adapt to business changes in the enterprise. (2) The processes generated by existing RPA products combined with artificial intelligence technology are mostly limited to generating business processes based on large language models. The model understanding ability is limited and does not consider the dynamic supervision of business process execution effects. One type of existing technology mostly uses static process generation methods, which cannot monitor the execution effect of each step in the process and cannot achieve adaptive dynamic response. That is, it cannot dynamically update the business status based on the execution results of the previous step of the process, so as to accurately call the RPA component in the next step; the other type is to simply call a large language model to understand user business needs, generate RPA processes only based on NLP technology, do not utilize existing multimodal data information, and have high requirements on the capabilities of the large model itself.

[0052] To solve at least one of the above problems, the embodiments of the present application provide a large-model-based RPA process generation method, device, medium, and product, which can reduce or avoid the occurrence of the above situations and realize the intelligent generation of RPA processes with no technical threshold, flexibility, and efficiency.

[0053] Please refer to Figure 1 , an embodiment of the present application provides an RPA process generation method based on a large model, including:

[0054] Step 11: Receive business requirement information in natural language input by the user.

[0055] This step of the application is to obtain the user's business needs expressed in natural language. For example, the user may enter "check how many days of annual leave are left this year." After receiving such a business need information description in natural language input by the user, it is used as the original input for subsequent processing.

[0056] Step 12: Based on the preset big model and business capability knowledge base, the business demand information is parsed to determine the business operation sequence adapted to the business scenario.

[0057] This application analyzes user needs using a pre-defined large model and a business capability knowledge base. The pre-defined large model has powerful natural language understanding capabilities and can parse key information from user needs, such as business objectives, involved operations, and data flows. The business capability knowledge base, on the other hand, stores standard operating procedures for various business scenarios. Together, these two transform users' natural language needs into a series of business operation sequences that match the business scenario.

[0058] For example, if the business requirement information entered by the user is "need to automatically download price reports from the supplier's website every day, then organize the data into an Excel spreadsheet, and finally send it to the sales team via email", step 12 can generate a business operation sequence of "log in to the website, navigate to the report page, filter the date range, download the report, and save the file".

[0059] Step 13: Based on the Robotic Process Automation (RPA) component knowledge base and the preset multimodal visual understanding model, the business operation sequence is parsed into a single-step executable atomic operation sequence.

[0060] Here, step 13 further breaks down the business operation sequence into atomic operations that can be performed by the RPA robot. The RPA component knowledge base contains various basic operation components, such as clicking a button, entering text, and reading table data. The multimodal visual understanding model plays a key role in this, analyzing interface elements and identifying controls such as buttons and input boxes. Using these two knowledge bases, business operations are converted into specific atomic operations. For example, the business operation "log in to a website" might be broken down into atomic operations such as "click the username input box, enter the account number, click the password input box, enter the password, and click the login button."

[0061] Step 14: Generate executable code for the target RPA component based on the atomic operation sequence and the large model.

[0062] In this embodiment of the present application, a large model is invoked to generate executable code for the target RPA component based on a sequence of atomic operations. The large model converts the atomic operations into corresponding code snippets, referencing the RPA platform's grammatical rules and component interface specifications. This code can be run directly on the RPA platform, thus fulfilling the user's original business requirements. When generating code, contextual information is also taken into account, and a global variable table is used to maintain parameter status, ensuring the consistency of the entire process execution.

[0063] This application proposes an intelligent generation method based on a large model, which realizes the intelligent generation of RPA processes with no technical threshold, flexibility and efficiency. In simple terms, the solution receives business requirements input by users, dynamically monitors the execution process of business processes, recommends RPA components based on the current execution effect and user intentions, analyzes screen images based on multimodal visual understanding models, fills in component parameters, and at the same time, uses the global variable table to maintain context information. In the process of multiple rounds of RPA process generation, it provides the ability to jointly reason about contextual semantic information, ensures the uniformity of parameter variables in the execution of the RPA process, and finally generates a complete executable RPA process. This application cooperates the business capability knowledge base and the RPA component knowledge base with each other, so that business logic and technical implementation can be effectively connected, realizing the collaboration of multiple knowledge bases; using visual understanding models to process interface elements, so that RPA can adapt to complex UI scenarios; using global variable tables to maintain context information in process execution, supporting multiple rounds of interaction and parameter passing; users only need to describe their needs in natural language to automatically generate executable code, which greatly reduces the technical threshold for using RPA.

[0064] In a specific implementation, referring to Figure 2 In the overall process diagram shown, the present application provides a corresponding device for the intelligent generation method based on the large model, which includes a business demand decomposition module, an operation sequence parsing module, and a business process execution module. These three modules are used to implement all the processes from step 11 to step 14 above. Specifically, the solution of the present application is based on the natural language understanding capability of the general large model, combined with a proprietary knowledge base of business capabilities, multimodal visual understanding technology, and an RPA component library specific to business scenarios, to achieve the understanding, decomposition, and execution of business requirements, and obtain flexible and efficient RPA process intelligent generation to meet the user's business needs.

[0065] First, the business requirements in the form of natural language input by the user are received, and the scope of understanding of the general large model is expanded in combination with the proprietary knowledge base of business capabilities unique to this application, the business requirements are parsed, and a business operation sequence adapted to the business scenario is obtained, where each operation sequence may contain one or more RPA atomic operations to complete the decomposition of business requirements. In the operation sequence parsing module, the multimodal visual understanding model is fine-tuned using model fine-tuning technology to parse the screen image of the current interface, understand the user's intention in combination with the business operation sequence, recommend RPA components that are adapted to the current business process, and fill in the component-related parameters to obtain a specific RPA atomic operation sequence. Finally, the RPA atomic operation sequence is input into the business process execution module, and each operation sequence is executed one by one to analyze the current business process status, update the global variable table in a timely manner, dynamically monitor the RPA process execution process, and analyze while executing, thereby improving the efficiency of business process execution.

[0066] Optionally, the above step 12 includes:

[0067] Based on the preset big model and business capability knowledge base, the business requirement information in natural language form is converted into structured information;

[0068] The structured information is subjected to semantic analysis, context understanding, and task decomposition to determine a business operation sequence adapted to the business scenario corresponding to the business requirement information; each operation sequence in the business operation sequence includes one or more RPA atomic operations.

[0069] In an embodiment of the present application, the process of converting user-entered natural language business requirement information into an executable business operation sequence involves two key stages: structured conversion and semantic analysis and task decomposition. This stage of converting natural language to structured information primarily utilizes a large model and a business capability knowledge base to convert unstructured natural language requirements into structured data. This can be fine-tuned based on a general large model combined with a proprietary business capability knowledge base to form a domain large model tailored to specific business scenarios. The large model is used to map natural language requirements into a predefined structured format (such as JSON). The business capability knowledge base provides a reference for entity recognition and relationship extraction at this stage, for example, identifying that "sales data" belongs to a business entity and that "backup" corresponds to a specific operation type. Semantic analysis and task decomposition conducts in-depth analysis of structured information to generate a specific business operation sequence. Semantic analysis can parse key semantic elements within structured information, including: business objectives (such as "data backup" and "report generation"); operation objects (such as "sales data" and "customer information"); and constraints (such as time limits and data format requirements). This application can utilize a global variable table and historical conversation information to parse contextual dependencies within requirements. For example, if a user mentions "update the last data to the system," the system needs to determine the specific "last data" through context. This application breaks down high-level business goals into specific business operation sequences, with each operation corresponding to one or more RPA atomic operations.

[0070] Optionally, this application can set a business process difficulty determination. After generating the operation sequence, the system will evaluate the task complexity based on the following factors: Knowledge base matching: whether the requirements completely match the predefined business template. Operation sequence length: a task is considered complex if the number of steps exceeds a threshold (such as 5 steps). Parameter complexity: whether it contains dynamic parameters, conditional judgments, etc. Based on the evaluation results, tasks will be classified as simple and complex. Simple tasks mean directly calling predefined shortcuts (such as pre-made RPA process templates) to reduce dependence on large models and improve execution efficiency. Complex tasks mean entering subsequent steps (such as atomic operation parsing and code generation), which may require multiple calls to large models for refined processing.

[0071] Specifically, this application is based on the powerful natural language understanding ability of the big model, combined with the proprietary knowledge base of business capabilities, to form a big model suitable for the current business scenario, and convert business requirements in natural language form into structured information, such as using JSON format to constrain it. Through technologies such as semantic analysis, context understanding and task decomposition, a business operation sequence is generated. At the same time, the difficulty of the business operation process is judged based on the knowledge base and the number of steps in the business operation sequence. If it is a simple task, the subsequent inquiries to the big model can be reduced directly through pre-set shortcuts, thereby improving business execution efficiency; if it is a complex task, the operation sequence parsing and business process execution module is entered. Based on the general big model, combined with the externally introduced proprietary knowledge base of business capabilities and pre-set business operation shortcuts, it is possible to flexibly disassemble the business process input by the user and improve business execution efficiency.

[0072] For example, let's take the example of querying the number of days remaining in annual leave. A user describes the business need in natural language as "check how many days of annual leave are left this year." This application converts this business need into a business operation sequence: Based on a proprietary knowledge base and general large model customized for the business scenario, we understand the user need and convert it into the following business operation sequence, completing the decomposition of the business need for querying the number of days remaining in annual leave. Specifically, the user enters "check how many days of annual leave are left this year." The following operations are required in the professional knowledge base: Querying annual leave in the human resources system requires multiple steps, including "opening the browser, clicking on the human resources system operation, clicking on the leave operation, and ending." The output is multiple RPA atomic operations: "open the browser, enter the human resources system, click on annual leave, query annual leave, and enter the number of days remaining in annual leave."

[0073] Optionally, the above step 13 includes:

[0074] Using the collected data information of the business scenario, fine-tuning the initial multimodal visual understanding model to determine the multimodal visual understanding model suitable for the business scenario;

[0075] Obtain the RPA component knowledge base and general knowledge base corresponding to the business scenario;

[0076] Based on the RPA component knowledge base, the general knowledge base, and the large model, a parsed single-step executable target operation sequence is obtained;

[0077] Inputting the business operation sequence and the current screen image screenshot into the multimodal visual understanding model to obtain the interactive elements in the current screen image screenshot;

[0078] The atomic operation sequence is determined according to the interactive element and the target operation sequence.

[0079] In an embodiment of the present application, this step is to convert the business operation sequence into an atomic operation sequence that can be executed by the RPA robot. This process is mainly achieved through the collaborative work of the multimodal visual understanding model and the knowledge base. This application collects typical interface screenshots in business scenarios (such as a system login page, report page, etc.). Label key elements (such as buttons, input boxes, tables, etc.) and their attributes (position, text content, function). Based on the initial multimodal visual understanding model (such as CLIP, LayoutLM, etc.), fine-tune using business scenario data to optimize the model's recognition ability for business-specific elements, such as invoice numbers and amount fields in financial systems. This application uses business scenario data to fine-tune the model, and the generated multimodal visual understanding model can accurately identify elements in specific business interfaces.

[0080] Optionally, the RPA component knowledge base of this application stores basic RPA operation component information, such as basic actions such as mouse clicks, keyboard input, text reading, and file operations; component parameter specifications, such as click coordinates, input content, and wait time. The general knowledge base contains domain-independent common sense information, such as the default behavior of interface elements, such as the "Login" button that usually triggers the authentication process; text format rules, such as date format, email format, etc. Leverage large models to understand the semantics of business operation sequences, for example, converting "select last month's date" into a specific date selection operation; and inferring the type and possible location of the button that needs to be clicked to "download the report."

[0081] Based on the knowledge base and big model, business operation sequences are transformed into more specific intermediate operation sequences, breaking down business operations into more detailed sub-operations. The big model and knowledge base are then used to infer the required parameters for these operations. For example, the "last month's date" can be extracted from business requirements to calculate the specific date range. Furthermore, the element positioning method can be determined based on the characteristics of the target system.

[0082] The multimodal visual understanding model is used to analyze the current screenshot and identify interactive elements. Specifically, the following steps are performed: obtain a screenshot of the current business process execution interface and pre-process the image (such as scaling and normalization) to facilitate model processing; the multimodal visual understanding model outputs element information in the interface, including element type, such as buttons, input boxes, drop-down menus, etc.; location coordinates; text content, such as text on buttons, placeholders in input boxes, etc.; status, such as whether the button is clickable, whether the input box is filled in, etc. Further, the knowledge base and the large model are combined to infer the function of the elements. For example: a button with the text "Export Excel" is inferred to have the function of exporting data; an input box with a calendar icon is inferred to be a date selector. Finally, the interface element information and the target operation sequence are combined to generate the final atomic operation sequence.

[0083] In this application, element and operation matching is to map the abstract operations in the target operation sequence, such as "click the login button", to specific interface elements; atomic operation generation generates a specific RPA atomic operation for each matching operation, including precise parameters; context parameter passing passes the parameters in the business operation, such as user name and date, to the atomic operation.

[0084] Furthermore, determining the atomic operation sequence according to the interactive element and the target operation sequence includes:

[0085] Based on the keyword features in the business operation sequence, matching a target interactive element from the interactive element; the target interactive element includes semantic information of the interactive element and corresponding position coordinates;

[0086] Update the preset global variable table based on the target interactive element to determine the target global variable table;

[0087] Based on the target global variable table, the target operation sequence is filled in and the atomic operation sequence is determined.

[0088] In an embodiment of the present application, keyword features, such as button IDs, text labels, etc., are extracted from the business operation sequence as a basis for matching interactive elements; the matched target interactive elements must contain semantic information (such as function descriptions) and position coordinates, such as screen coordinates or DOM node positions, for subsequent operation positioning. Here, the preset global variable table usually stores operation context information, such as the current page status, entered data, etc. According to the semantic information of the target interactive element, such as changes in the value of the input box, the variable table is updated to generate a target global variable table. For example, if it is detected that the "Submit button" is clicked, the variable table is updated. Based on the target global variable table, the abstract steps in the target operation sequence, such as "filling in the form", are broken down into specific atomic operations, such as "entering "test" in the input box at position ID X." The atomic operation sequence can directly drive the execution of the automated testing tool or RPA system.

[0089] The above steps in this application transform abstract business operations into concrete, executable sequences of atomic operations through the collaborative work of a multimodal visual understanding model and a knowledge base. The advantages of this approach include its ability to handle interface differences across different systems, improving the versatility of RPA; supporting interaction with complex, dynamically changing interfaces; automatically inferring operational parameters, reducing manual configuration; and providing a foundation for exception handling during subsequent process execution through element recognition and waiting mechanisms.

[0090] In a specific implementation method, this application combines the RPA component knowledge base and the fine-tuned multimodal visual understanding model to further parse the business operation sequence into a single-step executable atomic operation sequence, which is convenient for direct calling by the business process execution module. The RPA component library is used as external expert knowledge to assist the large model in parsing the business operation sequence. At the same time, by manually collecting business scenario data and using fine-tuning technology to fine-tune the multimodal visual understanding model, a multimodal visual understanding model suitable for the current business scenario is obtained. It is used to analyze the user interface elements and layout in the current screen image, determine the specific location and function of key components, fill the relevant parameters into the RPA component, complete the intelligent parsing and understanding of the task sequence, make intelligent decisions, form a structured information representation, and output the atomic operation sequence.

[0091] Reference Figure 3 As shown, in this specific implementation, take clicking "Vacation" in querying the remaining days of annual leave as an example:

[0092] (1) Input the business operation sequence: Click on "vacation"; (2) Analyze the business operation sequence: Combine the RPA component knowledge base and general knowledge base specific to the business scenario, and use the big model to obtain recommended RPA components and corresponding operations; (3) Analyze the screen image: Call the multimodal visual understanding model, input the current screen image screenshot and the business operation sequence into the multimodal visual understanding model, mark the semantic information and specific location coordinates of each interactive element in the image, match the corresponding interactive element based on the keyword "vacation", update the global variable table, and assign the extracted valid information to the corresponding parameters in the global variable table; (4) Fill in the RPA component parameters: Query the global variable table and complete the required variables in the current atomic operation sequence; (5) Output the atomic operation sequence.

[0093] Optionally, the above step 14 includes:

[0094] Based on the prompt word completion algorithm of the large model, the atomic operation sequence is completed into expression information that can be understood by the target RPA component;

[0095] Generate executable code of the target RPA component based on the expression information and the large model.

[0096] In an embodiment of the present application, the semantics of the atomic operation sequence is parsed by a large model, and converted into a structured expression containing contextual information. Combined with interface analysis technology, the attribute information of the operation target is supplemented. According to the grammatical rules of the target RPA platform, the expression is converted into a specific script, and the large model is used to achieve end-to-end code generation, including: input is a structured expression and RPA platform type, and the output is a complete and deployable automation script. This process achieves minute-level conversion from requirement description to executable code. That is, this application receives an atomic operation sequence, and based on the large model prompt word completion project, the atomic operation sequence description is completed into an expression that can be understood by the RPA component, and then the large model drives the generation of executable code, and finally executes the RPA code and optimizes the subsequent steps based on process feedback.

[0097] For example, refer to Figure 4 As shown in the figure, for example, clicking "Vacation" to query the remaining days of annual leave involves entering an atomic operation sequence: clicking "Vacation." Large-scale model prompt word completion involves interacting with the prompt word engineering agent according to a predefined template to generate a completed operation sequence. Large-scale model code generation involves interacting with the code generation agent based on the completed prompt word operation sequence to generate executable RPA code. Based on the generated RPA code, the system environment is analyzed and automatically configured and initialized according to the generated code requirements. Furthermore, to ensure stable operation of the RPA code, the system automatically adjusts the code execution logic based on the platform, ensuring consistent operation across platforms.

[0098] Optionally, after step 14, the method further includes:

[0099] Extracting environment-dependent parameters from the executable code;

[0100] Scan the system attributes of the target execution environment according to the environment dependency parameters and generate an environment matching evaluation report;

[0101] When it is detected in the environment matching evaluation report that the environment does not match, the large model is triggered to reconstruct the environment-dependent parameters in the executable code.

[0102] In an embodiment of the present application, three types of environment-dependent parameters are parsed from the generated executable code, including: basic environment, such as Python / Java version, operating system type; third-party dependent environment, such as library information and version number; hardware requirement information, such as GPU memory, CPU core number and other configuration parameters. The environment-dependent parameters are then converted into JSON format for structured storage. Based on the environment-dependent parameters, the system properties of the target execution environment are scanned to generate an environment matching evaluation report, and the matching score of the environment matching evaluation report is calculated using a difference algorithm to determine the matching degree of the environment in the environment matching evaluation report. For example, the parameter types in the environment matching evaluation report include major version number, minor version number and hardware configuration. The weights of the major version number, minor version number and hardware configuration are set to 0.4, 0.4 and 0.3 respectively. Based on the total score of the environment matching evaluation report of different parameter types, if the total score is less than the threshold, it is determined that the environment in the environment matching evaluation report does not match, and the large model is triggered to reconstruct the environment-dependent parameters in the executable code.

[0103] Here, large model refactoring strategies include, but are not limited to, version downgrades, alternative solutions, and conditional branching, such as inserting runtime environment detection code and dynamically loading different implementation classes. After refactoring, smoke testing is automatically performed to ensure functional equivalence.

[0104] Optionally, after step 14, the method further includes:

[0105] In the case of performing cross-platform operations, identifying characteristic parameters of the target execution environment;

[0106] Based on the characteristic parameters, converting the executable code into RPA code adapted to the target execution environment;

[0107] The RPA code is executed, and an environment change is detected during the execution process. When an environment change is detected, a generation strategy of the RPA code is automatically adjusted.

[0108] In an embodiment of the present application, when performing cross-platform operations, the characteristic parameters of the target execution environment are identified, including but not limited to: collecting operating system information; collecting software ecosystem information; collecting hardware configuration information, etc. Based on the characteristic parameters, a unified environment description model is established to abstract the commonalities and differences of different platforms, such as file path separators and command line tool names. The characteristic parameters are key parameters that may affect the execution of RPA. The present application sets up a predefined conversion rule library in advance, and defines code conversion logic for different platform characteristics, including: path processing: automatic replacement of file path separators; command line adaptation; UI automation adjustment, etc. The present application creates special code templates for different platforms, selects appropriate templates based on characteristic parameters; for network requests, adjusts the request header configuration based on the protocol support of the target environment. Injecting environmental detection logic into the code enables the RPA process to dynamically select the execution path at runtime.

[0109] In an embodiment of the present application, the RPA code is executed, and environmental changes are detected during the execution process. When an environmental change is detected, the generation strategy of the RPA code is automatically adjusted, specifically including: regularly collecting key environmental parameters such as network connection status and software process status during the execution of the RPA process; using system hooks or event monitoring mechanisms to capture environmental changes that may affect execution, such as screen resolution adjustment, software crashes, etc. The present application establishes an environmental baseline, compares the parameters collected in real time with it, and identifies abnormal changes: network interruption, unexpected software shutdown, etc. Abnormal performance indicators, such as response time exceeding a threshold, etc. When an environmental change is detected, the relevant part of the RPA code is dynamically modified without restarting the entire process.

[0110] Optionally, this application takes different response measures based on the severity of the change: Minor changes: Log the change and continue execution. Major changes: Pause the process, adjust parameters, and try again. Serious failures: Trigger a rollback mechanism, save the scene, and notify the administrator. Here, this application can save the intermediate state of the process execution so that it can be rolled back to the most recent stable state when the adjustment fails, such as using a transaction mechanism to ensure data consistency; record checkpoints of key operations for easy recovery.

[0111] This application can also adjust the execution strategy according to the type of environmental change. For example, if it is determined to be a network problem, the number of retries can be increased, the timeout period can be extended, etc.

[0112] In summary, the method of this application customizes and designs a dedicated capability knowledge base and RPA component knowledge base for business scenarios. By combining the reasoning capabilities of large models, the method retrieves RPA components suitable for current business needs from the knowledge base. In the subsequent operation sequence parsing module, the currently required RPA component knowledge base is dynamically modified to reduce system load. The multimodal visual understanding model is then fine-tuned to achieve business understanding by simply inputting operation sequence information and screen images, assisting in the intelligent generation of RPA process code. Compared to RPA process generation methods that rely on information such as HTML, this method enhances the cross-platform adaptability of RPA products.

[0113] This application only requires users to provide requirements in natural language, and the system automatically conducts multiple rounds of dialogue with the large model to automatically analyze the business requirements and business operation sequences proposed by the user, realizing flexible and efficient RPA process intelligent generation to meet the user's business needs. This application has the following technical advantages:

[0114] 1. Low user understanding cost. This application only requires users to input business process requirements in natural language. There is no technical threshold. Even non-professionals can use this system to generate the required business processes.

[0115] 2. Realize the understanding of complex operation sequences based on large model drive. Existing technologies have difficulty in effectively parsing key RPA operations in business processes and updating key parameters in a timely and synchronous manner when crossing applications or user graphical interfaces. This application is based on a large model and a multimodal visual understanding model, and maintains a global variable table at the same time. It effectively guarantees the synchronous update of key parameters in business processes in multiple rounds of dialogue, and synchronously assigns them to RPA components. When facing operations across applications or user graphical interfaces, it can also achieve seamless updates of user variables.

[0116] 3. Cross-platform RPA code generation and execution. This application uses a large model-driven RPA code generation method that is no longer limited to RPA process generation on a single platform. It can adaptively adjust RPA code generation requirements and execution logic based on the characteristics of the execution environment and dynamically adjust subsequent RPA code based on a feedback mechanism. At the same time, this application uses a technical solution of analysis-while-execution to achieve dynamic supervision of RPA code execution and improve the efficiency of business process execution.

[0117] The above describes various methods of the embodiments of the present application. The following further provides apparatuses for implementing the above methods.

[0118] Please refer to Figure 5 , an embodiment of the present application provides an RPA process generation device based on a large model, including:

[0119] Receiving module 51, for receiving business demand information in natural language form input by a user;

[0120] A first determination module 52 is configured to analyze the business requirement information based on a preset large model and a business capability knowledge base, and determine a business operation sequence that is suitable for the business scenario;

[0121] A first processing module 53 is configured to parse the business operation sequence into a single-step executable atomic operation sequence based on a robotic process automation (RPA) component knowledge base and a preset multimodal visual understanding model;

[0122] The second processing module 54 is configured to generate executable code of a target RPA component according to the atomic operation sequence and the large model.

[0123] Optionally, the first determining module 52 includes:

[0124] A first processing unit is configured to convert the business requirement information in natural language into structured information based on a preset large model and a business capability knowledge base;

[0125] The second processing unit is used to perform semantic analysis, context understanding and task decomposition on the structured information to determine a business operation sequence that is adapted to the business scenario corresponding to the business requirement information; each operation sequence in the business operation sequence includes one or more RPA atomic operations.

[0126] Optionally, the first processing module 53 includes:

[0127] A first determining unit is configured to fine-tune an initial multimodal visual understanding model using the collected data information of the business scenario, and determine the multimodal visual understanding model suitable for the business scenario;

[0128] A first acquisition unit is configured to acquire an RPA component knowledge base and a general knowledge base corresponding to the business scenario;

[0129] A second acquisition unit is configured to acquire a parsed single-step executable target operation sequence based on the RPA component knowledge base, the general knowledge base, and the large model;

[0130] a third acquisition unit, configured to input the business operation sequence and the current screen image screenshot into the multimodal visual understanding model, and acquire the interactive elements in the current screen image screenshot;

[0131] The second determining unit is configured to determine the atomic operation sequence according to the interactive element and the target operation sequence.

[0132] Optionally, the second determining unit is specifically configured to:

[0133] Based on the keyword features in the business operation sequence, matching a target interactive element from the interactive element; the target interactive element includes semantic information of the interactive element and corresponding position coordinates;

[0134] Update the preset global variable table based on the target interactive element to determine the target global variable table;

[0135] Based on the target global variable table, the target operation sequence is filled in and the atomic operation sequence is determined.

[0136] Optionally, the second processing module 54 includes:

[0137] A third processing unit is configured to complete the atomic operation sequence into expression information understandable to the target RPA component based on a prompt word completion algorithm of the large model;

[0138] The fourth processing unit is configured to generate executable code of a target RPA component based on the expression form information and the large model.

[0139] Optionally, the device of the present application further includes:

[0140] An extraction module, configured to extract environment-dependent parameters from the executable code;

[0141] A generation module, configured to scan the system attributes of the target execution environment according to the environment dependency parameters and generate an environment matching evaluation report;

[0142] The trigger module is used to trigger the large model to reconstruct the environment-dependent parameters in the executable code when detecting that the environment does not match in the environment matching evaluation report.

[0143] Optionally, the device of the present application further includes:

[0144] An identification module, used to identify characteristic parameters of a target execution environment when performing cross-platform operations;

[0145] a third processing module, configured to convert the executable code into RPA code adapted to a target execution environment based on the characteristic parameters;

[0146] The fourth processing module is used to execute the RPA code and detect environmental changes during the execution process, and automatically adjust the generation strategy of the RPA code when an environmental change is detected.

[0147] It should be noted that the device in this embodiment is a device corresponding to the above-mentioned method, and the implementation methods in the above-mentioned embodiments are all applicable to the embodiments of this device and can achieve the same technical effects. The above-mentioned device provided in the embodiment of this application can implement all the method steps implemented in the above-mentioned method embodiment and can achieve the same technical effects. The parts and beneficial effects of this embodiment that are the same as those in the method embodiment will not be specifically described here.

[0148] The present application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the aforementioned large-scale model-based RPA process generation method embodiment, achieving the same technical effects. To avoid repetition, the details are omitted here. The computer-readable storage medium may be a read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0149] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned large-model-based RPA process generation method embodiment are implemented, and the same technical effects can be achieved. To avoid repetition, they are not described here.

[0150] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network equipment, etc.) to execute the methods described in each embodiment of the present application.

[0152] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A RPA process generation method based on a large model, characterized in that: include: Receive business requirement information in natural language input from users; Based on the preset big model and business capability knowledge base, the business demand information is analyzed to determine the business operation sequence adapted to the business scenario; Based on the RPA component knowledge base and the preset multimodal visual understanding model, the business operation sequence is parsed into a single-step executable atomic operation sequence; Generate executable code of the target RPA component according to the atomic operation sequence and the large model.

2. The method according to claim 1, characterized in that Based on the preset big model and business capability knowledge base, the business requirement information is analyzed to determine the business operation sequence adapted to the business scenario, including: Based on the preset big model and business capability knowledge base, the business requirement information in natural language form is converted into structured information; The structured information is subjected to semantic analysis, context understanding, and task decomposition to determine a business operation sequence adapted to the business scenario corresponding to the business requirement information; each operation sequence in the business operation sequence includes one or more RPA atomic operations.

3. The method according to claim 1, characterized in that Based on the RPA component knowledge base and the preset multimodal visual understanding model, the business operation sequence is parsed into a single-step executable atomic operation sequence, including: Using the collected data information of the business scenario, fine-tuning the initial multimodal visual understanding model to determine the multimodal visual understanding model suitable for the business scenario; Obtain the RPA component knowledge base and general knowledge base corresponding to the business scenario; Based on the RPA component knowledge base, the general knowledge base, and the large model, a parsed single-step executable target operation sequence is obtained; Inputting the business operation sequence and the current screen image screenshot into the multimodal visual understanding model to obtain the interactive elements in the current screen image screenshot; The atomic operation sequence is determined according to the interactive element and the target operation sequence.

4. The method according to claim 3, characterized in that Determining the atomic operation sequence according to the interactive element and the target operation sequence includes: Based on the keyword features in the business operation sequence, matching a target interactive element from the interactive element; the target interactive element includes semantic information of the interactive element and corresponding position coordinates; Update the preset global variable table based on the target interactive element to determine the target global variable table; Based on the target global variable table, the target operation sequence is filled in and the atomic operation sequence is determined.

5. The method according to claim 1, wherein Generating executable code of a target RPA component according to the atomic operation sequence and the large model, including: Based on the prompt word completion algorithm of the large model, the atomic operation sequence is completed into expression information that can be understood by the target RPA component; Generate executable code of the target RPA component based on the expression information and the large model.

6. The method according to claim 1, characterized in that After generating the executable code of the target RPA component according to the atomic operation sequence and the large model, the method further includes: Extracting environment-dependent parameters from the executable code; Scan the system attributes of the target execution environment according to the environment dependency parameters and generate an environment matching evaluation report; When it is detected in the environment matching evaluation report that the environment does not match, the large model is triggered to reconstruct the environment-dependent parameters in the executable code.

7. The method according to claim 1, characterized in that After generating the executable code of the target RPA component according to the atomic operation sequence and the large model, the method further includes: In the case of performing cross-platform operations, identifying characteristic parameters of the target execution environment; Based on the characteristic parameters, converting the executable code into RPA code adapted to the target execution environment; The RPA code is executed, and an environment change is detected during the execution process. When an environment change is detected, a generation strategy of the RPA code is automatically adjusted.

8. An RPA process generation device based on a large model, characterized in that: include: A receiving module is used to receive business requirement information in natural language input by the user; A first determination module is used to analyze the business demand information based on a preset large model and a business capability knowledge base, and determine a business operation sequence adapted to the business scenario; A first processing module is configured to parse the business operation sequence into a single-step executable atomic operation sequence based on a robotic process automation (RPA) component knowledge base and a preset multimodal visual understanding model; The second processing module is used to generate executable code of the target RPA component according to the atomic operation sequence and the large model.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 7.

Images