RPA system based on natural language interaction and implementation method thereof

Abstract

The invention relates to an RPA system based on natural language interaction and an implementation method thereof. The system comprises a user input module, an intention recognition module, a process planning module, an execution module, an inspection module and an output module. A user inputs a natural language instruction through a textbox or a microphone, the intention recognition module recognizes the intention of the user through an artificial intelligence large language model, and the process planning module plans an operation target and operation steps according to the intention of the user and a screen interface. The execution module controls the mouse and the keyboard to execute planned operation on the operation target through the technology related to automatic testing, the checking module checks whether the execution result meets the intention of the user or not after the operation is completed, and finally the output module feeds back the execution result to the user. According to the method, a common user can easily create a complex automatic process through a natural language, and the use threshold of an RPA system is greatly reduced.

Description

Technical Field

[0001] The present invention relates to a Robotic Process Automation (RPA) system and an implementation method thereof, and in particular to an RPA system based on natural language interaction and an implementation method thereof, in which a user communicates with a robot through natural language instructions, and the robot recognizes the user's intention and performs corresponding mouse and keyboard operations, and an automation system and an implementation method thereof. Background Art

[0002] Traditional RPA systems mainly rely on users to write processes or codes by themselves to automate tasks, which requires users to have professional programming knowledge. For professionals, this process is time-consuming and laborious; for ordinary users, especially business personnel in enterprises, who lack programming background, it is difficult to use RPA systems. Therefore, the popularity of traditional RPA systems among ordinary users is low, and their potential to improve office efficiency cannot be fully utilized, resulting in a large amount of simple repetitive work still needing to be completed manually, which consumes manpower and time costs. In order to solve this problem, the present invention proposes a new RPA system based on natural language interaction and its implementation method, which allows users to interact with robots through simple natural language, thereby simplifying the process of process creation and lowering the threshold for using the RPA system. Summary of the invention

[0003] The present invention provides an RPA system based on natural language interaction and its implementation method, aiming to simplify the creation process of automated tasks, lower the usage threshold, and improve work efficiency. The system can be widely used in various application scenarios, such as automated testing, data capture, group messaging, intelligent reply, real-time monitoring, etc.

[0004] The system mainly includes the following modules: User input module: Users input natural language commands through ordinary text boxes or microphones.

[0005] Intent recognition module: Uses the artificial intelligence large language model (LLM) to identify user intent.

[0006] Process planning module: Combines the screen interface and user intent to plan the target operation object and operation method for the next operation.

[0007] Execution module: Control the mouse and keyboard to perform related operations on the target operation object.

[0008] Check module: Checks whether the execution result meets the user's intention. If not, continues process planning and execution until the user's intention is met or the number of executions exceeds the threshold set by the system.

[0009] Output module: Feedback the execution results to the user.

[0010] The innovation of the present invention is: through natural language interaction ( Figure 1 ), ordinary users can also easily create complex automated processes, greatly reducing the threshold for using the RPA system. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 : System UI interaction diagram.

[0012] Figure 2 :System modules and workflow diagram.

[0013] Figure 3 : The working principle diagram of the process planning module. DETAILED DESCRIPTION

[0014] The RPA system based on natural language interaction and its implementation method of the present invention realize the execution of automated tasks (such as Figure 2 ).

[0015] The following are specific implementations of each module: User input module: The user inputs natural language instructions through a normal text box or microphone. For example, the user can enter "Baidu search [Kidi], open the first non-advertising page, and check whether the opened website is kidi.bot" in the text box, or speak the same instruction through the microphone.

[0016] The system automatically recognizes the input method and pre-processes the input natural language instructions, such as converting speech into text.

[0017] Intent recognition module: Uses the artificial intelligence large language model (LLM) to identify the user's intent. For example, in the above example, the user's intent is identified as "Search [Kidi] on Baidu and check whether the first non-advertising page is kidi's official homepage kidi.bot".

[0018] The intent recognition module passes the recognition results to the process planning module, including the user's intent type and specific parameters (such as website address).

[0019] Process planning module: Combines the screen interface and user intent to plan the target operation object and operation method for the next operation (such as Figure 3). For example, if the user's intention is to "search [Kidi] on Baidu and check whether the first non-advertising page is kidi's official homepage kidi.bot", the process planning module will identify the browser icon on the current screen as the target operation object, and the first step of the planning operation is to "open the browser and visit Baidu's official website "https: / / www.baidu.com".

[0020] The process planning module generates detailed operation instructions, including the location of the target operation object, the operation type (such as mouse click, keyboard input, etc.) and the operation parameters (such as the input URL).

[0021] Execution module: Controls the mouse and keyboard to perform related operations on the target operation object. For example, according to the operation instructions generated by the process planning module, the execution module controls the mouse to move to the position of the browser icon and click it, and then controls the keyboard to enter the URL "www.baidu.com" and press the Enter key.

[0022] The execution module records all steps and results of the operation process for subsequent repeated execution and verification by the inspection module.

[0023] Check module: Checks whether the execution result meets the user's intention. For example, check whether the browser has achieved the user's final intention, that is, clicking on the first non-advertising search result. If the check result does not meet the user's intention, it means that the operation has not been completed or the operation is wrong. The check module will take the new screenshot as a new input, return to the process planning module and the execution module, and continue the process planning and execution operation until the user's final intention is met or the number of executions exceeds the threshold set by the system.

[0024] The inspection module uses a large language model to determine whether the expected intent has been achieved, for example, checking whether the domain name of the search result page opened is kidi.bot.

[0025] Output module: Feedback the execution results to the user. For example, the output module can display a message "The page opened is (or is not) kidi.bot".

[0026] In addition to outputting normal text information, the output results can also be saved in files according to user instructions.

[0027] The output module can also provide log files to record all steps and results of the entire operation process for users to audit and trace. Example

[0028] Example 1: Automated testing Background: A software company needs to perform automated testing on a newly developed web application to ensure that the functions are functioning properly.

[0029] Specific implementation steps: User input: The user enters a natural language instruction in the text box: "Open the browser and visit the test web page, enter the username AAA and password BBB, and check whether the login function is normal." The system recognizes the input method as text input and performs preprocessing.

[0030] Intent recognition: The intent recognition module uses a pre-trained large language model (LLM) to recognize the user's intention, and the recognition result is "open the browser and visit a specific web page to check the login function."

[0031] The intent recognition module passes the recognition results to the process planning module, including the intent type and specific parameters (such as the URL of the test web page, user name, password, etc.).

[0032] Process planning: The process planning module combines the screen interface and user intention to plan the first step: open the browser.

[0033] Generate detailed operation instructions, including the location of the target operation object, operation type and operation parameters.

[0034] Execute operation: The execution module controls the mouse and keyboard to perform the planned operation: control the mouse to move to the browser icon and click it.

[0035] The execution module records all steps and results during the operation.

[0036] Check result: The inspection module checks whether the execution result meets the user's final intention: check whether the login can be normal.

[0037] If the inspection result does not meet the user's final intention, the inspection module takes the new screenshot as input and returns to the process planning module and the execution module (the next step is to open the URL and enter the username and password), and continues to operate until the user's final intention is met or the number of executions exceeds the threshold set by the system.

[0038] Output results: The output module will feedback the execution results to the user, displaying a message: "The task was completed successfully and the login function is normal." The output module can also provide a log file to record all steps and results of the entire operation process for users to audit and trace.

[0039] Example 2: Batch reply to private messages Background: In social media platforms or corporate internal communication systems, users often need to reply to a large number of private messages. Manually replying to these private messages is not only time-consuming but also prone to errors. This example shows how to use a natural language-based RPA system to automatically reply to private messages, thereby improving work efficiency and user satisfaction.

[0040] Specific implementation steps: User input: Users enter natural language instructions in a normal text box: "Reply to all unread private messages. The reply content is automatically determined by the large language model based on the content of the other party's private message.

[0041] Intent recognition: The intent recognition module uses a large language model to recognize the user's intention, and the recognition result is "batch reply to unread private messages";

[0042] The intent recognition module passes the recognition results to the process planning module, including the intent type and specific parameters (such as reply content).

[0043] Process planning: The process planning module combines the screen interface and user intention to plan each operation step: open the private message list.

[0044] Generate detailed operation instructions, including the location of the target operation object, operation type and operation parameters.

[0045] Execute operation: The execution module controls the mouse and keyboard to perform the planned operation: control the mouse to move to the private message list icon and click it.

[0046] The execution module records all steps and results during the operation.

[0047] Check results: The inspection module checks whether the execution results meet the user's intention: Check whether all unanswered private messages have been sent successfully.

[0048] If the inspection result does not meet the user's intention, the inspection module takes the new screenshot as input and returns to the process planning module and the execution module to continue the operation until the user's intention is met or the number of executions exceeds the threshold set by the system.

[0049] Output results: The output module will feedback the execution results to the user and display a message: "The task was successfully completed and all unread private messages have been replied." This information can also be saved in file format according to user requirements.

[0050] The output module can also provide log files to record all steps and results of the entire operation process for users to audit and trace.

[0051] Summary: Through the above specific implementation methods and examples, the natural language-based RPA system and its implementation method of the present invention can effectively realize automated tasks, improve work efficiency, and reduce labor costs. The collaborative work of each module ensures the accuracy and reliability of the task and meets the needs of various application scenarios. Advantages (innovative points):

[0052] Through natural language interaction, ordinary users can easily create complex automated processes without programming experience and thinking, which greatly reduces the threshold for using RPA systems.

Claims

1. An RPA system based on natural language interaction, characterized in that: include: A user input module, used to receive natural language instructions input by the user through a text box or a microphone; Intent recognition module, which uses the artificial intelligence large language model (LLM) to identify the user's intent; The process planning module uses the artificial intelligence large language model (LLM) to plan the next operation steps, target operation objects and operation methods based on the identified user intentions and the current screen interface, and generates operation instructions; the execution module controls the mouse and keyboard to execute the planned operations through technologies related to existing automated user interface (UI) operation technologies; Check module to check whether the execution result meets the user's intention; The output module outputs the execution results and displays them to the user.

2. The RPA system based on natural language interaction according to claim 1, characterized in that: In addition to text box input, the user input module also includes voice input and automatic capture of the current screen interface as auxiliary input functions.

3. The natural language based RPA system according to claim 1, characterized in that: The intent recognition module uses a pre-trained artificial intelligence large language model (LLM) and performs semantic analysis on the natural language instructions input by the user to identify the user's intent.

4. The RPA system based on natural language interaction according to claim 1, characterized in that: The process planning module combines the operable objects of the screen interface and the user's intention to find the target operation object and operation method. The target operation object is the operable element in the interface, which can be text, picture, video, link and form element. The operation methods include mouse movement, mouse click, mouse double-click, holding down the mouse to drag, mouse scrolling up and down, keyboard input and option selection.

5. The RPA system based on natural language interaction according to claim 1, characterized in that: The execution module utilizes the existing automated user interface (UI) operation technology to simulate user operations, control the mouse and keyboard to execute the planned operation mode on the target operation object, and realize the execution of the automated task.

6. The RPA system based on natural language interaction according to claim 1, characterized in that: After the execution module performs the operation, the checking module verifies the operation result to ensure that the user's intention is met. When the checking module detects that the operation result does not meet the user's intention, the system will return to the process planning module and the execution module and repeat the execution until the user's intention is met or the number of executions exceeds the threshold set by the system.

7. A method for implementing an RPA system based on natural language interaction, characterized in that The process includes the following steps: a) The user inputs natural language instructions through a text box or microphone; b) The intention recognition module uses an artificial intelligence language model to recognize the user's intention; c) The process planning module plans the target operation object and operation method for the next operation based on the user's intention and the screen interface; d) The execution module controls the mouse and keyboard to execute the planned operations; e) The checking module checks whether the execution result meets the user's intention. If it does not meet the user's intention, the process planning module and the execution module will be repeatedly executed until the user's intention is met or the number of executions exceeds the threshold set by the system; f) The output module feeds back the execution results to the user.

8. The method for implementing the RPA system based on natural language interaction according to claim 7, characterized in that: In step a), the user input module further includes: automatically capturing the current screen interface as auxiliary input; in addition to text input, a voice input option is also provided, allowing the user to input natural language instructions through a microphone.

9. The method for implementing the RPA system based on natural language interaction according to claim 7, characterized in that: In step b), the intent recognition module uses a pre-trained artificial intelligence large language model (LLM) to perform semantic analysis to identify user intent.

10. The method for implementing the RPA system based on natural language interaction according to claim 7, characterized in that: In step c), the process planning module further includes: an interface element recognition unit, which is used to recognize operable elements in the screen interface; and to find out the target operation object and operation method of the next operation by combining the user intention and the operable element information of the screen interface.

11. The method for implementing the RPA system based on natural language interaction according to claim 7, characterized in that: In step d), the execution module: utilizes the existing automated user interface (UI) operation technology to simulate user operations and control the mouse and keyboard to perform corresponding operations on the target operation element; Includes error handling mechanisms to deal with exceptional situations that may arise during execution.

12. The method for implementing the RPA system based on natural language interaction according to claim 7, characterized in that: In step e), the checking module verifies the operation results to ensure that the user's intention is met; when the intention is not met, the current screen is captured as a new input to the process planning module, and the process planning step c) and execution step d) operations are continued until the user's intention is met or the number of executions exceeds the threshold set by the system.

Images