A user interaction interface operation method based on RPA agent

Abstract

The present application relates to the field of artificial intelligence, and in particular to a user interaction interface operation method based on RPA agent, which mainly includes: collecting user input data, using a multimodal model to identify the input data, and the multimodal model converting the input data into a business process diagram; the subtask agent analyzes the task intent through a multimodal model, breaks down the task intent into executable steps, and opens the corresponding user interaction interface according to the analysis result of the task intent; the subtask agent generates a corresponding RPA operation control based on the page operation description data, and executes the RPA operation control. The present application reduces the degree of manual participation in RPA operations, shortens the time for task execution, significantly improves the overall automation efficiency, and lowers the entry threshold of RPA technology.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a user interaction interface operation method based on RPA agent. Background Art

[0002] With the rapid development of information technology, especially the continuous advancement of large language models (LLMs), the field of automation applications has ushered in unprecedented changes. The core of this change is to promote the development of advanced network-based intelligent agents, aiming to achieve more efficient and intelligent automation operations.

[0003] However, most of the network agents currently on the market, such as web agents, face problems such as single functions and limited application environments. These agents mainly focus on the field of web automation, cannot solve the automation problems of the client, and have significant limitations in practical applications. Specifically, existing web agents mainly rely on a single input mode, and their performance evaluation is often limited to a simplified network simulation environment or a static network snapshot. This evaluation method cannot fully reflect the performance of the agent in the actual complex and changeable network environment, thereby limiting its effectiveness and reliability in practical applications. In addition, each repeated task requires multimodal agents to navigate the network, which not only consumes a lot of token costs, but also causes the multimodal large model to explore the network navigation for a long time, and the trial and error cost is high. For tasks that need to be repeated multiple times, the network navigation needs to be re-performed each time, and it is impossible to optimize the network navigation path manually, thereby reducing the overall efficiency. On the other hand, although the traditional RPA (Robotic Process Automation) technology has achieved process automation to a certain extent, there are also many inconveniences in its application process. For example, RPA technology usually requires developers to interact through manual drag-and-drop or recording to design automation scripts. This process not only requires developers to have a certain code foundation, but also requires them to fill in a large number of properties and think about how to choose appropriate controls to implement business processes. This not only increases the entry threshold for business personnel, but also limits the application of RPA technology in complex processes. In addition, traditional RPA technology cannot effectively use existing business process knowledge to assist developers in designing processes, further limiting its intelligence and scope of application. In summary, existing web agents and RPA technologies have significant deficiencies in the field of automation applications.

[0004] Therefore, how to lower the entry threshold for using RPA to automate user interaction interfaces such as web pages and clients is an urgent problem that needs to be solved. Summary of the invention

[0005] This application provides a user interaction interface operation method based on RPA agent. By combining the parsing capability of multimodal large models and the execution capability of RPA tools, it realizes the intelligent and automated processing of user needs, lowers the threshold for using RPA, and enables business personnel to implement complex RPA processes without the help of professional developers.

[0006] In order to achieve the above object, the present invention adopts the following technical scheme:

[0007] In a first aspect, the present invention provides a user interaction interface operation method based on an RPA agent, comprising:

[0008] Collect user input data, use a multimodal model to identify the input data, and convert the input data into a business process diagram. During the conversion, each subtask in the process diagram is limited to one task intention and at most one user interaction interface is operated.

[0009] Execute the steps in the flowchart according to the business process diagram. When executing the subtask, input the task intent of the subtask into the subtask agent of the RPA. The subtask agent analyzes the task intent through a multimodal model, decomposes the task intent into executable steps, and opens a corresponding user interaction interface according to the analysis result of the task intent;

[0010] Use the RPA page control to obtain the page structure data of the corresponding user interaction interface;

[0011] Taking a screenshot of the user interaction interface;

[0012] Analyzing the page structure data, the screenshots, and the task intent using the multimodal model to obtain page operation description data;

[0013] Input the page operation description data into the subtask agent, and the subtask agent generates a corresponding RPA operation control based on the page operation description data, and executes the RPA operation control;

[0014] After obtaining the execution result of the subtask, the next step is executed until all steps of the business process diagram are executed.

[0015] In a preferred example of the present application, it can be further configured that the subtask agent analyzes the task intention through a multimodal model and decomposes the task intention into executable steps:

[0016] The subtask agent retrieves system information related to the task intent through RAG, then analyzes the task intent through a multimodal model, and breaks down the task intent into executable steps based on the system information. The system information includes the operation object type, initialization information, and the address or name of the operation object.

[0017] In a preferred example of the present application, it can be further configured to include:

[0018] If the multimodal model of the subtask agent determines that the task intent lacks system information, execution is stopped and the user is prompted to supplement the system information. After the user completes the supplement, execution is continued.

[0019] In a preferred example of the present application, it can be further configured that the page structure data of the corresponding user interaction interface is obtained using the page control of the RPA, including:

[0020] If the corresponding user interaction interface is a browser, use the RPA page control to obtain the DOM structure of the browser, obtain the structured information of the operable elements from the DOM structure, and add labels to the operable elements. The structured information includes the coordinate center point, label number, type and text content.

[0021] In a preferred example of the present application, it can be further configured that the page structure data of the corresponding user interaction interface is obtained using the page control of the RPA, including:

[0022] If the corresponding user interaction interface is a client, tags are added to the operable elements through the OmniParser tool, and structured information of the operable elements is obtained.

[0023] In a preferred example of the present application, it can be further configured that the page operation description data is data in JSON format.

[0024] In a second aspect, the present application provides a user interaction interface operation device based on an RPA agent, the device comprising:

[0025] The data collection module is used to collect user input data, use a multimodal model to identify the input data, and convert the input data into a business process diagram. During the conversion, each subtask in the process diagram is limited to a task intention and at most one user interaction interface is operated.

[0026] An agent module is used to execute the steps in the flowchart according to the business process diagram, and when executing the subtask, input the task intent of the subtask into the subtask agent of the RPA, and the subtask agent analyzes the task intent through a multimodal model, decomposes the task intent into executable steps, and opens a corresponding user interaction interface according to the analysis result of the task intent;

[0027] A page structure analysis module, used to obtain page structure data of the corresponding user interaction interface using the page control of the RPA; take a screenshot of the user interaction interface; and use the multimodal model to analyze the page structure data, the screenshot, and the task intent to obtain page operation description data;

[0028] The execution module inputs the page operation description data into the subtask agent, and the subtask agent generates a corresponding RPA operation control based on the page operation description data and executes the RPA operation control; after obtaining the execution result of the subtask, the next step is executed until all steps of the business process diagram are executed.

[0029] In a third aspect, the present application provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the user interaction interface operation method based on the RPA agent are implemented as described in any one of the above items.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium having a program stored thereon, wherein when the program is executed by a processor, the user interaction interface operation method based on the RPA agent as described in any one of the above items is implemented.

[0031] In a fifth aspect, the present application provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the user interaction interface operation method based on the RPA agent as described in any one of the above items.

[0032] In summary, compared with the prior art, the technical solution provided in the embodiment of the present application has at least the following beneficial effects:

[0033] 1. This application combines multimodal big models with RPA technology to automatically decompose business, generate business process diagrams, and call RPA tools to perform specific operations. This greatly reduces the degree of manual participation, shortens the time for task execution, and significantly improves the overall automation efficiency.

[0034] 2. Traditional RPA technology usually requires developers to have a high level of code foundation and to design automation scripts by manually dragging or recording. However, this application can automatically generate business process diagrams and RPA controls through intelligent analysis of multimodal large models, without requiring developers to have complex programming skills, thus reducing the complexity of operations and the threshold for using RPA technology.

[0035] 3. After generating the RPA control, this application can use the capture capability of the RPA tool to obtain the element location of the interactive interface and fill this location information back into the RPA control. This avoids the need to call the multimodal large model for element recognition every time a task is executed, that is, repeated tasks do not need to rely on multimodal agents for network navigation, and the token cost consumed is low. Users can participate in the adjustment and optimization of the network navigation path, and the next time the task is repeated, it can be executed according to the best path, thereby reducing the token cost and improving the running speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A flowchart of a user interaction interface operation method based on an RPA agent is provided for one embodiment of the present application.

[0037] Figure 2 A structural diagram of a user interaction interface operating device based on an RPA agent provided for one embodiment of the present application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0039] In one embodiment of the present application, a user interaction interface operation method based on RPA agent is provided. Figure 1 As shown, the method includes:

[0040] S100: Collecting user input data, using a multimodal model to identify the input data, and converting the input data into a business process diagram. During the conversion, each subtask in the process diagram is limited to one task intention, and at most one user interaction interface is operated.

[0041] Specifically, the multimodal model refers to a model that combines multimodal information such as text, images, videos, and audio for training, such as large language models (LLMs) and multi-model large language models (MLLM). The user interaction interface is the operation object of the RPA technology of this application, and can be a user interaction interface of a website system or client software, such as a browser. The task intent is the operation target of RPA, and each task intent operates at most one website system or one client software. The business process diagram consists of start, end, conditions, loops, and subtasks.

[0042] After the business process diagram is generated, a modification function is provided so that users can adjust the business process diagram based on their familiarity with the business.

[0043] S200: Execute the steps in the flowchart according to the business process diagram. When executing the subtask, input the task intent of the subtask into the subtask agent of RPA. The subtask agent analyzes the task intent through a multimodal model, breaks down the task intent into executable steps, and opens a corresponding user interaction interface according to the analysis result of the task intent.

[0044] Specifically, after the user determines the business process diagram, the system executes in order according to the business process diagram. When encountering a subtask, the task intent of the subtask is sent to the RPA subtask agent as input. The RPA subtask agent will schedule the intent analysis and process disassembly agent, retrieve the system information related to the intent through RAG, and let the multimodal model (LLM) further disassemble the task intent into executable steps. If the multimodal model determines that system information is missing, such as the website address, or information that needs to be entered, the user is prompted to complete it before disassembling the steps. Finally, the user confirms whether to execute according to the disassembled steps.

[0045] If executed, it enters the execution part of the RPA subtask agent. The RPA subtask agent is executed based on the system information analyzed by the multimodal model in the previous step. The system information includes execution-related information, such as the operation type is browser or client, the corresponding website URL, the client name or the initialization automation environment. If it is a browser, the RPA tool is used to open the browser and URL. If it is client software, the RPA tool is used to start the client software.

[0046] S300: using the page control of the RPA to obtain the page structure data of the corresponding user interaction interface; taking a screenshot of the user interaction interface; using the multimodal model to analyze the page structure data, the screenshot and the task intent, and obtaining page operation description data;

[0047] Specifically, a screenshot of the current operation object is obtained, and the screenshot needs to be labeled, as well as the structured information of the operable elements, including the coordinate center point, label number, type, and text content. Among them, if the operation object is a browser, the DOM structure of the browser's web page is obtained through the RPA tool, and conventional operable elements such as buttons and input boxes are screened and found, and JS code is executed to add labels to them, and then the structured information of the operable elements is obtained. If it is client software, the operable element tags are generated through Microsoft's open source OmniParser tool, and the structured information of the operable elements is obtained.

[0048] After receiving the screenshot, task intent, and structured information of operable elements, the multimodal large model gives a response for the next operation, namely, the page operation description data, which may be JSON data.

[0049] S400: inputting the page operation description data into the subtask agent, the subtask agent generating a corresponding RPA operation control based on the page operation description data, and executing the RPA operation control;

[0050] Specifically, the execution part of the subtask agent translates the page operation description data into specific RPA operation controls and fills in the attributes based on the page operation description data returned by the multimodal large model.

[0051] Execute the RPA operation control, and repeatedly execute S300 and S400 until the multimodal large model outputs the final result, and then stop executing.

[0052] S500: After obtaining the execution result of the subtask, execute the next step until all steps of the business process diagram are executed.

[0053] Specifically, the execution results output by the multimodal large model are displayed, and the RPA controls generated by the execution process are saved as the RPA process of the subtask, and the operation of the next subtask is entered until the entire business process diagram is completed.

[0054] In this embodiment, the beneficial effects include at least:

[0055] 1. This application combines multimodal big models with RPA technology to automatically decompose business, generate business process diagrams, and call RPA tools to perform specific operations. This greatly reduces the degree of manual participation, shortens the time for task execution, and significantly improves the overall automation efficiency.

[0056] 2. Traditional RPA technology usually requires developers to have a high level of code foundation and to design automation scripts by manually dragging or recording. However, this application can automatically generate business process diagrams and RPA controls through intelligent analysis of multimodal large models, without requiring developers to have complex programming skills, thus reducing the complexity of operations and the threshold for using RPA technology.

[0057] 3. After generating the RPA control, this application can use the capture capability of the RPA tool to obtain the element location of the interactive interface and fill these location information back into the RPA control. This avoids the need to re-call the multimodal large model for element recognition every time a task is executed, that is, repeated tasks do not need to rely on multimodal agents for network navigation, and the token cost consumed is low. Users can participate in the adjustment and optimization of the network navigation path, and the next time the task is repeated, it can be executed according to the best path, thereby reducing the token cost and increasing the running speed. The principles for achieving this effect include: when locating the automated operation of the RPA process, the positioning using the RPA native tool will be much faster, while the recognition of images using the multimodal large model will be slower.

[0058] In some embodiments, the subtask agent analyzes the task intent through a multimodal model and breaks down the task intent into executable steps:

[0059] The subtask agent retrieves system information related to the task intent through RAG, then analyzes the task intent through a multimodal model, and breaks down the task intent into executable steps based on the system information. The system information includes the operation object type, initialization information, and the address or name of the operation object.

[0060] In this embodiment, by introducing RAG search enhancement technology and manual optimization, the decomposition capability and execution efficiency of automated tasks are improved. At the same time, by using RAG search enhancement technology to search for the URL of the corresponding system, it can also be used for the customer's intranet system, without the customer having to enter the URL knowledge of the intranet system.

[0061] In some embodiments, it also includes:

[0062] If the multimodal model of the subtask agent determines that the task intent lacks system information, execution is stopped and the user is prompted to supplement the system information. After the user completes the supplement, execution is continued.

[0063] In this embodiment, by providing users with the function of interrupting, adjusting and guiding the next operation of the multimodal large model at any time during the operation of the multimodal large model, the success rate of completing the RPA operation is greatly improved.

[0064] In some embodiments, using the page control of the RPA to obtain the page structure data of the corresponding user interaction interface includes:

[0065] If the corresponding user interaction interface is a browser, use the RPA page control to obtain the DOM structure of the browser, obtain the structured information of the operable elements from the DOM structure, and add labels to the operable elements. The structured information includes the coordinate center point, label number, type and text content.

[0066] In this embodiment, by introducing DOM parsing to generate operable element tags during the web page automation process, the accuracy of the multimodal large model in identifying elements is improved, and it is convenient to find the element objects of the corresponding RPA automation process script.

[0067] In some embodiments, using the page control of the RPA to obtain the page structure data of the corresponding user interaction interface includes:

[0068] If the corresponding user interaction interface is a client, tags are added to the operable elements through the OmniParser tool, and structured information of the operable elements is obtained.

[0069] In this embodiment, Microsoft's open source OmniParser tool is introduced to generate operational element tags, which improves the accuracy of multimodal large models in identifying elements and facilitates finding the corresponding element objects of the RPA automation process script.

[0070] In some embodiments, the page operation description data is data in JSON format.

[0071] In this embodiment, by instructing the multimodal large model to output data in JSON format, the accuracy and efficiency of generating corresponding controls are improved.

[0072] This application also provides a user interaction interface operation device based on RPA agent, see Figure 2 As shown, the device comprises:

[0073] The data collection module 100 is used to collect the input data of the user, identify the input data using a multimodal model, and convert the input data into a business process diagram. During the conversion, each subtask in the process diagram is limited to a task intention, and at most one user interaction interface is operated;

[0074] The agent module 200 is used to execute the steps in the flowchart according to the business process diagram, and when executing the subtask, input the task intent of the subtask into the subtask agent of the RPA, and the subtask agent analyzes the task intent through a multimodal model, decomposes the task intent into executable steps, and opens a corresponding user interaction interface according to the analysis result of the task intent;

[0075] The page structure analysis module 300 is used to obtain the page structure data of the corresponding user interaction interface using the page control of the RPA; take a screenshot of the user interaction interface; and use the multimodal model to analyze the page structure data, the screenshot and the task intent to obtain page operation description data;

[0076] Execution module 400 inputs the page operation description data into the subtask agent, and the subtask agent generates a corresponding RPA operation control based on the page operation description data and executes the RPA operation control; after obtaining the execution result of the subtask, the next step is executed until all steps of the business process diagram are executed.

[0077] The functional implementation of each module in the above-mentioned RPA agent-based user interaction interface operation device corresponds to the steps in the above-mentioned RPA agent-based user interaction interface operation method embodiment, and its functions and implementation process will not be repeated here one by one.

[0078] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the user interaction interface operation method based on the RPA agent as described in any of the above embodiments are implemented.

[0079] The present application also provides a computer-readable storage medium on which a program is stored, wherein the computer-readable storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive, and / or a memory stick, etc., and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The working process, working details, and technical effects of the computer-readable storage medium provided in this embodiment can be found in the above embodiment of a user interaction interface operation method based on an RPA agent, which will not be repeated here.

[0080] The application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the steps of the user interaction interface operation method based on the RPA agent as described in any of the above embodiments.

[0081] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0082] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification. The above-mentioned embodiments only express several implementation methods of the present application, and their descriptions are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that for ordinary technicians in this field, without departing from the concept of the present application, several deformations and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of this application shall be based on the attached claims.

Claims

1. A user interaction interface operation method based on RPA agent, characterized in that: include: Collecting user input data, using a multimodal model to identify the input data, and converting the input data into a business process diagram by the multimodal model. During the conversion, each subtask in the process diagram is limited to one task intention, and at most one user interaction interface is operated; Execute the steps in the flowchart according to the business process diagram. When executing the subtask, input the task intent of the subtask into the subtask agent of the RPA. The subtask agent analyzes the task intent through a multimodal model, decomposes the task intent into executable steps, and opens a corresponding user interaction interface according to the analysis result of the task intent; Use the RPA page control to obtain the page structure data of the corresponding user interaction interface; Taking a screenshot of the user interaction interface; Analyzing the page structure data, the screenshots, and the task intent using the multimodal model to obtain page operation description data; Input the page operation description data into the subtask agent, and the subtask agent generates a corresponding RPA operation control based on the page operation description data, and executes the RPA operation control; After obtaining the execution result of the subtask, the next step is executed until all steps of the business process diagram are executed.

2. The user interaction interface operation method based on RPA agent according to claim 1 is characterized in that: The subtask agent analyzes the task intent through a multimodal model and breaks down the task intent into executable steps: The subtask agent retrieves system information related to the task intent through RAG, then analyzes the task intent through a multimodal model, and breaks down the task intent into executable steps based on the system information. The system information includes the operation object type, initialization information, and the address or name of the operation object.

3. The user interaction interface operation method based on RPA agent according to claim 2 is characterized in that: Also includes: If the multimodal model of the subtask agent determines that the task intent lacks system information, execution is stopped and the user is prompted to supplement the system information. After the user completes the supplement, execution is continued.

4. The user interaction interface operation method based on RPA agent according to claim 3 is characterized in that: Using the RPA page control to obtain the page structure data of the corresponding user interaction interface includes: If the corresponding user interaction interface is a browser, use the RPA page control to obtain the DOM structure of the browser, obtain the structured information of the operable elements from the DOM structure, and add labels to the operable elements. The structured information includes the coordinate center point, label number, type and text content.

5. The user interaction interface operation method based on RPA agent according to claim 4 is characterized in that: Using the RPA page control to obtain the page structure data of the corresponding user interaction interface includes: If the corresponding user interaction interface is a client, tags are added to the operable elements through the OmniParser tool, and structured information of the operable elements is obtained.

6. The user interaction interface operation method based on RPA agent according to claim 5 is characterized in that: The page operation description data is in JSON format.

7. A user interaction interface operation device based on RPA agent, characterized in that: include: A data collection module is used to collect user input data, identify the input data using a multimodal model, and convert the input data into a business process diagram. During the conversion, each subtask in the process diagram is limited to one task intention, and at most one user interaction interface is operated; An agent module is used to execute the steps in the flowchart according to the business process diagram, and when executing the subtask, input the task intent of the subtask into the subtask agent of the RPA, and the subtask agent analyzes the task intent through a multimodal model, decomposes the task intent into executable steps, and opens a corresponding user interaction interface according to the analysis result of the task intent; A page structure analysis module, used to obtain page structure data of the corresponding user interaction interface using the page control of the RPA; Taking a screenshot of the user interaction interface; Analyzing the page structure data, the screenshots, and the task intent using the multimodal model to obtain page operation description data; An execution module inputs the page operation description data into the subtask agent, and the subtask agent generates a corresponding RPA operation control based on the page operation description data and executes the RPA operation control; After obtaining the execution result of the subtask, the next step is executed until all steps of the business process diagram are executed.

8. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the user interaction interface operation method based on the RPA agent are implemented as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, wherein when the program is executed by the processor, the user interaction interface operation method based on the RPA agent as described in any one of claims 1 to 6 is implemented.

10. A computer program product comprising computer instructions, characterized in that: When executed by a processor, the computer instructions implement the steps of the RPA agent-based user interaction interface operation method described in claims 1 to 6.

Images