Industrial RPA automation system based on AI and construction method

Abstract

The invention discloses an AI-based industrial RPA automatic construction method and system. The construction method comprises the following steps: constructing a real-time data channel and an Mes system interface connected with an Mes docking service module; realizing cross-system UI automatic operation based on the data provided by the real-time data channel, and debugging an instruction of the UI automatic operation; the AI enhancement module identifies interface elements for screen data provided by the UI automatic actuator, compensates positioning errors, optimizes decisions and assists in process reconstruction; the data processing module is used for processing data provided by the real-time data channel and storing equipment data, so as to realize data interaction with the Mes docking service module; and carrying out optimized task scheduling and resource allocation on the equipment data and the data in the Mes docking service module. According to the industrial RPA automation system based on the AI, the dependence on a third-party system is reduced through a non-intrusive UI automation technology, and the intelligent level of the RPA system is improved.

Description

Technical Field

[0001] The present invention relates to the field of robots, and particularly to an AI-based industrial RPA automation system and a construction method thereof. Background Art

[0002] RPA (Robotic Process Automation) technology is an automated solution that uses software robots to simulate humans performing repetitive and rule-based tasks on a computer. RPA can achieve the automation of business processes by interacting with the interfaces of existing applications and systems without large-scale transformation of the underlying systems.

[0003] Currently, RPA technology is mostly applied in the fields of games and office work. In order to accelerate the development of the industrial field, RPA has been introduced into the industrial field. However, the existing RPA technology needs to use an invasive method to introduce a third party for control and drive, with high implementation risks and difficulties. At the same time, RPA cannot automatically perform intelligent supplementation, making it impossible to guarantee the accuracy of operations, thus unable to meet the development of RPA technology in the industrial field. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide an AI-based industrial RPA automation system and a construction method thereof, which can be accurately and efficiently applied to the industrial field and improve production efficiency.

[0005] To achieve the above object, the technical solution adopted by the present invention is: an AI-based industrial RPA automation construction method, including the following steps:

[0006] Construct a real-time data channel and a Mes system interface connected to the Mes docking service module;

[0007] The UI automation executor realizes cross-system UI automation operations based on the data provided by the real-time data channel, and then debugs the instructions of the UI automation operations;

[0008] The AI enhancement module identifies interface elements for the screen data provided by the UI automation executor, compensates for positioning errors, optimizes decisions, and assists in process reconstruction;

[0009] Process and store device data for the data provided by the real-time data channel, and perform data interaction with the Mes docking service module;

[0010] Optimize task scheduling and resource allocation for the device data and the data in the Mes docking service module, and then provide human-computer interaction support.

[0011] Further, after the visual cognitive center module captures the information of screen elements in a non-invasive manner, it sends the information to the UI automation executor, which uses the screen element information to perform cross-system UI automation operations; at the same time, the human-computer interaction behavior simulation component and the intelligent decision engine module cooperate to optimize the UI automation operations, and the multi-window / cross-system operation coordinator and the adaptive scheduler module manage the priority of executing the UI automation operations.

[0012] Further, the steps of capturing the information of screen elements in a non-invasive manner are as follows: first, capture the screen frame of the target window, and then use Tesseract OCR to identify: first locate the boundary of the scrolling area of the alarm list, and then parse the control tree structure; finally, generate structured JSON.

[0013] Further, when debugging and providing feedback on the instructions of the UI automation operations, the process breakpoint debugging module interacts with the human-machine collaboration interface module to provide debugging feedback.

[0014] Further, in the AI enhancement module, the dynamic interface element recognition model provides element positioning for the UI automation executor; the multi-modal positioning compensation algorithm and the intelligent decision engine module cooperate to optimize the positioning accuracy and then compensate for the positioning error; the interface change self-learning module and the Mes docking service module provide the adaptation ability of the dynamic interface.

[0015] Further, the adaptive scheduler module provides the processing strategy for exception decision-making through the exception mode knowledge graph; the LLM-assisted process reconstruction module provides optimization suggestions for the script interpreter module.

[0016] Further, after compressing the real-time data in the real-time data channel, historical data is provided for the intelligent decision engine module; the device status feature extraction provides analysis data for the Mes docking service module.

[0017] Further, the Mes docking service module realizes data interaction with the Mes system through the Mes interface.

[0018] Further, the data provided by the device data and the Mes docking service module, in cooperation with the multi-task priority management and the UI automation executor module, optimize the task execution order; the resource conflict detection mechanism and the intelligent decision engine module cooperate to solve the resource competition problem; the energy consumption optimization controller and the human-machine collaboration interface module cooperate to achieve energy consumption monitoring and optimization.

[0019] An AI-based industrial RPA automation system includes:

[0020] An initialization and construction module for constructing a real-time data channel and a Mes system interface connected to the Mes docking service module;

[0021] An automated operation module, which enables the UI automation executor to perform cross-system UI automated operations based on the data provided by the real-time data channel, and then debugs the instructions of the UI automated operations;

[0022] An AI optimization module, which enables the AI enhancement module to identify interface elements from the screen data provided by the UI automation executor, compensate for positioning errors, optimize decisions, and assist in process reconstruction;

[0023] A data center processing module, which processes the data provided by the real-time data channel, stores device data, and realizes data interaction with the Mes docking service module;

[0024] An optimized interaction module, which optimizes task scheduling and resource allocation for the device data and the data in the Mes docking service module, and then provides human-computer interaction support.

[0025] Due to the application of the above technical solutions, the present invention has the following advantages compared with the prior art:

[0026] 1. Through non-intrusive UI automation technology, the dependence on third-party systems is reduced, and the development and operation processes are simplified.

[0027] 2. By combining deep learning and large language models, the intelligence level of the RPA system is improved, and it can cope with complex industrial environments and dynamic task requirements.

[0028] 3. Integrating an industrial data gateway and a Mes docking service, real-time collection, analysis, and reporting of industrial data are realized, supporting intelligent management of industrial production. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The technical solutions of the present invention will be further described below with reference to the accompanying drawings:

[0030] Figure 1 It is a flowchart of a method for constructing an AI-based industrial RPA automation in an embodiment of the present invention;

[0031] Figure 2 It is a structural diagram of an AI-based industrial RPA automation construction system in an embodiment of the present invention;

[0032] Among them: 1. Initialization construction module; 2. Automated operation module; 3. AI optimization module; 4. Data center processing module; 5. Optimized interaction module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0033] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0034] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these example embodiments are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted.

[0035] The present invention provides an AI-based industrial RPA automation construction method and system to solve the problem in the prior art that the RPA technology cannot be accurately and efficiently applied to the industrial field, and the implementation risk and difficulty are high.

[0036] For the sake of easy understanding, the specific processes in the embodiments of the present application will be described below. Please refer to Figure 1 An AI-based industrial RPA automation construction method in the embodiments of the present application includes the following steps:

[0037] S1. Construct a real-time data channel and a Mes system interface connected to the Mes docking service module; in step S1, it is to ensure that various tasks can be smoothly carried out before the system runs and provide a stable and efficient operating environment. Specifically, integrate protocol parsers such as CUA / Modbus / Profinet, and establish a real-time data channel with PLC / DCS. In addition, the Mes system interface is configured to dock with the work order database to provide a data source for the Mes docking service module.

[0038] S2. The UI automation executor realizes cross-system UI automation operations based on the data provided by the real-time data channel, and then debugs the instructions of the UI automation operations.

[0039] In step S2, the interface element recognition capability of the visual cognitive center module captures the information of screen elements in a non-invasive manner and sends it to the UI automation executor, which uses the screen element information to perform cross-system UI automation operations. At the same time, the human-computer interaction behavior simulation component and the intelligent decision-making engine module collaborate to optimize the UI automation operations, and the multi-window / cross-system operation coordinator and the adaptive scheduler module manage the priority of executing UI automation operations. This eliminates the need to control and drive third-party software and systems in an "invasive" manner by referencing third-party SDKs or plug-ins, greatly reducing the implementation risk and difficulty, while also simplifying the development and operation processes.

[0040] The specific process of capturing information of screen elements in a non-invasive manner includes: 1. Capturing the screen frame of the target window (including the rendering result of the .NET control); 2. Using Tesseract OCR for recognition. Specifically, first locate the scrolling area boundary of the alarm list, pixel coordinates: x=120, y=240, w=800, h=600; then parse the control tree structure (including 20 dynamically generated alarm entry div layers); 3. Generate structured JSON.

[0041] Secondly, when debugging feedback on the instructions of UI automation operations, the process breakpoint debugging module interacts with the human-computer collaborative interface module to provide debugging feedback.

[0042] S3, AI enhancement module identifies interface elements based on the screen data provided by the UI automation executor and compensates for positioning errors while optimizing decisions and assisting process reconstruction;

[0043] In step S3, for the AI ​​enhancement module, the dynamic interface element recognition model provides element positioning for the UI automation executor; the multimodal positioning compensation algorithm and the intelligent decision engine module jointly optimize the positioning accuracy and then compensate for the positioning error; the interface change self-learning module and the Mes docking service module provide dynamic interface adaptation capabilities. In the interface change self-learning module, a deep learning framework is introduced to intelligently supplement RPA automation. AI automation planning includes: AI visual planning, AI large language model execution planning, etc. This means using new deep learning methods to improve some of the drawbacks brought about by traditional visual methods and traditional execution planning methods, such as the lack of flexibility in locating elements by traditional visual methods, and the modification of too many and too complex parameters; traditional execution planning methods cannot provide positive feedback on execution failures and errors, etc.

[0044] Furthermore, the adaptive scheduler module provides a processing strategy for abnormal decisions through the abnormal pattern knowledge graph; the LLM-assisted process reconstruction module provides optimization suggestions for the script interpreter module.

[0045] S4. Process and store device data provided by the real-time data channel, and perform data interaction with the Mes docking service module;

[0046] In step S4, after compressing the real-time data in the real-time data channel, historical data is provided for the intelligent decision-making engine module; the extraction of device status features provides analysis data for the Mes docking service module, so as to prepare for subsequent use.

[0047] S5. Perform optimization task scheduling and resource allocation on the device data and the data in the Mes docking service module, and then provide human-computer interaction support.

[0048] In step S5, the Mes docking service module realizes data interaction with the Mes system through the Mes interface.

[0049] Furthermore, the device data and the data provided by the Mes docking service module are collaborated by the multi-task priority management and the UI automation executor module to optimize the task execution order; the resource conflict detection mechanism collaborates with the intelligent decision-making engine module to solve the resource competition problem; the energy consumption optimization controller collaborates with the human-machine collaboration interface module to realize energy consumption monitoring and optimization.

[0050] Furthermore, the human-machine collaboration interface is used to provide human-computer interaction support.

[0051] The method for constructing an AI-based industrial RPA automation system of the present invention reduces the dependence on third-party systems through non-intrusive UI automation technology and simplifies the development and operation processes. Secondly, by combining deep learning and large language models, the intelligence level of the RPA system is improved, and it can cope with complex industrial environments and dynamic task requirements. In addition, the industrial data gateway and the Mes docking service are integrated to realize real-time collection, analysis and reporting of industrial data, and support the intelligent management of industrial production.

[0052] Refer to Figure 2 , the present invention also discloses an AI-based industrial RPA automation system, including: an initialization construction module for constructing a real-time data channel and a Mes system interface connected to the Mes docking service module; an automation operation module for the UI automation executor to perform cross-system UI automation operations based on the data provided by the real-time data channel, and then debug the instructions of the UI automation operations; an AI optimization module for the AI enhancement module to identify interface elements and compensate for positioning errors on the screen data provided by the UI automation executor, while optimizing decisions and assisting in process reconstruction; a data center processing module for processing and storing device data provided by the real-time data channel, and performing data interaction with the Mes docking service module; an optimization interaction module for performing optimization task scheduling and resource allocation on the device data and the data in the Mes docking service module, and then providing human-computer interaction support.

[0053] Through the above system, the implementation difficulty and cost of AI-based industrial RPA can be reduced, and the relevant control accuracy can be improved.

[0054] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An AI-based industrial RPA automation construction method, characterized in that, Including the following steps: Construct a real-time data channel and a Mes system interface connected to the Mes docking service module; The UI automation executor realizes cross-system UI automation operations based on the data provided by the real-time data channel, and then debugs the instructions of the UI automation operations; The AI enhancement module identifies interface elements from the screen data provided by the UI automation executor, compensates for positioning errors, optimizes decisions, and assists in process reconstruction; Process and store device data from the data provided by the real-time data channel, as well as data interaction with the Mes docking service module; Optimize task scheduling and resource allocation for the device data and the data in the Mes docking service module, and then provide human-computer interaction support.

2. The AI-based industrial RPA automation construction method according to claim 1, wherein: The visual cognitive center module captures the information of screen elements in a non-invasive manner and sends it to the UI automation executor. The UI automation executor performs cross-system UI automation operations using the screen element information; at the same time, the human-computer interaction behavior simulation component and the intelligent decision engine module cooperate to optimize the UI automation operations, and the multi-window / cross-system operation coordinator and the adaptive scheduler module manage the priority of executing the UI automation operations.

3. The AI-based industrial RPA automation construction method according to claim 2, wherein The steps of capturing the information of screen elements in a non-invasive manner are as follows: First, capture the screen frame of the target window, and then use Tesseract OCR to identify: first locate the boundary of the scrolling area of the alarm list, then parse the control tree structure; finally, generate structured JSON.

4. The AI-based industrial RPA automation construction method according to claim 2, wherein: When providing debug feedback on the instructions of the UI automation operations, the process breakpoint debug module interacts with the human-machine collaboration interface module to provide debug feedback.

5. The AI-based industrial RPA automation construction method according to claim 1, wherein: In the AI enhancement module, the dynamic interface element recognition model provides element positioning for the UI automation executor; the multi-modal positioning compensation algorithm and the intelligent decision engine module cooperate to optimize the positioning accuracy and then compensate for the positioning errors; the interface change self-learning module and the Mes docking service module provide the adaptation ability of the dynamic interface.

6. The AI-based industrial RPA automation construction method according to claim 1, wherein: The adaptive scheduler module provides a processing strategy for exception decision-making through the exception mode knowledge graph; the LLM-assisted process reconstruction module provides optimization suggestions for the script interpreter module.

7. The AI-based industrial RPA automation construction method according to claim 1, wherein: After compressing the real-time data in the real-time data channel, provide historical data for the intelligent decision engine module; device status feature extraction provides analysis data for the Mes docking service module.

8. The AI-based industrial RPA automation construction method according to claim 1, wherein: The Mes docking service module realizes data interaction with the Mes system through the Mes interface.

9. The AI-based industrial RPA automation construction method according to claim 1, wherein: For the device data and the data provided by the Mes docking service module, the multi-task priority management and the UI automation executor module cooperate to optimize the task execution order; the resource conflict detection mechanism and the intelligent decision engine module cooperate to solve the resource competition problem; the energy consumption optimization controller and the human-machine collaboration interface module cooperate to realize energy consumption monitoring and optimization.

10. An AI-based industrial RPA automation system, characterized in that: Including: An initialization construction module for constructing a real-time data channel and a Mes system interface connected to the Mes docking service module; An automation operation module for the UI automation executor to realize cross-system UI automation operations based on the data provided by the real-time data channel, and then debug the instructions of the UI automation operations; AI Optimization Module, which is used to enable the AI Enhancement Module to identify interface elements from the screen data provided by the UI Automation Executor, compensate for positioning errors, optimize decisions, and assist in process reconstruction; Data Center Processing Module, which is used to process and store device data from the real-time data channel, and implement data interaction with the Mes Docking Service Module; Optimized Interaction Module, which is used to optimize task scheduling and resource allocation for the device data and the data in the Mes Docking Service Module, and then provide human-computer interaction support.

Images