A general data acquisition system based on RPA robots

Abstract

The application discloses a kind of general data acquisition system based on RPA robot, comprising;Terminal equipment, server end and background control end, the terminal equipment is connected with the background control end by the server, the server end acquires the terminal equipment in RPA data acquisition state data acquisition system operating state parameter, when RPA is reached in the process of running data running space upper limit, close the application program in suspended state, and whether the remaining space of terminal equipment expected operation meets the normal operation of RPA data extraction is calculated, if not satisfied, then the task amount of RPA data extraction is adjusted, avoid due to RPA data extraction terminal equipment running memory too large, leading to terminal computer equipment system crash.RPA robot data extraction setting task amount is larger in the prior art, computer terminal appears failure under the overload state of RPA robot, and the problem that RPA robot cannot normally run and extract data is solved.

Description

Technical Field

[0001] This invention relates to the field of network technology, and in particular to a general data acquisition system based on RPA robots. Background Technology

[0002] Robotic Process Automation (RPA), also known as digital labor, can simulate human operations such as copying, pasting, clicking, and inputting, freeing human workers from tasks that are "relatively fixed in rules, highly repetitive, and low in added value." Currently, this technology is being implemented in finance, banking, e-commerce, manufacturing, and retail. In practical work, data scraping is often required. Current data scraping techniques typically involve manual scraping by business personnel, scraping using general data scraping software, or IT departments writing traditional data scraping programs to scrape specific data and format it into the system. RPA's data scraping process is similar to that of general data scraping software; however, due to RPA's built-in visual processes and the abundance of built-in and component markets, RPA's data scraping process can seamlessly connect with other processes compared to general data scraping software.

[0003] Traditional technical solutions involve writing code to simulate the data scraping process at the network layer, requiring specialized developers. Due to its aggressive nature, many websites and apps block this approach. The main reason for this blocking is that the technology doesn't run in a real browser environment. Overcoming these blocks requires significant technical capabilities and development costs. RPA (Robotic Process Automation) scraping, on the other hand, runs directly in the browser, simulating human interaction, and therefore is not blocked. This demonstrates that current automated data scraping solutions fall into two categories: general-purpose data scraping software and traditional technical solutions.

[0004] However, in actual operation, existing technologies suffer from several drawbacks. Firstly, RPA robots consume a large amount of system memory during data acquisition. Secondly, RPA robots only handle the data acquisition steps and cannot monitor the computer terminal's operation in real time. This leads to problems where the computer terminal malfunctions due to the large workload of the RPA robot's data acquisition settings, causing the RPA robot to fail to acquire data properly. To address this issue, a general-purpose data acquisition system based on RPA robots has been developed. Summary of the Invention

[0005] The purpose of this invention is to provide a general data acquisition system based on RPA robots to solve the problem that existing RPA robots cannot properly acquire data because the computer terminal malfunctions when the RPA robot is overloaded due to the large amount of data acquisition tasks.

[0006] This invention provides a general data acquisition system based on RPA robots, comprising: a terminal device, a server, and a backend control terminal, wherein the terminal device is connected to the backend control terminal through the server;

[0007] The acquisition unit, wherein the server acquires the data capture system operating status parameters of the terminal device in the RPA data acquisition state;

[0008] The evaluation and detection unit, the server side, inputs the system operation status parameters of the terminal device in the RPA data acquisition state into the terminal system operation status evaluation model to obtain the system operation status result of the terminal device;

[0009] The logical construction unit, the server, constructs an application management business process processing logic database through RPA based on the system operation status results of the terminal device, and executes the application management business process processing logic;

[0010] The operation status analysis unit, the server side performs system operation status detection and data analysis on the terminal device system that has executed the application management business process processing logic;

[0011] The runtime space analysis unit, the server side performs data analysis and processing based on the terminal device system detection results under normal terminal device system operation, to obtain the maximum amount of general RPA data to be collected.

[0012] Furthermore, the evaluation and detection unit includes:

[0013] The server will collect parameters from the terminal device in RPA data collection state, including the number of application software running on the terminal device and the percentage of memory space occupied by the application software.

[0014] Construct an RPA robot time-segment data collection workflow to collect collection parameters in different time periods under the RPA data collection state;

[0015] If any abnormal parameter occurs during the RPA data acquisition process, the RPA robot data acquisition task will be paused, and an error report will be sent to the backend control terminal.

[0016] Furthermore, the logical building unit includes:

[0017] After the RPA data capture task is started, a static and constant safe operating memory limit is established based on the existing terminal device's operating space memory size;

[0018] After the RPA data crawling task is run, the running memory of the application software in the running state other than the RPA data crawling task is stored to the cloud server, and the running of the application software in the running state other than the RPA data crawling task is paused.

[0019] When RPA reaches the critical point, it transmits the time node record data to the cloud storage space corresponding to the application software, establishes a shutdown time node, and shuts down the application software.

[0020] Furthermore, the operational status analysis unit includes:

[0021] The server performs system operation detection on the terminal device system that has executed the application management business process processing logic, and obtains the detection result of the terminal device system.

[0022] If the operating status of the terminal device system meets the preset value, then the terminal device system is operating normally.

[0023] Furthermore, the operating space analysis unit includes:

[0024] The data analysis and processing includes the terminal device's running memory space data, the terminal device's hardware temperature data, and the terminal device's expected load capacity data over a period of time.

[0025] The operational space requirements of the terminal device during the estimated time period of RPA data collection are evaluated to obtain the estimated time period of RPA data collection operation space requirements.

[0026] The RPA data collection and operation space requirement data for the expected time period is compared with the load capacity data of the terminal device within the expected time period. If the data comparison result is abnormal, the information is fed back to the background control terminal.

[0027] The beneficial effects of this invention are as follows: This invention provides a general data acquisition system based on RPA robots, comprising: a terminal device, a server, and a backend control terminal. The terminal device is connected to the backend control terminal through the server. The server collects system operation status parameters of the terminal device in the RPA data acquisition state. The server inputs the system operation status parameters of the terminal device in the RPA data acquisition state into the terminal system operation status evaluation model to obtain the terminal device system operation status result. Based on the terminal device system operation status result, the server constructs an application management business process processing logic database through RPA and executes the application management business process processing logic. The server performs system operation status detection and data collection on the terminal device system that has executed the application management business process processing logic. The analysis shows that the server performs data analysis and processing based on the terminal device system detection results under normal operating conditions to obtain the maximum general data collection volume for RPA. By collecting operational data from the terminal device running RPA, the maximum load capacity of the terminal device is determined. During RPA data capture, the operation of other application software on the terminal device is paused, and the data generated by the application software during operation is stored to the cloud server and timed. When the upper limit of the running space is reached during RPA data capture, RPA is shut down, the paused applications are closed, and it is calculated whether the expected remaining running space of the terminal device is sufficient for the normal operation of RPA data capture. If not, the workload of RPA data capture is adjusted to avoid the terminal computer system crashing due to excessive memory occupation by RPA data capture. This solves the problem that existing RPA robot data capture settings with large workloads cause computer terminal failures under RPA robot overload conditions, preventing the RPA robot from performing normal data capture. Attached Figure Description

[0028] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0029] Figure 1 A price illustration of a general data acquisition system based on RPA robots provided in an embodiment of the present invention;

[0030] Figure 2 This is a schematic diagram of the evaluation and detection unit steps of a general data acquisition system based on RPA robots provided in an embodiment of the present invention;

[0031] Figure 3This invention provides a schematic diagram of the logical construction unit steps of a general data acquisition system based on an RPA robot, as shown in the embodiment of the invention.

[0032] Figure 4 This is a schematic diagram of the steps of a general data acquisition system operation status analysis unit based on an RPA robot, provided in an embodiment of the present invention.

[0033] Figure 5 This is a schematic diagram of the steps of a general data acquisition system based on an RPA robot for analyzing the operational space, provided in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. The technical solutions provided by various embodiments of this invention will be described in detail below with reference to the accompanying drawings.

[0035] Please see Figure 1 This invention provides a general data acquisition system based on RPA robots, including: a terminal device, a server, and a back-end control terminal, wherein the terminal device is connected to the back-end control terminal through the server;

[0036] The acquisition unit 101 is used by the server to acquire the data capture system operation status parameters of the terminal device in the RPA data acquisition state.

[0037] The server collects parameters from the operating status of terminal devices such as computers, mobile phones, and tablets.

[0038] The evaluation and detection unit 102, the server side, inputs the system operation status parameters of the terminal device in the RPA data acquisition state into the terminal system operation status evaluation model to obtain the terminal device system operation status result;

[0039] The system operating status parameters under RPA data acquisition are input into the terminal system operating status evaluation model. The resulting data is analyzed and judged to determine whether RPA data acquisition and capture is operating normally.

[0040] Logical construction unit 103: The server constructs an application management business process processing logic database through RPA based on the system operation status results of the terminal device, and executes the application management business process processing logic.

[0041] When RPA robots frequently crawl and collect data, they consume a large amount of memory on the terminal device. To prevent the terminal device system from crashing, we can establish RPA robot business processing logic, manage the application software, and create multiple sets of business processing logic to facilitate RPA robots to select the appropriate logic based on usage.

[0042] The operation status analysis unit 104, the server side performs system operation status detection and data analysis on the terminal device system that has executed the application management business process processing logic;

[0043] The server analyzes the data under running status and performs system operation status detection on the terminal device system that has executed the application management business process processing logic to obtain the detection result of the terminal device system. If the running status of the terminal device system meets the preset value, the terminal device system is running normally.

[0044] The runtime space analysis unit 105, the server side performs data analysis and processing based on the detection results of the terminal device system under normal operation, to obtain the maximum amount of RPA general data to be collected.

[0045] The server performs data analysis and processing based on the terminal device system detection results under normal operating conditions to obtain the maximum amount of general RPA data to be collected. By collecting operating data from the terminal device running RPA, the maximum load capacity of the terminal device is determined. During the RPA data capture process, the operation of other application software on the terminal device is paused, and the data generated by the application software during operation is stored in the cloud server and marked with time. When the upper limit of the running space is reached during the RPA data capture process, RPA is turned off, the paused application is closed, and it is calculated whether the expected remaining running space of the terminal device is sufficient for the normal operation of RPA data capture. If not, the workload of RPA data capture is adjusted to avoid the terminal computer system crashing due to excessive memory occupation by RPA data capture.

[0046] Further, please refer to Figure 2 The evaluation and detection unit includes:

[0047] S201, the server side collects the terminal device's RPA data collection parameters in the terminal device's RPA data collection state, including the number of application software running on the terminal device and the percentage of running memory space occupied by the application software.

[0048] S202, Construct the RPA robot time-segment data acquisition business process, and collect the acquisition parameters in the RPA data acquisition state in time segments;

[0049] For the data collection time of RPA robot data collection, a time-segmented method is adopted to collect the remaining operating space of the terminal device. This allows for real-time monitoring of the remaining operating space of the terminal device and enables adaptive decision-making.

[0050] S203, if the acquisition parameters in the RPA data acquisition state are abnormal, the RPA robot data acquisition task is paused and an error report is sent to the backend control terminal.

[0051] If a system error occurs during the RPA data acquisition process, the RPA robot's data acquisition task will be paused. The erroneous data will then be sent to the backend control panel, where administrators will process the data and repair the system.

[0052] Further, please refer to Figure 3 The logical building unit includes:

[0053] S301, after the RPA data capture task is started, a static and constant safe operating memory limit is established based on the existing terminal device's operating space memory size;

[0054] The static constant safe operating memory limit is the maximum remaining operating memory space that ensures application software on the terminal device can run normally.

[0055] S302: After the RPA data capture task is run, the running memory of the application software in the running state other than the RPA data capture task is stored to the cloud server, and the running of the application software in the running state other than the RPA data capture task is paused.

[0056] S303, when RPA reaches the critical point, it transmits the time node record data to the cloud storage space corresponding to the application software, establishes a shutdown time node, and shuts down the application software.

[0057] When the RPA data capture process reaches the upper limit of the running space, the RPA is shut down and the paused application is closed. The remaining running space of the terminal device is calculated to determine whether it is sufficient for the normal operation of RPA data capture. If not, the workload of RPA data capture is adjusted to avoid the terminal computer system crashing due to excessive memory usage by RPA data capture.

[0058] Further, please refer to Figure 4 The operation status analysis unit includes:

[0059] S401, the server performs system operation detection on the terminal device system that has executed the application management business process processing logic, and obtains the terminal device system detection result;

[0060] It is necessary to monitor the data processing workflow in real time to ensure it is running smoothly and to prevent errors from occurring. It is also necessary to organize, summarize, and store the test results of the terminal equipment system.

[0061] S402, if the operating status of the terminal device system meets the preset value, then the terminal device system is operating normally.

[0062] Further, please refer to Figure 5 The operating space analysis unit includes:

[0063] S501, the data analysis and processing includes the terminal device's running memory space data, the terminal device's hardware temperature data, and the terminal device's expected load capacity data within a certain time period.

[0064] S502, Evaluate the content occupied by the terminal device during the expected time period of RPA data collection, and obtain the expected time period RPA data collection operation space requirement data;

[0065] S503, compare the expected RPA data collection and operation space requirement data with the expected load capacity data of the terminal device during the expected time period. If the data comparison result is abnormal, the information is fed back to the background control terminal.

[0066] As can be seen from the above embodiments, the general data acquisition system based on RPA robots provided by the present invention includes: a terminal device, a server, and a backend control terminal. The terminal device is connected to the backend control terminal through the server. The server collects the system operation status parameters of the terminal device in the RPA data acquisition state. The server inputs the system operation status parameters of the terminal device in the RPA data acquisition state into the terminal system operation status evaluation model to obtain the terminal device system operation status result. Based on the terminal device system operation status result, the server constructs an application management business process processing logic database through RPA and executes the application management business process processing logic. The server performs system operation status detection and data analysis on the terminal device system that has executed the application management business process processing logic. The server-side performs data analysis and processing based on the terminal device system detection results under normal operating conditions to obtain the maximum general data collection volume for RPA. By collecting operational data from the terminal device running RPA, the maximum load capacity of the terminal device is determined. During RPA data capture, the operation of other application software on the terminal device is paused, and the data generated during application software operation is stored on the cloud server and time-marked. When the upper limit of the running space is reached during RPA data capture, RPA is shut down, the paused applications are closed, and it is calculated whether the expected remaining running space of the terminal device is sufficient for the normal operation of RPA data capture. If not, the workload of RPA data capture is adjusted to avoid the terminal computer system crashing due to excessive RPA data capture consuming too much running memory. This solves the problem that existing RPA robot data capture settings, which result in computer terminal failure under RPA robot overload conditions, prevent the RPA robot from properly capturing data.

[0067] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0068] The embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention.

Claims

1. A general-purpose data acquisition system based on RPA robots, characterized in that, It includes: a terminal device, a server, and a back-end control terminal, wherein the terminal device is connected to the back-end control terminal through the server; The data acquisition unit, wherein the server acquires the data capture system operating status parameters of the terminal device in the RPA data acquisition state; The evaluation and detection unit, the server side, inputs the system operation status parameters of the terminal device in the RPA data acquisition state into the terminal system operation status evaluation model to obtain the terminal device system operation status result; The logical construction unit is a server that constructs an application management business process processing logic database based on the system operating status results of the terminal device through RPA, and executes the application management business process processing logic. The operation status analysis unit, the server side performs system operation status detection and data analysis on the terminal device system that has executed the application management business process processing logic; The server-side operation space analysis unit performs data analysis and processing based on the detection results of the terminal device system under normal operation, and obtains the maximum amount of general RPA data to be collected. The logical building unit includes: After the RPA data capture task is started, a static and constant safe operating memory limit is established based on the existing terminal device's operating space memory size; After the RPA data crawling task is run, the running memory of the application software in the running state other than the RPA data crawling task is stored to the cloud server, and the running of the application software in the running state other than the RPA data crawling task is paused. When RPA reaches the critical point, it transmits the time node record data to the cloud storage space corresponding to the application software, establishes a shutdown time node, and shuts down the application software. The runtime space analysis unit includes: The data analysis and processing includes the terminal device's running memory space data, the terminal device's hardware temperature data, and the terminal device's expected load capacity data over a period of time. The operational content of the terminal device occupied during the expected time period of RPA data collection is evaluated to obtain the expected RPA data collection operational space requirement data for the time period. The RPA data collection and operation space requirement data for the expected time period is compared with the load capacity data of the terminal device within the expected time period. If the data comparison result is abnormal, the information is fed back to the background control terminal.

2. The system as described in claim 1, characterized in that, The evaluation and detection unit includes: The server will collect parameters from the terminal device in RPA data collection state, including the number of application software running on the terminal device and the percentage of memory space occupied by the application software. Construct an RPA robot time-segment data collection workflow to collect collection parameters in different time periods under the RPA data collection state; If any abnormal parameter occurs during the RPA data acquisition process, the RPA robot data acquisition task will be paused, and an error report will be sent to the backend control terminal.

3. The system as described in claim 1, characterized in that, The operational status analysis unit includes: The server performs system operation detection on the terminal device system that has executed the application management business process processing logic, and obtains the detection result of the terminal device system. If the operating status of the terminal device system meets the preset value, then the terminal device system is operating normally.

Images