A data processing method and device based on RPA system
By acquiring and classifying RPA operation behaviors, marking and replaying key operations, the problems of high learning costs and unstable use of testers are solved, and stable automated recording and playback of RPA products are achieved, improving user experience.
Patent Information
- Application Number
- CN202211245462.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-10-12
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-10-12
AI Technical Summary
Testers need to learn multiple sets of different product rules when using RPA products, which leads to high learning costs and difficult to use them stably, prone to confusion of rules, and it is difficult to ensure stability during RPA recording and playback.
By obtaining the data flow for recording and listening operation behavior, determining and classifying key operation behaviors, generating a second data flow, and marking and playing back the operation behaviors according to preset marking characteristics and time intervals, ensuring the accuracy and stability of the operation behaviors. Provides operation prompt window and device interface parameter mapping function, supporting cross-device playback.
It reduces learning costs, improves the user experience of RPA products, ensures the stability of recording and playback, avoids confusion of rules, and enhances the user's friendliness to RPA products.
Smart Images

Figure CN115599375B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of robotic process automation technology, and in particular to a data processing method and device based on an RPA system. Background Art
[0002] Robotic Process Automation (RPA) tools are often used in the field of automated testing. Testers use the RPA platform to design repeatable test processes. With the development of office automation, RPA tools have gradually covered financial management, supply chain management, human resource management and other aspects. RPA is committed to freeing manpower from some repetitive and regular work, and realizing the execution, processing and judgment of these tasks and processes through automated programs, so as to achieve the purpose of reducing costs and increasing efficiency.
[0003] However, the inventors found that when testers use the RPA platform, they often face RPA products with different product rules. Therefore, testers need to learn multiple sets of product rules, which has a high learning cost. Moreover, when testers learn complicated product rules, there is a high probability of rule confusion, and it is difficult to use RPA products stably when recording and replaying RPA. Summary of the invention
[0004] The embodiments of the present application provide a data processing method and device based on an RPA system, which are used to solve the problem that the learning cost of testers using RPA products is high and it is difficult for testers to use RPA products stably.
[0005] On the one hand, an embodiment of the present application provides a data processing method based on an RPA system, the method comprising:
[0006] Obtain a first data stream of recorded monitoring operation behaviors. The recorded monitoring operation behaviors are arranged according to a preset operation timing stream. Determine key operation behaviors corresponding to the first data stream. And classify each key operation behavior. Generate a second data stream based on the first data stream and each classified key operation behavior. Parse the second data stream to play back the recorded monitoring operation behavior.
[0007] In one implementation of the present application, several key behaviors in the first database are determined. The key behaviors include at least: mouse hover, mouse drag, and keyboard combination keys. According to each key behavior, the corresponding recording and monitoring operation behavior in the first data stream is matched. And according to a preset marking feature, the corresponding recording and monitoring operation behavior is marked, and the key operation behavior is determined. Among them, the marking feature includes at least one or more of the following: image position feature, image feature.
[0008] In one implementation of the present application, the duration of each key operation behavior is determined. When the duration of the behavior is less than a first preset threshold, the corresponding key operation behavior is used as the first key behavior. When the duration of the behavior is greater than or equal to the first preset threshold and less than the second preset threshold, the corresponding key operation behavior is used as the second key behavior. When the duration of the behavior is greater than or equal to the second preset threshold, the corresponding key operation behavior is used as the third key behavior.
[0009] In one implementation of the present application, a plurality of data record arrays corresponding to the second data stream are determined according to each key operation behavior and each recording and monitoring operation behavior. Each data record array is arranged according to the operation timing flow to generate a second data stream corresponding to the first data stream. The data storage format of the second data stream is the JSON data format.
[0010] In one implementation of the present application, each recording and monitoring operation behavior corresponding to the data record array includes at least: an input type identifier, a Boolean identifier, and an operation timing flow identifier. Each key operation behavior corresponding to the data record array also includes: an image position feature identifier and an impact feature identifier.
[0011] In one implementation of the present application, the recorded monitoring operation behaviors that meet the preset rules in the second data stream are eliminated. The preset rules include at least a continuous mouse movement input type, a predetermined drag input type, and a predetermined combination key input type. According to the data record array of the second data stream after the elimination process, the corresponding recorded monitoring operation behavior is restored to play back the restored recorded monitoring operation behavior according to the operation timing flow. The restoration at least includes determining the input type and the corresponding image position of the recorded monitoring operation behavior to re-execute the recorded monitoring operation behavior according to the input type and the image position.
[0012] In one implementation of the present application, non-critical operation behaviors in the restored recorded monitoring operation behaviors are determined, and the first time interval of each non-critical operation behavior is matched. The first time interval is the duration of performing non-critical operation behaviors during playback. Determine the classification results of each critical operation behavior. In the case where the critical operation behavior is the first critical behavior, match the second time interval of the first critical behavior. Wherein, the second time interval is less than the first preset threshold. In the case where the critical operation behavior is the second critical behavior, match the third time interval of the second critical behavior. Wherein, the third time interval is the duration of the behavior of the second critical behavior. In the case where the critical operation behavior is the third critical behavior, match the fourth time interval of the third critical behavior. Wherein, the fourth time interval is the second preset threshold. According to the corresponding playback duration of the recorded monitoring operation behavior, the restored recorded monitoring operation behavior is played back. The playback duration includes at least: the first time interval, the second time interval, the third time interval, and the third time interval.
[0013] In one implementation of the present application, when replaying the recorded monitoring operation behavior, determine whether the image position features of each key operation behavior match the image position features of the current playback. Wherein, the matching includes image similarity matching and operation position matching. If they match, determine through a preset algorithm whether the SSIM value of the image features of the key operation behavior and the image features of the current playback is greater than a preset value. If so, replay the recorded monitoring operation behavior. If not, stop replaying the recorded monitoring operation behavior, and generate playback prompt information. The playback prompt information includes at least one or more of the following: text, sound.
[0014] In one implementation of the present application, when a recorded listening operation behavior is obtained, a corresponding operation prompt window is generated, and the operation prompt window is sent to a user terminal that triggers the recorded listening operation behavior. Among them, the operation prompt window includes at least: current operation behavior prompt information, shortcut key input prompt information. Before replaying the recorded listening operation behavior, the method also includes: determining the device interface parameters for replaying the recorded listening operation behavior. When the device interface parameters do not match the terminal interface parameters of the user terminal of the recorded listening operation behavior, a mapping function between the device interface parameters and the terminal interface parameters is determined. The mapping function is a function of the mapping relationship between the terminal interface parameters and the device interface parameters. The mapping function is pre-stored in a second database. The terminal interface parameters corresponding to the second data stream are input into the mapping function to process the second data stream so as to replay the processed recorded listening operation behavior.
[0015] On the other hand, an embodiment of the present application further provides a data processing device based on an RPA system, the device comprising:
[0016] At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:
[0017] Obtain a first data stream of recorded monitoring operation behaviors. The recorded monitoring operation behaviors are arranged according to a preset operation timing stream. Determine key operation behaviors corresponding to the first data stream. And classify each key operation behavior. Generate a second data stream based on the first data stream and each classified key operation behavior. Parse the second data stream to play back the recorded monitoring operation behavior.
[0018] Through the above scheme, this application marks the key operation behavior of the corresponding recording monitoring operation behavior, generates a second data stream, and then replays the recorded operation behavior, which can avoid the high learning cost of testers when using RPA products due to different product rules. And by accurately marking key operation behaviors, it is ensured that key behaviors are not missed during recording, so that this application can perform stable RPA automated recording and playback, improving the user experience of RPA products. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0020] Figure 1 This is a flow chart of a data processing method based on an RPA system in an embodiment of the present application;
[0021] Figure 2 This is another flowchart of a data processing method based on an RPA system in an embodiment of the present application;
[0022] Figure 3 This is a structural diagram of a data processing device based on an RPA system in an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.
[0024] At present, the implementation of a reliable RPA process usually involves the following steps: selecting an RPA design platform; learning RPA components and process design rules; using RPA components to combine complete task processes; and executing task processes to complete specific tasks.
[0025] When testers use the RPA platform to design automated processes, they need to learn a set of product rules. Based on different task processes, testers, i.e. process designers, are required to master the relevant basic technologies of the RPA platform, which is less friendly to cross-industry personnel.
[0026] When recording and replaying the RPA platform, there is a mismatch between the operations during recording and playback, making it difficult to use the RPA platform stably. Testers are forced to redesign product processes in real time, reducing their product experience.
[0027] Based on this, the embodiments of the present application provide a data processing method and device based on the RPA system, which are used to solve the high learning cost for testers to use RPA products and the difficulty in using RPA products stably, thereby improving the testers' experience of using the product.
[0028] The following describes in detail various embodiments of the present application in conjunction with the accompanying drawings.
[0029] The present application embodiment provides a data processing method based on an RPA system, such as Figure 1 As shown, the method may include steps S101-S104:
[0030] S101, the server obtains a first data stream recording a monitoring operation behavior.
[0031] Among them, the recording and monitoring operation behaviors are arranged according to the preset operation timing flow.
[0032] It should be noted that the server, as the executor of the data processing method based on the RPA system, is only an example. The executor is not limited to the server, and this application does not make any specific limitations on this.
[0033] In the embodiment of the present application, the RPA system performs business process recording and playback, including two processes of recording and playback, and the two processes interact through a second data stream (RPA data stream) composed of operation time sequence stream sorting. This method ensures that the playback process completely restores the various operations performed by the recording to achieve the purpose of automated execution.
[0034] Among them, recording and playback are two independent processing processes. Recording is used as an input item to record all operations during the recording process, and it is also used as a generated item to form an RPA data flow. Replay is an execution process, which is the parsing and execution of the RPA data flow. Through the RPA data flow, playback is repeatable and stable. From the specific implementation, it also includes recording function design, operation sequence flow, key process marking, RPA data flow, playback function design, interactivity optimization, universality optimization and other processes. Among them, interactivity optimization and universality optimization are considered as optimization solutions.
[0035] The recording monitoring operation behavior of this application is all the input types generated during the captured recording process. The input is generally generated by external devices such as mouse and keyboard. For example, a mouse click is a recording monitoring operation behavior, and the server uses this behavior as the first data stream of the mouse click. In addition, during the recording process, the server classifies and processes different inputs. These basic input types include: mouse movement, mouse pause, left and right click of the mouse, double click of the left and right mouse buttons, mouse wheel scrolling, keyboard pressing and releasing. Based on the basic input types, there are also types such as mouse movement trajectory, mouse dragging, and keyboard combination key input.
[0036] This application adopts the method of monitoring all input types to obtain each first data stream, that is, the server completely retains all operation behaviors in the recording process, thereby providing a pre-stability guarantee for the subsequent playback process analysis.
[0037] In addition, the operation sequence flow refers to the user operation behavior for the recording process. First, the user operation behavior process recorded is in time order and has a specific time series. There is a time interval between each independent user operation behavior and the previous and subsequent actions. For example, the operation behavior before the mouse click is the mouse sliding, and the operation behavior after the mouse click is the mouse staying. These specific operation behaviors form the operation sequence flow.
[0038] The setting of the operation sequence flow ensures the restoration of some operation behavior waiting actions while ensuring the recording of complete user operation behaviors, thus ensuring the robustness of the recording and monitoring operation behaviors in specific recording scenarios.
[0039] S102: The server determines key operation behaviors corresponding to the first data stream, and classifies each key operation behavior.
[0040] In the embodiment of the present application, the server determines the key operation behavior corresponding to the first data stream, specifically including:
[0041] The server determines several key behaviors in the first database. The key behaviors include at least: mouse hover, mouse drag, and keyboard combination keys. Then, according to each key behavior, the corresponding recording and monitoring operation behavior in the first data stream is matched. And according to the preset marking feature, the corresponding recording and monitoring operation behavior is marked, and the key operation behavior is determined. The marking feature includes at least one or more of the following: image position feature and image feature.
[0042] The key behavior can be any operation behavior selected from: mouse movement, mouse pause, left and right click of the mouse, double click of the left and right mouse button, mouse wheel scrolling, keyboard pressing and releasing, mouse movement trajectory, mouse dragging, and keyboard key combination.
[0043] When the server obtains the recorded monitoring operation behavior that matches the key behavior, the behavior can be marked. The mark can have image position features and image features corresponding to the behavior. The image position features include the location information generated by the key behavior, such as the UI image of the mouse click, and the coordinates of the click position relative to the current display interface; the image feature is an image record of the location where the key behavior is generated.
[0044] Combining the image and position into image position features can avoid the situation where the same image exists in multiple places on the interface and the mouse clicks on an image but cannot distinguish the clicked images. Combined with the position coordinates, the same images can be distinguished to ensure the accuracy of key process marking.
[0045] In addition, the server classifies each key operation behavior, including:
[0046] The server determines the duration of each key operation behavior. When the duration of the behavior is less than the first preset threshold, the corresponding key operation behavior is used as the first key behavior. When the duration of the behavior is greater than or equal to the first preset threshold and less than the second preset threshold, the corresponding key operation behavior is used as the second key behavior. When the duration of the behavior is greater than or equal to the second preset threshold, the corresponding key operation behavior is used as the third key behavior.
[0047] That is to say, the server is pre-set with two time thresholds, such as the first preset threshold t a , the second preset threshold t b , the server compares the duration of the behavior with t a , t b Compare and distinguish different key operation behaviors. For example, if the duration of the first key behavior is slip, the duration of the second key behavior is hover, and the duration of the third key behavior is sleep, then their relationship is as follows:
[0048] slip <t a ≤hover< b ≤sleep.
[0049] S103: The server generates a second data stream according to the first data stream and the classified key operation behaviors.
[0050] In the embodiment of the present application, the server generates a second data stream according to the first data stream and each classified key operation behavior, specifically including:
[0051] First, the server determines a plurality of data record arrays corresponding to the second data stream according to each key operation behavior and each recording and monitoring operation behavior.
[0052] Next, the server arranges each data record array according to the operation sequence flow to generate a second data stream corresponding to the first data stream. The data storage format of the second data stream is the JSON data format.
[0053] The second data stream is an RPA data stream, which is a result file generated after recording the process record, operation sequence flow and key operation behavior marking corresponding to the recording monitoring operation behavior. It is used as the input of the subsequent playback process. The data record array consists of multiple operation behavior records in JSONObject format. The operation behavior record in the data record array, that is, the recording monitoring operation behavior, at least includes: input type identifier, Boolean identifier, and operation sequence flow identifier. Each key operation behavior corresponding to the data record array also includes: image position feature identifier and impact feature identifier.
[0054] That is to say, the key operation behaviors in the data record array include: input type identifier, Boolean identifier, operation timing flow identifier, image position feature identifier, and impact feature identifier.
[0055] S104: The server parses the second data stream to play back the recorded monitoring operation behavior.
[0056] The server can perform playback operations on the second data stream, including: operation record compression and restoration, time sequence recovery, anti-interference, and key process verification. The specific process of operation record compression and restoration is as follows:
[0057] The server removes the recorded monitoring operation behaviors that meet the preset rules in the second data stream. The preset rules at least include a continuous mouse movement input type, a predetermined drag input type, and a predetermined combination key input type.
[0058] The preset rules are set by the user and this application does not make any specific restrictions on this.
[0059] The server restores the corresponding recorded monitoring operation behavior according to the data record array of the second data stream after the elimination process, so as to play back the restored recorded monitoring operation behavior according to the operation timing stream. The restoration at least includes determining the input type and the corresponding image position of the recorded monitoring operation behavior, so as to re-execute the recorded monitoring operation behavior according to the input type and the image position.
[0060] Among them, restoration is to restore the recorded monitoring operation behavior retained after the elimination process in the second data stream. For example, if the recorded monitoring operation behavior is a mouse click on a UI icon, then restoration can execute the recorded monitoring operation behavior during playback. Action restoration is to restore different input types, input positions and other elements in the operation behavior record.
[0061] The chronological restoration specifically includes: determining non-critical operation behaviors in the restored recording monitoring operation behaviors, and matching the first time intervals of each non-critical operation behavior. The first time interval is the duration of executing the non-critical operation behavior during playback. Then, determining the classification results of each critical operation behavior.
[0062] Among them, when the critical operation behavior is the first critical behavior, matching the second time interval of the first critical behavior. Among them, the second time interval is less than the first preset threshold.
[0063] When the critical operation behavior is the second critical behavior, matching the third time interval of the second critical behavior. Among them, the third time interval is the behavior duration of the second critical behavior.
[0064] When the critical operation behavior is the third critical behavior, matching the fourth time interval of the third critical behavior. Among them, the fourth time interval is the second preset threshold.
[0065] According to the playback duration corresponding to the recording monitoring operation behavior, performing playback on the restored recording monitoring operation behavior. The playback duration at least includes: the first time interval, the second time interval, the third time interval, the third time interval.
[0066] In other words, the chronological restoration is to maintain the correct playback execution of each recording monitoring operation behavior, and the restored content is the processed time interval. In the chronological restoration of non-critical operation behaviors, restoring the corresponding second data stream to the specified time interval, that is, the first time interval, and the first time interval is set according to actual needs, and the present application does not make specific limitations on this; when performing chronological restoration on critical operation behaviors, according to the marked slip, hover, sleep, different chronological restorations are performed respectively.
[0067] For the slip type, it is defaulted that no additional actions will occur during this duration, and select to restore this duration to a fixed shorter time, the second time interval t s ( s <a). The processing of the hover type usually generates additional actions, such as the display of a pop-up menu, the adsorption of draggable components, the interruption of continuous actions, etc. These additional actions are often related to the next input operation. In the selection of chronological restoration, the same time restoration is performed on the duration of the hover type, that is, the third time interval. The same time restoration means that when the recording time is a during the recording monitoring operation behavior, then the chronological restoration is a. The duration of the sleep type is long and there is redundant waiting, and the time restoration of this type is specified as a fixed time t b .
[0068] In another embodiment of the present application, the server can perform anti-interference operation during playback. Anti-interference is mainly aimed at additional input interference during playback, and the generation of additional input is regarded as an interruption. When an interruption is triggered, the playback operation pauses the reading and execution of the RPA data stream and marks the pause position. The anti-interference implementation generates a set of interference operation records for the additional input, and restores the state at the pause mark based on the current state, the pause mark and the interference operation record.
[0069] That is to say, during playback, if the page has interference that did not appear during recording and monitoring operation behavior but appeared during playback, for example, the mouse clicked a certain UI icon x during recording and monitoring operation behavior, and the UI icon x was replaced by another UI image y during playback, then this behavior is an additional input interference, playback is paused, and the interference is marked. After that, a set of interference operation behavior records will be input, and according to the current state during playback, the pause mark, and the interference operation behavior records, the state at the time of the pause mark will be restored.
[0070] Additionally, key process verification is performed during playback, including:
[0071] When the recorded monitoring operation behavior is played back, it is determined whether the image position features of each key operation behavior match the image position features of the current playback, wherein the matching includes image similarity matching and operation position matching.
[0072] When the image position features of each key operation behavior match the image position features of the current playback, a preset algorithm is used to determine whether the SSIM value of the image features of the key operation behavior and the image features of the current playback is greater than a preset value. The preset value is, for example, 0.85, and this application does not specifically limit the specific value of the preset value.
[0073] SSIM combines brightness contrast l(x,y), contrast contrast c(x,y) and structure contrast s(x,y) to measure image features, namely the quality distortion and image structure changes of the two images. The specific formula of the preset algorithm is as follows:
[0074] SSIM(x,y)=[l(x,y)] α [c(x,y)] β ·[s(x,y)] γ
[0075] Among them, α, β, γ>0, which is used to adjust the importance of the three modules, and x, y are the coordinates of the image pixels.
[0076] When the image position features of each key operation behavior do not match the image position features of the current playback, playback error information is generated and sent to the user terminal. The user terminal may be a mobile phone, computer, tablet computer or other device, which is not specifically limited in this application.
[0077] When it is determined that the SSIM value of the image feature of the key operation behavior and the image feature of the current playback is greater than a preset value, the recorded monitoring operation behavior is played back.
[0078] When it is determined that the SSIM value of the image feature of the key operation behavior and the image feature of the current playback is not greater than the preset value, the playback of the recorded monitoring operation behavior is stopped, and playback prompt information is generated. The playback prompt information includes at least one or more of the following: text, sound.
[0079] The playback prompt information may be a prompt that the playback image quality is too low and does not meet the playback requirements.
[0080] In one embodiment of the present application, the method further includes:
[0081] When the recording and monitoring operation behavior is obtained, a corresponding operation prompt window is generated and sent to the user terminal that triggers the recording and monitoring operation behavior. The operation prompt window at least includes: current operation behavior prompt information and shortcut key input prompt information.
[0082] That is to say, during the recording process, prompts are given for the operation behavior, and the corresponding operation prompt window and the user terminal interface of the operation recording behavior are displayed, so that the current status, shortcut keys, etc. are continuously displayed during the recording process. In this way, the operation behavior, such as the mark of the key operation behavior, is visually displayed to the user, which enhances the user's experience during recording and improves the accuracy of the recording process. It can avoid the need to learn different product rules before using RPA products, so that users can use RPA products stably.
[0083] In addition, before replaying the recorded monitoring operation behavior, the method further includes:
[0084] The server can determine the device interface parameters for playing back the recorded monitoring operation behavior.
[0085] In the case where the device interface parameters do not match the terminal interface parameters of the user terminal for recording the monitoring operation behavior, a mapping function between the device interface parameters and the terminal interface parameters is determined. The mapping function is a function of the mapping relationship between the terminal interface parameters and the device interface parameters. The mapping function is pre-stored in the second database. Thus, the terminal interface parameters corresponding to the second data stream are input into the mapping function to process the second data stream so as to play back the processed recorded monitoring operation behavior.
[0086] That is, by matching the mapping function in the second database, the mapping function is used to map the screen during recording to the screen during playback. Specifically, if the recording is performed on a screen with a first display resolution and the playback is performed on another device with a second display resolution, by matching the first display resolution with the second display resolution, the mapping function in the second database can map the second data stream of the first display resolution recorded to the second display resolution, thereby ensuring normal playback of the second data stream.
[0087] This enables universal playback across devices and universal sharing among users. Cross-device playback mainly solves differences in interface scaling and display resolution between different devices. Interface scaling parameters can cause expansion or compression of input position information, and display resolution differences can cause offset of input position information. By associating interface scaling and display resolution to process part of the input in the RPA data stream, the correct input mapping between devices is obtained to ensure cross-device playback stability. The improvement of universality of sharing among users usually also depends on the input mapping processing between devices. Here, it is assumed that in most cases, sharing among users is the sharing of RPA data streams on different user devices. At this time, universality processing is used as a detection process to detect whether the playback device migrated to has the playback conditions.
[0088] The above solution can solve the problem that the learning cost of RPA products is high and it is difficult for testers to use RPA products stably.
[0089] Another schematic diagram of the data processing method based on the RPA system in this application is as follows Figure 2 As shown, during the playback process, it is necessary to perform operation record compression and restoration, timing recovery, anti-interference, key process verification, and determine whether there are operation records that need to be restored, and then perform playback.
[0090] Figure 3 A schematic diagram of a data processing device based on an RPA system provided in an embodiment of the present application, the device includes:
[0091] At least one processor; and a memory in communication with the at least one processor. The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:
[0092] Obtain a first data stream of recorded monitoring operation behaviors. The recorded monitoring operation behaviors are arranged according to a preset operation timing stream. Determine key operation behaviors corresponding to the first data stream. And classify each key operation behavior. Generate a second data stream based on the first data stream and each classified key operation behavior. Parse the second data stream to play back the recorded monitoring operation behavior.
[0093] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0094] The device and method provided in the embodiments of the present application correspond one to one, and therefore, the device also has similar beneficial technical effects as its corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device will not be repeated here.
[0095] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.
[0096] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.
Claims
1. A data processing method based on an RPA system, characterized in that: The method comprises: Acquire a first data stream of recorded monitoring operation behaviors; wherein the recorded monitoring operation behaviors are arranged according to a preset operation timing stream; Determining key operation behaviors corresponding to the first data stream; and classifying each of the key operation behaviors; Generate a second data stream according to the first data stream and the classified key operation behaviors; Parsing the second data stream to play back the recorded monitoring operation behavior; The step of determining the key operation behavior corresponding to the first data flow specifically includes: Determine a number of key behaviors in the first database; the key behaviors at least include: mouse hover, mouse drag, keyboard combination key; According to each of the key behaviors, matching the corresponding recording and monitoring operation behaviors in the first data stream; and According to the preset marking features, the corresponding recording and monitoring operation behaviors are marked, and the key operation behaviors are determined; wherein the marking features include at least one or more of the following: image position features, image features; Specifically, they include: Determine the duration of each of the key operating behaviors; When the duration of the behavior is less than a first preset threshold, the corresponding key operation behavior is used as a first key behavior; When the duration of the behavior is greater than or equal to the first preset threshold and less than the second preset threshold, the corresponding key operation behavior is used as the second key behavior; When the duration of the behavior is greater than or equal to the second preset threshold, taking the corresponding key operation behavior as the third key behavior; The second data stream is generated according to the first data stream and the classified key operation behaviors, specifically including: Determine a plurality of data record arrays corresponding to the second data stream according to each of the key operation behaviors and each of the recording and monitoring operation behaviors; Arrange the data record arrays according to the operation sequence flow to generate the second data stream corresponding to the first data stream; the data storage format of the second data stream is the JSON data format; Among them, the second data stream is replayed, including operation record compression and restoration, time sequence recovery, anti-interference, and key process verification; The specific process of parsing the second data stream to play back the recorded monitoring operation behavior and compressing and restoring the operation record includes: Eliminate the recording monitoring operation behaviors that meet preset rules in the second data stream; the preset rules at least include a continuous mouse movement input type, a predetermined drag input type, and a predetermined combination key input type; According to the data record array of the second data stream after the elimination process, the corresponding recorded monitoring operation behavior is restored, so as to play back the restored recorded monitoring operation behavior according to the operation timing flow; the restoration at least includes determining the input type and the corresponding image position of the recorded monitoring operation behavior, so as to re-execute the recorded monitoring operation behavior according to the input type and the image position; The restored recorded monitoring operation behavior is played back, and the time sequence recovery specifically includes: Determine the non-critical operation behaviors in the restored recorded monitoring operation behaviors, and match the first time interval of each of the non-critical operation behaviors; the first time interval is the duration of executing the non-critical operation behaviors during playback; Determine the classification results of each of the key operational behaviors; When the key operation behavior is a first key behavior, matching a second time interval of the first key behavior; wherein the second time interval is less than a first preset threshold; In the case where the key operation behavior is a second key behavior, matching a third time interval of the second key behavior; wherein the third time interval is the duration of the second key behavior; In the case where the key operation behavior is the third key behavior, matching a fourth time interval of the third key behavior; wherein the fourth time interval is a second preset threshold; Playing back the restored recorded monitoring operation behavior according to the corresponding playback duration of the recorded monitoring operation behavior; the playback duration at least includes: the first time interval, the second time interval, the third time interval, and the fourth time interval; Among them, anti-interference operation is performed during playback, and the anti-interference is aimed at additional input interference during playback, and the generation of additional input is regarded as an interruption; when the interruption is triggered, the playback operation pauses the reading and execution of the RPA data stream and marks the pause position; the anti-interference implementation generates a set of interference operation records for the additional input, and restores the state at the pause mark according to the current state, the pause mark and the interference operation record; Wherein, key process verification is performed during the playback process, and the method further comprises: When replaying the recorded monitoring operation behavior, determining whether the image position feature of each of the key operation behaviors matches the image position feature of the current playback; wherein the matching includes image similarity matching and operation position matching; If they match, determine, through a preset algorithm, whether the SSIM value of the image feature of the key operation behavior and the image feature of the current playback is greater than a preset value; If so, replay the recorded monitoring operation behavior; If not, stop replaying the recorded monitoring operation behavior and generate playback prompt information; the playback prompt information includes at least one or more of the following: text, sound.
2. The method according to claim 1, characterized in that: Each of the recording and monitoring operation behaviors corresponding to the data record array at least includes: an input type identifier, a Boolean identifier, and an operation timing flow identifier; Each of the key operation behaviors corresponding to the data record array further includes: an image position feature identifier and an image feature identifier.
3. The method according to claim 1, characterized in that: The method further comprises: In the case of obtaining the recording and monitoring operation behavior, a corresponding operation prompt window is generated, and the operation prompt window is sent to the user terminal that triggers the recording and monitoring operation behavior; wherein the operation prompt window at least includes: current operation behavior prompt information, shortcut key input prompt information; Before playing back the recorded monitoring operation behavior, the method further includes: Determine the device interface parameters for playing back the recorded monitoring operation behavior; In the case where the device interface parameters do not match the terminal interface parameters of the user terminal recording the monitoring operation behavior, determining a mapping function between the device interface parameters and the terminal interface parameters; the mapping function is a mapping relationship function between the terminal interface parameters and the device interface parameters; the mapping function is pre-stored in the second database; The terminal interface parameters corresponding to the second data stream are input into the mapping function to process the second data stream so as to play back the processed recorded monitoring operation behavior.
4. A data processing device based on an RPA system, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a data processing method based on an RPA system as described in any one of claims 1 to 3 above.
