Anomaly detection method, device and equipment
By acquiring click interval data from the graphical system and calculating the similarity of screenshot feature maps, the accuracy problem of anomaly detection in the graphical system was solved, and more efficient anomaly detection results were achieved.
Patent Information
- Application Number
- CN202011472222.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-15
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2040-12-15
AI Technical Summary
In existing technologies, anomaly detection in graphical systems relies on manually setting time intervals for screenshots, leading to inaccurate detection results.
By acquiring multiple sets of click interval datasets for the same automated process in the graphical system, the screen capture time interval after each click operation is determined, and the similarity is calculated based on the feature map of the screen capture to determine the system status.
It improves the accuracy and efficiency of anomaly detection, reduces computational load, and accurately locates the time point of anomalies.
Smart Images

Figure CN112581444B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of automated testing, and in particular to an anomaly detection method, device and equipment. Background Art
[0002] With the rapid development of RPA (Robotic Process Automation) technology, in many scenarios, people's work is replaced by digital labor based on RPA technology. Since the stability of the graphical system is more important to users, the rapid response of the graphical system is an important prerequisite for ensuring user-friendliness. Therefore, when using RPA technology to replace people to do some work, it is necessary to consider the stability of the graphical system in the scenario where the technology itself is applied. However, due to reasons such as communication between the graphical system and the client and network fluctuations, the stability of the system cannot be fully guaranteed. For example, when clicking a button, a response timeout occurs, the page freezes, and other unstable factors occur. Therefore, anomaly detection of RPA processes developed by graphical systems becomes increasingly important.
[0003] In the prior art, screenshots of the graphical system are taken at fixed time intervals set manually, and then the similarity of the captured images is compared to determine whether the graphical system has an unresponsive abnormality. However, the capture time set based on human experience is prone to large errors from the actual scene, resulting in inaccurate abnormality detection results for the graphical system.
[0004] Therefore, the industry is in urgent need of a technical solution that can solve the above problems. Summary of the invention
[0005] The embodiments of this specification provide an anomaly detection method, apparatus, and device, which can make the screen capture time more accurate, thereby making subsequent anomaly detection results more accurate.
[0006] The anomaly detection method, device and equipment provided in this specification are implemented in the following ways.
[0007] An anomaly detection method includes: obtaining multiple groups of click interval data sets of the same automated process in a graphical system; wherein each group of click interval data sets includes a preset number of elements, and each element represents the time interval corresponding to adjacent click operations; determining the screen capture time interval after each click operation based on the elements of the same order in the multiple groups of click interval data sets; sequentially capturing the screen of the graphical system according to the screen capture time interval after each click operation to obtain screenshots corresponding to different screen capture times; calculating the similarity between the screenshots corresponding to adjacent screen capture times based on the feature graphs of the screenshots; and determining the state of the automated process when it is running in the graphical system based on the similarity.
[0008] An abnormality detection device includes: a click interval data acquisition module, which is used to acquire multiple groups of click interval data sets of the same automated process in a graphical system; wherein each group of click interval data sets includes a preset number of elements, and each element represents the time interval corresponding to adjacent click operations; a capture time interval determination module, which is used to determine the screen capture time interval after each click operation based on the elements of the same order in the multiple groups of click interval data sets; a screenshot acquisition module, which is used to sequentially capture the screen of the graphical system according to the screen capture time interval after each click operation, and obtain screenshots corresponding to different screen capture times; a similarity calculation module, which is used to calculate the similarity between the screenshots corresponding to adjacent screen capture times according to the feature graphs of the screenshots; and an operation status determination module, which is used to determine the state of the automated process when it is running in the graphical system based on the similarity.
[0009] An anomaly detection device includes a processor and a memory for storing processor executable instructions, wherein the processor implements the steps of any method embodiment in the embodiments of this specification when executing the instructions.
[0010] A computer-readable storage medium stores computer instructions, which, when executed, implement the steps of any method embodiment in the embodiments of this specification.
[0011] This specification provides an abnormality detection method, device and equipment. In some embodiments, multiple groups of click interval data sets of the same automated process in a graphical system can be obtained; wherein each group of click interval data sets includes a preset number of elements, each element represents the time interval corresponding to adjacent click operations; based on the elements of the same order in the multiple groups of click interval data sets, the screen capture time interval after each click operation is determined. Further, the screen of the graphical system is sequentially captured according to the screen capture time interval after each click operation to obtain screenshots corresponding to different screen capture times. Further, according to the feature graph of the screenshots, the similarity between the screenshots corresponding to the adjacent screen capture times is calculated, and based on the similarity, the state of the automated process when it is running in the graphical system is determined. Since the time of locating the screenshot in the embodiment of the present application is located in the form of a time interval, it is more accurate than the method of setting the screenshot time according to human experience in the prior art, so that the subsequent abnormality detection results can be more accurate. Since the embodiment of the present application can calculate the similarity between the screenshots corresponding to the adjacent screen capture times according to the feature graph of the screenshots, and then determine the state of the automated process when it is running in the graphical system based on the similarity, compared with the prior art of continuously taking screenshots and continuously calculating the similarity, the calculation amount is less and the efficiency is higher. By adopting the implementation scheme provided in this specification, the screen capture time can be made more accurate, thereby making the subsequent abnormality detection results more accurate. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] The drawings described herein are used to provide a further understanding of this specification, constitute a part of this specification, and do not constitute a limitation of this specification. In the drawings:
[0013] Figure 1 It is a flowchart of an embodiment of an abnormality detection method provided in this specification;
[0014] Figure 2 It is a schematic diagram provided in this specification for extracting a corresponding feature map from a screenshot;
[0015] Figure 3 It is a schematic diagram of the module structure of an embodiment of an abnormality detection device provided in this specification;
[0016] Figure 4 This is a hardware structure block diagram of an embodiment of an anomaly detection server provided in this specification. DETAILED DESCRIPTION
[0017] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments in this specification, not all of the embodiments. Based on one or more embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the embodiments of this specification.
[0018] The following is an example of a specific application scenario to illustrate the implementation plan of this specification. Specifically, Figure 1 It is a flowchart of an embodiment of an abnormality detection method provided in this specification. Although this specification provides method operation steps or device structures as shown in the following embodiments or drawings, the method or device may include more or fewer operation steps or module units after partial combination based on routine or no creative labor.
[0019] An implementation scheme provided in this specification can be applied to a client, a server, etc. The client can include a terminal device, such as a smart phone, a tablet computer, etc. The server can include a single computer device, or a server cluster composed of multiple servers, or a server structure of a distributed system, etc.
[0020] It should be noted that the following embodiment description does not limit the technical solutions in other application scenarios that can be extended based on this specification. Figure 1 As shown, in an embodiment of an abnormality detection method provided in this specification, the method may include the following steps.
[0021] S0: Acquire multiple groups of click interval data sets of the same automated process in the graphical system; wherein each group of click interval data sets includes a preset number of elements, and each element represents the time interval corresponding to adjacent click operations.
[0022] Among them, the automated process can also be called robotic process automation, which can replace or assist humans in completing repetitive work and tasks in digital devices such as computers and RPA phones. In some embodiments, RPA is usually based on a graphical system and operates based on an interface, so the stability of the graphical system is very important for RPA.
[0023] In some embodiments of this specification, multiple groups of click interval data sets of the same automated process in a graphical system can be obtained to provide a basis for subsequently determining the state of the RPA when it is running in the graphical system. Each group of click interval data sets may include a preset number of elements, and each element can be used to represent the time interval corresponding to adjacent click operations.
[0024] In some embodiments, the elements in each group of click interval data sets can be displayed in the form of a sequence, or in other forms such as a table, which is not limited in this specification. The number of the above-mentioned click interval data sets can be set according to the actual scenario, for example, it can be 100, 120, etc. The above-mentioned preset number can be set according to the actual scenario, for example, it can be 10, 15, etc., which is not limited in this specification. Among them, the number of elements included in each group of click interval data sets is the same. The elements of the same order in each group of click interval data sets have the same meaning. For example, the first element included in each group of click interval data sets can represent the time interval between the first click operation and the start of the graphical system (or when the graphical system process is detected), and the second element included in each group of click interval data sets can represent the time interval between the second click operation and the first click operation, etc. The above-mentioned click operation can be an operation of clicking the graphical system with a mouse.
[0025] In some embodiments, the step of obtaining multiple sets of click interval data sets for the same automated process in the graphical system may include: obtaining multiple click event data sets for the same automated process in the graphical system; and calculating the difference between adjacent data in each click event data set to obtain multiple sets of click interval data sets. Each click event data set includes the time corresponding to the click operation on the graphical system in sequence when the graphical system process is detected.
[0026] In some embodiments, the data in each of the above-mentioned click event data sets can be displayed in the form of a sequence, or in other forms such as a table, which is not limited in this specification. The number of the above-mentioned click event data sets can be set according to the actual scenario, for example, it can be 100, 120, etc. Usually the number of click event data sets is the same as the number of click event data sets. The number of time points included in each of the above-mentioned click event data sets is the same. The time points of the same sequence in each click event data set have the same meaning. For example, the first time point included in each click event data set can represent the time when the graphical system is started (or when the graphical system process is detected), the second time point included in each click event data set can represent the time of the first click operation, and the third time point included in each click event data set can represent the time of the second click operation, and so on.
[0027] In some embodiments, starting from detecting a graphical system process or starting a graphical system, the time point corresponding to each click operation may be recorded, and then the recorded time points may be stored in sequence to obtain a click event data set.
[0028] For example, in some implementation scenarios, the time when the graphical system is started (or the time when the graphical system process is detected) can be recorded as time_0, the time point when the mouse clicks the graphical system for the first time can be recorded as time_1, the time point when the mouse clicks the graphical system for the second time can be recorded as time_2, ..., and the time point when the mouse clicks the graphical system for the last time can be recorded as time_m. In this way, a click event data set can be obtained. Repeat the above steps n times to obtain n click event data sets. Among them, n>1, which can be set according to the actual scenario. Usually, n can be set to more than 100.
[0029] Further, after obtaining n click event data sets, the time intervals between adjacent click operations in the first click event data set can be calculated, such as calculating the time interval between the first click operation and starting the graphical system (or detecting the graphical system process), the time interval between the second click operation and the first click operation, ..., the time interval between the last click operation and the last click operation, thereby obtaining the first group of click interval data sets. Among them, the elements in the first group of click interval data sets can be recorded in sequence as: interv_11, interv_12, ..., interv_1m. Correspondingly, the time intervals between the data in the 2nd to n-1th click event data sets can be calculated and recorded, thereby obtaining n groups of click interval data sets. Among them, the elements in the nth group of click interval data sets can be recorded in sequence as: interv_n1, interv_n2, ..., interv_nm. The elements in other groups of click interval data sets can be recorded similarly, which will not be repeated here.
[0030] S2: Determine the screen capture time interval after each click operation based on the elements of the same sequence in the multiple groups of click interval data sets.
[0031] In an embodiment of the present specification, after obtaining multiple groups of click interval data sets for the same automated process in a graphical system, the screen capture time interval after each click operation can be determined based on the elements of the same order in the multiple groups of click interval data sets. Wherein, the screen refers to the screen corresponding to the graphical system, and the size of the screen captured during the entire process is the same. The screen capture time interval is the time interval from the click operation. For example, the screen capture time interval can be 2 seconds, 0.5 seconds, etc. It should be noted that since the number of elements included in each group of click interval data sets is the same, and the elements of the same order in different click interval data sets have the same meaning, the screen capture time interval after each click operation can be determined more accurately by using the elements of the same order in multiple groups of click interval data sets.
[0032] In some embodiments, determining the screen capture time interval after each click operation based on the elements of the same order in the multiple groups of click interval data sets may include: calculating the mean and standard deviation of the elements of the same order in the multiple groups of click interval data sets; determining the screen capture time interval after each click operation based on the mean and standard deviation of the elements of the same order.
[0033] In some embodiments, the screen capture time interval after each click operation can be determined according to the following method:
[0034] T=ave-3×std
[0035] Where T represents the screen capture time interval, ave represents the mean, and std represents the standard deviation.
[0036] Based on the specific implementation scenario listed in step S0, the mean ave_1 and standard deviation std_1 of the data set interv_11, interv_21, …, interv_n1 can be calculated here, and ave_1-3×std_1, ave_1 and ave_1+3×std_1 can be saved; the mean ave_2 and standard deviation std_2 of the data set interv_12, interv_22, …, interv_n2 can be calculated, and ave_2-3×std_2, ave_2 and ave_2+3×std_2 can be saved; …; the mean ave_m and standard deviation std_m of the data set interv_1m, interv_2m, …, interv_nm can be calculated, and ave_m-3×std_m, ave_m and ave_m+3×std_m can be saved.
[0037] Further, ave_1-3×std_1, ave_2-3×std_2, ..., ave_m-3×std_m can be used as the screen capture time interval after each click operation. In some implementation scenarios, ave_1, ave_2, ..., ave_m can also be used as the screen capture time interval after each click operation. It should be noted that according to statistical knowledge, in the normal distribution of statistics, the probability that the value is distributed in the range of (mean-3 times standard deviation, mean+3 times standard deviation) is 99.74, which can largely guarantee the stability of the data after the mouse click, so the screen capture time interval in the above embodiment is preferably (mean-3 times standard deviation), which can largely guarantee the accuracy of the selection of the screen capture time point.
[0038] Of course, the above is only an exemplary explanation, and the method of determining the screen capture time interval is not limited to the above example. Technical personnel in the relevant field may make other changes inspired by the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.
[0039] S4: taking screenshots of the graphical system screen in sequence according to the screen capture time interval after each click operation, to obtain screenshots corresponding to different screen capture times.
[0040] In the embodiment of this specification, after determining the screen capture time interval after each click operation, the screen of the graphical system can be captured in sequence according to the screen capture time interval after each click operation to obtain screenshots corresponding to different screen capture times. The screen capture time can be understood as the time corresponding to the screenshot.
[0041] In some embodiments, taking screenshots of the graphical system screen in sequence according to the screen capture time interval after each click operation to obtain screenshots corresponding to different screen capture times may include: obtaining the time of each click on the graphical system; determining the screen capture time after each click operation based on the time corresponding to each click operation and the screen capture time interval after each click operation; taking screenshots of the graphical system screen in sequence based on the screen capture time after each click operation to obtain screenshots corresponding to different screen capture times. When obtaining the time of each click on the graphical system, it may be counted whether the click operation is the number of click operations since the graphical system is started (or the graphical system process is detected) so as to determine the screen capture time interval after the corresponding click operation.
[0042] In some embodiments, the time corresponding to each click operation and the screen capture time interval after each click operation may be added together to determine the screen capture time after each click operation.
[0043] In some embodiments, after determining the screen capture time after each click operation, the screen capture time after each click operation can be recorded in sequence, and then screenshots are taken in sequence according to the recorded time. For example, if only one click operation is performed on the graphical system, and the time of the first click operation is 16:15:23, and the screen capture time interval after the first click operation is 2 seconds, then the screen capture time is 16:15:25, that is, the screen of the graphical system is captured at 16:15:25.
[0044] Based on the above embodiment, in which ave_1-3×std_1, ave_2-3×std_2, …, ave_m-3×std_m are used as the screen capture time intervals after each click operation, the time when the graphical system process is captured in the background can be recorded as T_0, the time point after ave_1-3×std_1 after T_0 can be recorded as T_1, the time point after ave_2-3×std_2 after the first click operation can be recorded as T_2, …, and the time point after ave_m-3×std_m after the mth click operation can be recorded as T_m. At this time, T_0, T_1, ..., T_m are respectively the screen capture time points of a certain RPA process during operation, and then the screen of the graphical system can be captured according to T_0, T_1, ..., T_m, and the screenshots corresponding to different screen capture times can be obtained, that is, after starting the graphical system (or detecting the graphical system process), the screenshot is taken after (ave_1-3×std_1) time, after the first click operation, the screenshot is taken after (ave_2-3×std_2) time, ..., after the mth click operation, the screenshot is taken after (ave_m-3×std_m) time. It should be noted that since the time of starting the graphical system may be different, the size of the above m+1 values corresponding to the same automated process at each run may be the same or different, so that the screen capture time can be dynamically determined according to the actual scenario. Of course, the above is only an exemplary description and does not constitute a limitation on other embodiments of this specification.
[0045] S6: Calculate similarities between screenshots corresponding to adjacent screenshot capture times based on the feature graphs of the screenshots.
[0046] In the embodiment of the present specification, after obtaining the screenshots corresponding to different screen capture times, the similarity between the screenshots corresponding to adjacent screen capture times can be calculated according to the feature graph of the screenshots. Wherein, the adjacent screen capture time refers to the time with adjacent positions among all the screen capture times determined in step S4. For example, the screen capture time after the first click operation is adjacent to the screen capture time after the second click operation, and the screen capture time after the second click operation is adjacent to the screen capture time after the third click operation, etc.
[0047] In some embodiments, before calculating the similarity between the screenshots corresponding to the adjacent screenshot capture times based on the feature graph of the screenshots, the method may include: extracting the corresponding feature graph from the screenshots. The screenshots may include one or more feature graphs. The feature graph may be used to characterize the screenshots. The feature graph may be understood as a region graph extracted from the screenshots that can represent most of the attributes of the screenshots.
[0048] In some embodiments, extracting the corresponding feature map from the screenshot may include: dividing the screenshot into multiple areas of the same size; selecting a screenshot corresponding to a preset area from the multiple areas of the same size as the feature map; wherein, when extracting the corresponding feature maps from different screenshots, the selected preset areas are the same.
[0049] like Figure 2 As stated, Figure 2 This is a schematic diagram of extracting a corresponding feature map from a screenshot provided by this specification. Among them, the entire area can be understood as a screenshot. The screenshot is divided into 5 parts both horizontally and vertically, and 25 areas of the same size can be obtained. Then, a preset number of areas are selected from the 25 areas according to preset rules as feature maps extracted from the screenshot. The preset rules can be used to determine which areas in the screenshot can better represent the screenshot. The specific implementation method can be determined according to the actual scenario, for example, it can be factor analysis, gray correlation analysis, etc. The preset number can be determined according to the actual scenario, and this specification does not limit this. For example, Figure 2 The preset number is 4, then for each screenshot, 4 dark areas in the screenshot are selected as feature maps extracted from the screenshot. Of course, the above is only an exemplary description, and the horizontal and vertical division of the screenshot is not limited to the above examples. Under the inspiration of the technical essence of this application, technicians in the relevant field may make other changes, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the protection scope of this application.
[0050] In the embodiments of the present specification, by extracting the feature map of the screenshot, the computational complexity and computational cost of subsequent similarity calculations can be reduced, thereby improving processing efficiency.
[0051] In the embodiment of the present specification, after the corresponding feature graph is extracted from the screenshots, the similarity between the screenshots corresponding to adjacent screenshot capture times can be further calculated based on the feature graph of the screenshots.
[0052] In some embodiments, the similarity between screenshots corresponding to adjacent screen capture times is calculated based on the feature map of the screenshots, and may include: normalizing the feature map to obtain a normalized feature map; the normalization is used to normalize the feature map to a preset size; based on the size of the normalized feature map, the color value of each pixel in the normalized feature map is adjusted to obtain an adjusted feature map; the average color value of all pixels in the adjusted feature map is calculated; the color value of each pixel in the adjusted feature map is compared with the average value to generate a fingerprint map corresponding to the feature map; the similarity between screenshots corresponding to adjacent screen capture times is calculated based on the fingerprint map corresponding to the feature map of adjacent screen capture times. The preset size can be set according to the actual scenario, for example, it can be 10, 20, etc.
[0053] In some embodiments, adjusting the color value of each pixel in the normalized feature map based on the size of the normalized feature map to obtain the adjusted feature map may include: obtaining the color range of the normalized feature map; determining a color adjustment rule based on the size of the normalized feature map; dividing the color range using the color adjustment rule; and adjusting the color value of each pixel in the normalized feature map according to the division result to obtain the adjusted feature map. The color range of the normalized feature map may be a range consisting of the minimum and maximum values of the color values corresponding to all pixels in the normalized feature map, or may be a preset color range, such as [0, 255]. The color adjustment rule may be used to determine how many regions the color range is divided into. For example, if the size of the normalized feature map is 2×2 pixels, then based on the size of the normalized feature map, the color adjustment rule determined is to divide the color range into 4 regions. For another example, if the size of the normalized feature map is 10×10 pixels, then based on the size of the normalized feature map, the color adjustment rule determined is to divide the color range into 100 regions.
[0054] In some implementation scenarios, after the color adjustment rule is determined, the color range can be divided using the color adjustment rule. For example, if the color range of the normalized feature map is [0, 255] and the size of the normalized feature map is 2×2 pixels, the result of dividing the color range using the color adjustment rule is [0, 63), [64, 127), [128, 191), [191, 255]. Furthermore, the color value of each pixel in the normalized feature map can be adjusted using the division result to obtain an adjusted feature map. For example, the color values of pixels in the normalized feature map whose color values are in the range of [0, 63) are adjusted to R1, the color values of pixels in the normalized feature map whose color values are in the range of [64, 127) are adjusted to R2, the color values of pixels in the normalized feature map whose color values are in the range of [128, 191) are adjusted to R3, and the color values of pixels in the normalized feature map whose color values are in the range of [191, 255] are adjusted to R4, so that an adjusted feature map can be obtained. At this time, the adjusted feature map includes 4 color values, namely R1, R2, R3, and R4. Of course, the above is only an exemplary description, and the method for obtaining the adjusted feature map is not limited to the above example. Technicians in the relevant field may make other changes under the inspiration of the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.
[0055] In some embodiments, the color value of each pixel in the adjustment feature map is compared with the average value to generate a fingerprint map corresponding to the feature map, which may include: marking a first identifier for the pixel whose color value is greater than or equal to the average value in the adjustment feature map, and marking a second identifier for the pixel whose color value is less than the average value in the adjustment feature map; sorting the identifiers of the pixel points in the adjustment feature map in a preset order to generate a fingerprint map corresponding to the feature map; wherein each fingerprint map includes the first identifier and the second identifier. Wherein, the first identifier and the second identifier may be one or more combinations of numerical values, letters, etc. For example, in some implementation scenarios, 1 is marked for the pixel points whose color value is greater than or equal to the average value in the adjustment feature map, and 0 is marked for the pixel points whose color value is less than the average value in the adjustment feature map, and then the mark values (0 or 1) corresponding to all the pixels in the adjustment feature map are sorted in a preset order to form a vector, which can be understood as a fingerprint map. The preset order may be to sort each column from top to bottom, or to sort each row from left to right, or other, which is not limited in this specification.
[0056] In some embodiments, after generating the fingerprint map corresponding to the feature map, the similarity between the screenshots corresponding to the adjacent screenshot times can be calculated based on the fingerprint maps corresponding to the feature maps at the adjacent screenshot times. Since the screenshots may include one or more feature maps, after generating the fingerprint map corresponding to the feature map, the similarity can be calculated based on the fingerprint maps corresponding to the feature maps at the corresponding positions in the adjacent screenshots. In this way, the similarity between the screenshots corresponding to the adjacent screenshot times may include one or more similarities. Each similarity is calculated based on the feature maps of the same sequence in the screenshots corresponding to the adjacent screenshot times, such as Figure 2 The dark area on the left side calculates the similarity between the feature maps.
[0057] In some embodiments, the calculation of the similarity between the screenshots corresponding to adjacent screen capture times based on the fingerprint map corresponding to the feature map of the adjacent screen capture times may include: obtaining a first fingerprint map and a second fingerprint map; the first fingerprint map and the second fingerprint map are fingerprint maps corresponding to the feature maps of the same sequence area in the screenshots corresponding to the adjacent screen capture times; counting the number of pixels with the same identification at the same sequence in the first fingerprint map and the second fingerprint map; and using the statistical result as the similarity between the screenshots corresponding to the adjacent screen capture times.
[0058] Specifically, the screen of the graphical system is captured at T_0, T_1, ..., T_m in step S4, and the Figure 2 Taking the extraction of the feature map of each screenshot from the dark area as an example, the similarity between the screenshots corresponding to the adjacent screenshot times is calculated for the four feature maps in the screenshots at each time point T_0, T_1, ..., T_m, such as calculating the similarity between the first feature map in T_0 and the first feature map in T_1, the similarity between the second feature map in T_0 and the second feature map in T_1, the similarity between the third feature map in T_0 and the third feature map in T_1, the similarity between the fourth feature map in T_0 and the fourth feature map in T_1, and the similarity between the first feature map in T_1 and the first feature map in T_1, respectively. 2, the similarity between the second feature map in T_1 and the second feature map in T_2, the similarity between the third feature map in T_1 and the third feature map in T_2, the similarity between the fourth feature map in T_1 and the fourth feature map in T_2, the similarity between the first feature map in T_m-1 and the first feature map in T_m, the similarity between the second feature map in T_m-1 and the second feature map in T_m, the similarity between the third feature map in T_m-1 and the third feature map in T_m, and the similarity between the fourth feature map in T_m-1 and the fourth feature map in T_m.
[0059] Next, we calculate the first feature map in T_0 (assuming Figure 2 The dark area on the left in the figure) and the first feature map in T_1 (assuming that Figure 2 The similarity of the first feature map in T_0 and the first feature map in T_1 is taken as an example for illustrative explanation. The similarity calculation method of other corresponding feature maps is similar and will not be described in detail. Specifically, firstly, the first feature map in T_0 and the first feature map in T_1 are normalized to M pixels × M pixels (such as M = 10), and then based on the size of the normalized feature map, M × M kinds (such as M = 10, then 100 kinds) of color values are used to simplify the color of the normalized feature map. Further, the average value of all pixels in the color simplified feature map is calculated, and the color value of each pixel in the color simplified feature map is compared with the average value. Pixels greater than or equal to the average value are recorded as 1, and pixels less than the average value are recorded as 0. Then, the pixels in each feature map are arranged from top to bottom in each column to obtain the fingerprint map corresponding to the first feature map in T_0 and the fingerprint map corresponding to the first feature map in T_1. Among them, each fingerprint map can be represented as a vector composed of 0 and 1. Furthermore, the marking information in the fingerprint map corresponding to the first feature map in T_0 and the fingerprint map corresponding to the first feature map in T_1 can be compared in sequence, and the number of identical markings in the same sequence in the fingerprint maps can be counted, and the statistical result can be used as the similarity between the first feature map in T_0 and the first feature map in T_1.
[0060] Of course, the above is only an exemplary description, and the method of calculating the similarity between feature maps is not limited to the above example. For example, after obtaining the fingerprint map of each feature map, the cosine value of the angle between the two fingerprint maps can be calculated as the corresponding similarity, or the similarity can be determined by calculating the Euclidean distance between the two. Inspired by the technical essence of this application, technical personnel in the relevant field may also make other changes, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.
[0061] In some implementation scenarios, feature maps may not be extracted, and the similarities between screenshots corresponding to adjacent screenshot times may be calculated directly based on screenshots corresponding to different screenshot times. The specific implementation method is similar to the above-mentioned implementation method and will not be described in detail.
[0062] S8: Based on the similarity, determine the state of the automation process when running in the graphical system.
[0063] In the embodiment of the specification, after obtaining the similarity between the screenshots corresponding to the adjacent screenshot times, the state of the automated process when running in the graphical system can be determined based on the similarity, wherein the state when running can include normal and abnormal.
[0064] In some embodiments, determining the state of the automated process when it is running in the graphical system based on the similarity may include: sequentially comparing the relationship between the similarity between the screenshots corresponding to the adjacent screenshot times and a preset threshold; when the similarity between the screenshots corresponding to the adjacent screenshot times is less than the preset threshold, determining that the state of the automated process when it is running in the graphical system is abnormal. The preset threshold may be set according to the actual scenario, for example, it may be 85, 90, etc.
[0065] In some implementation scenarios, since a screenshot may include one or more feature graphs, the similarity between screenshots corresponding to adjacent screenshot capture times may include one or more feature graphs.
[0066] In some implementation scenarios, when the similarity between the screenshots corresponding to adjacent screenshot times includes one, the similarity can be compared with a preset threshold. If it is less than the preset threshold, it can be explained that the screenshots corresponding to the screenshot times before and after the click operation have changed, that is, after the click operation is performed on the graphical system, the graphical system responds, so it can be determined that the state of the automated process when running in the graphical system is normal. If it is greater than or equal to the preset threshold, it can be explained that the screenshots corresponding to the screenshot times before and after the click operation have not changed, that is, after the click operation is performed on the graphical system, the graphical system does not respond, so it can be determined that the state of the automated process when running in the graphical system is abnormal.
[0067] In some implementation scenarios, when there are multiple similarities between screenshots corresponding to adjacent screenshot times, the multiple similarities can be compared with the preset thresholds respectively. If there is one that is less than the preset threshold, it can be said that the screenshots corresponding to the screenshot times before and after the click operation have changed, that is, after the click operation is performed on the graphical system, the graphical system responds, so it can be determined that the state of the automated process when running in the graphical system is normal. If all are greater than or equal to the preset threshold, it can be said that the screenshots corresponding to the screenshot times before and after the click operation have not changed, that is, after the click operation is performed on the graphical system, the graphical system does not respond, so it can be determined that the state of the automated process when running in the graphical system is abnormal.
[0068] In some embodiments, when it is determined that the state of the automated process when running in the graphical system is abnormal, the time corresponding to two screenshots corresponding to a similarity less than a preset threshold can be obtained, and then the abnormal location is determined based on the time corresponding to the two screenshots.
[0069] In some implementation scenarios, the adjacent screen capture times corresponding to the similarity less than the preset threshold can be obtained, and then the interval corresponding to the adjacent screen capture times is used as the abnormal position when the automated process is running in the graphical system. For example, in some implementation scenarios, when the similarity between the screenshots corresponding to the adjacent screen capture times is less than the preset threshold or there is a similarity less than the preset threshold, the capture times of the two screenshots corresponding to the similarity less than the preset threshold can be obtained, and then the interval between the capture times of the two screenshots is marked as the abnormal position.
[0070] In some implementation scenarios, after obtaining the capture time of two screenshots corresponding to a similarity less than a preset threshold, the abnormal time point can be determined based on the related values of the capture time of the two screenshots. For example, when determining the screen capture time interval after each click operation in the above step S2, the mean and standard deviation of the elements of the same order in multiple groups of click interval data sets are calculated, and then the mean-3 times the standard deviation is used as the screen capture time interval. At this time, when determining the abnormal time point, the mean can be used as the screen capture time interval and then the screen capture time is calculated, and then the interval between the screen capture times is marked as an abnormal position. Of course, the above is only an exemplary explanation, and the method for determining the abnormal position is not limited to the above example. The technical personnel in the relevant field may make other changes under the inspiration of the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.
[0071] The embodiments of this specification can reduce the amount of calculation while accurately locating the abnormal time point when the RPA process is unresponsive when running in the graphical system.
[0072] In some embodiments, after determining the abnormal location, the abnormal location can be fed back to the system developer so that the abnormality can be further processed. The abnormal location can also be referred to as the abnormal time point. In some implementation scenarios, the abnormal location can be visualized so that the developer can intuitively understand the abnormal area when the automated process is running in the graphical system.
[0073] In this specification, each embodiment of the above method is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. For relevant parts, please refer to the partial description of the method embodiment.
[0074] From the above description, it can be seen that the embodiment of the present application can obtain multiple groups of click interval data sets for the same automated process in the graphical system; wherein each group of click interval data sets includes a preset number of elements, each element represents the time interval corresponding to adjacent click operations; based on the elements of the same order in the multiple groups of click interval data sets, the screen capture time interval after each click operation is determined. Further, the screen of the graphical system is captured in sequence according to the screen capture time interval after each click operation to obtain screenshots corresponding to different screen capture times. Further, according to the feature graph of the screenshots, the similarity between the screenshots corresponding to the adjacent screen capture times is calculated, and based on the similarity, the state of the automated process when it is running in the graphical system is determined. Since the time of the screenshot is located in the manner of a time interval in the embodiment of the present application, it is more accurate than the method of setting the screen capture time according to human experience in the prior art, so that the subsequent abnormality detection results can be more accurate. Since the embodiment of the present application can calculate the similarity between the screenshots corresponding to the adjacent screen capture times according to the feature graph of the screenshots, and then determine the state of the automated process when it is running in the graphical system based on the similarity, compared with the prior art of continuously taking screenshots and continuously calculating the similarity, the calculation amount is less and the efficiency is higher.
[0075] Based on the above-mentioned anomaly detection method, one or more embodiments of this specification also provide an anomaly detection device. The device may include a system (including a distributed system), software (application), module, component, server, client, etc. using the method described in the embodiment of this specification and a device combined with necessary implementation hardware. Based on the same innovative concept, the device in one or more embodiments provided in the embodiment of this specification is as described in the following embodiments. Since the implementation scheme and method of the device to solve the problem are similar, the implementation of the specific device in the embodiment of this specification can refer to the implementation of the aforementioned method, and the repetitions will not be repeated. As used below, the term "unit" or "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.
[0076] Specifically, Figure 3 is a schematic diagram of a module structure of an embodiment of an abnormality detection device provided in this specification, such as Figure 3 As shown, an abnormality detection device provided in this specification may include: a click interval data acquisition module 120, a capture time interval determination module 122, a screenshot acquisition module 124, a similarity calculation module 126, and an operation status determination module 128.
[0077] The click interval data acquisition module 120 may be used to acquire multiple groups of click interval data sets of the same automated process in the graphical system; wherein each group of click interval data sets includes a preset number of elements, each element representing the time interval corresponding to adjacent click operations;
[0078] The capture time interval determination module 122 may be used to determine the screen capture time interval after each click operation based on the elements of the same sequence in the plurality of groups of click interval data sets;
[0079] The screenshot acquisition module 124 may be used to sequentially capture the screen of the graphical system according to the screen capture time interval after each click operation, and obtain screenshots corresponding to different screen capture times;
[0080] A similarity calculation module 126 may be used to calculate similarities between screenshots corresponding to adjacent screenshot capture times based on the feature graphs of the screenshots;
[0081] The running state determination module 128 may be configured to determine the state of the automation process when it is running in the graphical system based on the similarity.
[0082] It should be noted that the above-mentioned device may also include other implementation modes according to the description of the method embodiment. The specific implementation mode may refer to the description of the relevant method embodiment, which will not be described one by one here.
[0083] This specification also provides an embodiment of an abnormality detection device, including a processor and a memory for storing processor executable instructions, and the processor can implement any of the above method embodiments when executing the instructions. For example, when the instructions are executed by the processor, the following steps are implemented: obtaining multiple groups of click interval data sets of the same automated process in a graphical system; wherein each group of click interval data sets includes a preset number of elements, and each element represents the time interval corresponding to adjacent click operations; based on the elements of the same order in the multiple groups of click interval data sets, determining the screen capture time interval after each click operation; taking screenshots of the graphical system in sequence according to the screen capture time interval after each click operation, and obtaining screenshots corresponding to different screen capture times; calculating the similarity between the screenshots corresponding to adjacent screen capture times according to the feature graph of the screenshots; and determining the state of the automated process when it is running in the graphical system based on the similarity.
[0084] It should be noted that the above-mentioned device may also include other implementations according to the description of the method or device embodiment. The specific implementation methods can refer to the description of the relevant method embodiments, which will not be described one by one here.
[0085] The method embodiments provided in this specification can be executed in a mobile terminal, a computer terminal, a server or a similar computing device. Taking running on a server as an example, Figure 4 1 is a hardware structure block diagram of an embodiment of an abnormality detection server provided in this specification. The server may be an abnormality detection device or abnormality detection equipment in the above embodiment. Figure 4 As shown, the server 10 may include one or more (only one is shown in the figure) processors 100 (the processor 100 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 200 for storing data, and a transmission module 300 for communication functions. A person skilled in the art can understand that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components shown in the figure may also include other processing hardware, such as a database or multi-level cache, GPU, or other hardware with Figure 4 Different configurations shown.
[0086] The memory 200 can be used to store software programs and modules of application software, such as program instructions / modules corresponding to the abnormality detection method in the embodiment of this specification, and the processor 100 executes various functional applications and data processing by running the software programs and modules stored in the memory 200. The memory 200 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 200 may further include a memory remotely arranged relative to the processor 100, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0087] The transmission module 300 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of a computer terminal. In one example, the transmission module 300 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 300 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0088] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0089] The methods or devices described in the above embodiments provided in this specification can implement business logic through computer programs and record them on storage media, and the storage media can be read and executed by computers to achieve the effects of the solutions described in the embodiments of this specification. The storage medium may include a physical device for storing information, which is usually to digitize the information and then store it in a medium using electrical, magnetic or optical methods. The storage medium may include: devices that use electrical energy to store information, such as various types of memories, such as RAM, ROM, etc.; devices that use magnetic energy to store information, such as hard disks, floppy disks, magnetic tapes, magnetic core memories, bubble memories, and U disks; devices that use optical methods to store information, such as CDs or DVDs. Of course, there are other types of readable storage media, such as quantum memories, graphene memories, and so on.
[0090] The above-mentioned abnormality detection method or device embodiment provided in this specification can be implemented by a processor in a computer executing corresponding program instructions, such as using the C++ language of the Windows operating system to implement it on a PC, on a Linux system, or other systems such as Android and iOS system programming languages to implement it on a smart terminal, as well as based on the processing logic of a quantum computer.
[0091] It should be noted that the devices and equipment described in the specification may also include other implementation methods according to the description of the relevant method embodiments. The specific implementation methods can refer to the description of the corresponding method embodiments, which will not be described one by one here.
[0092] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0093] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of some modules can be implemented in the same or more software and / or hardware, and the modules that implement the same function can also be implemented by a combination of multiple sub-modules or sub-units.
[0094] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices, and apparatuses according to embodiments of the present invention. It should be understood that it can be implemented by computer program instructions, which can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing a specified function. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0095] Those skilled in the art will appreciate that one or more embodiments of the present specification may be provided as methods, systems or computer program products. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware.
[0096] The above is only an example of one or more embodiments of this specification, and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application should be included in the scope of the claims.
Claims
1. An anomaly detection method, characterized in that: The method comprises: Acquire multiple groups of click interval data sets of the same automated process in a graphical system; wherein each group of click interval data sets includes a preset number of elements, and each element represents the time interval corresponding to adjacent click operations; Determine the screen capture time interval after each click operation based on the elements of the same order in the multiple groups of click interval data sets; Sequentially capturing the screen of the graphical system according to the screen capturing time interval after each click operation, to obtain screen captures corresponding to different screen capturing times; Calculating similarities between screenshots corresponding to adjacent screenshot capture times based on the feature graphs of the screenshots; Based on the similarity, determining a state of the automation process when running in the graphical system; The step of determining the screen capture time interval after each click operation based on the elements of the same order in the multiple groups of click interval data sets includes: Calculate the mean and standard deviation of the elements of the same order in the multiple groups of click interval data sets; Determine the screen capture time interval after each click operation based on the mean and standard deviation of the elements of the same sequence; The screen capture time interval after each click operation is determined according to the following method: T=ave-3×std Where T represents the screen capture time interval, ave represents the mean, and std represents the standard deviation.
2. The method according to claim 1, characterized in that The step of obtaining multiple sets of click interval data sets of the same automated process in the graphical system includes: Acquire multiple click event data sets of the same automated process in the graphical system; wherein each click event data set includes the time corresponding to the click operation on the graphical system in sequence when the graphical system process is detected; The difference between adjacent data in each click event data set is calculated to obtain multiple sets of click interval data sets.
3. The method according to claim 1, characterized in that Before calculating the similarity between the screenshots corresponding to adjacent screenshot capture times according to the feature graph of the screenshots, the method includes: A corresponding feature map is extracted from the screenshot.
4. The method according to claim 3, characterized in that The extracting the corresponding feature map from the screenshot comprises: dividing the screenshot into a plurality of regions of equal size; A screenshot corresponding to a preset area is selected from the multiple areas of the same size as a feature map; wherein when extracting corresponding feature maps from different screenshots, the selected preset area is the same.
5. The method according to claim 1, characterized in that The calculating, based on the feature graph of the screenshots, the similarity between the screenshots corresponding to adjacent screenshot capture times comprises: Normalizing the feature map to obtain a normalized feature map; the normalization process is used to normalize the feature map to a preset size; Based on the size of the normalized feature map, the color value of each pixel in the normalized feature map is adjusted to obtain an adjusted feature map; Calculate the average color value of all pixels in the adjusted feature map; Compare the color value of each pixel in the adjusted feature map with the average value to generate a fingerprint map corresponding to the feature map; According to the fingerprint graph corresponding to the feature graph of adjacent screen capture times, the similarity between the screenshots corresponding to the adjacent screen capture times is calculated.
6. The method according to claim 5, characterized in that The adjusting the color value of each pixel in the normalized feature map based on the size of the normalized feature map to obtain the adjusted feature map includes: Obtaining a color range of the normalized feature map; Determining a color adjustment rule based on the size of the normalized feature map; Dividing the color range by using the color adjustment rule; According to the division result, the color value of each pixel in the normalized feature map is adjusted to obtain an adjusted feature map.
7. The method according to claim 5, characterized in that The step of comparing the color value of each pixel in the adjusted feature map with the average value to generate a fingerprint map corresponding to the feature map comprises: Marking a first identifier for a pixel point in the adjusted feature map whose color value is greater than or equal to the average value, and marking a second identifier for a pixel point in the adjusted feature map whose color value is less than the average value; The identifiers of the pixel points in the adjusted feature map are sorted in a preset order to generate a fingerprint map corresponding to the feature map; wherein each fingerprint map includes the first identifier and the second identifier.
8. The method according to claim 7, characterized in that The calculating the similarity between the screenshots corresponding to the adjacent screenshot times according to the fingerprint graphs corresponding to the feature graphs at the adjacent screenshot times comprises: Acquire a first fingerprint image and a second fingerprint image; the first fingerprint image and the second fingerprint image are fingerprint images corresponding to feature images of the same sequence area in the screenshots corresponding to adjacent screenshot times; Counting the number of pixels with the same identifier at the same sequence in the first fingerprint image and the second fingerprint image; The statistical result is used as the similarity between the screenshots corresponding to the adjacent screenshot capture times.
9. The method according to claim 1, characterized in that: The determining, based on the similarity, a state of the automation process when running in the graphical system includes: Comparing the relationship between the similarities between the screenshots corresponding to the adjacent screenshot capture times and a preset threshold in sequence; When the similarity between the screenshots corresponding to adjacent screenshot capture times is less than the preset threshold, it is determined that the state of the automation process when running in the graphical system is abnormal.
10. The method according to claim 9, characterized in that The determining that the state of the automation process when running in the graphical system is abnormal includes: Obtaining adjacent screen capture times corresponding to similarities less than the preset threshold; The interval corresponding to the adjacent screen capture times is used as the abnormal position of the automation process when it is running in the graphical system.
11. An abnormality detection device, characterized in that: include: A click interval data acquisition module is used to acquire multiple groups of click interval data sets of the same automated process in a graphical system; wherein each group of click interval data sets includes a preset number of elements, and each element represents the time interval corresponding to adjacent click operations; A capture time interval determination module, used to determine the screen capture time interval after each click operation based on the elements of the same sequence in the multiple groups of click interval data sets; A screenshot acquisition module, used to sequentially capture screenshots of the graphical system according to the screen capture time interval after each click operation, to obtain screenshots corresponding to different screen capture times; A similarity calculation module, used to calculate the similarity between the screenshots corresponding to adjacent screenshot capture times according to the feature graph of the screenshots; An operation status determination module, configured to determine, based on the similarity, a state of the automation process when it is running in the graphical system; The capture time interval determination module is specifically used to: calculate the mean and standard deviation of the elements of the same order in the multiple groups of click interval data sets; determine the screen capture time interval after each click operation based on the mean and standard deviation of the elements of the same order; The capture time interval determination module is specifically used to determine the screen capture time interval after each click operation according to the following method: T=ave-3×std Where T represents the screen capture time interval, ave represents the mean, and std represents the standard deviation.
12. An abnormality detection device, characterized in that: The method comprises a processor and a memory for storing processor executable instructions, wherein the processor implements the steps of the method according to any one of claims 1 to 10 when executing the instructions.
13. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
14. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 10 are implemented.
