A computer-controlled behavior class recognition method and medium

Abstract

The application relates to a computer control behavior category identification method and medium, and relates to the technical field of computer control. By identifying current computer control parameters, the computer control parameters are compared with corresponding reference thresholds to identify whether the current computer control behavior category is a biological control behavior category or a program control behavior category, wherein the computer control parameters include any one or several of a curvature fluctuation value of a mouse moving track, a speed fluctuation rate, a target offset rate and a random pause rate. In this way, the biological control behavior and the program control behavior of the computer are identified, so that if it is necessary to give priority to the control right to a person, program control is stopped based on the judgment result, and the operation behavior of the person is prevented from being affected.

Description

A method and medium for recognizing computer-controlled behavior categories Technical Field

[0001] This invention relates to the field of computer control technology, and specifically to a method and medium for identifying computer control behavior categories. Background Technology

[0002] Currently, computer control generally involves human intervention through a human-computer interaction interface, such as controlling the mouse cursor (hereinafter referred to as the mouse) via a mouse and / or keyboard. Whether remote or local, this control method can be collectively categorized as biological control behavior.

[0003] However, with the development of the times, in order to save manpower, more and more human behaviors of controlling the mouse and keyboard have begun to be imitated. For example, RPA robot programs use RPA to crawl data on the computer. When using RPA to retrieve data, RPA will replace human behavior and automatically control the movement and clicking of the mouse. This control method can be collectively referred to as the program control behavior category.

[0004] At this point, the categories of biologically controlled behaviors and program-controlled behaviors become an urgent problem to be solved. Summary of the Invention

[0005] The technical problem to be solved by this application is to provide a method and medium for identifying computer control behavior categories, which has the characteristics of being able to identify both biological control behavior and program control behavior of a computer.

[0006] In a first aspect, this application provides a method for identifying computer-controlled behavior categories, including:

[0007] Identify the current computer control parameters and compare them with the corresponding reference thresholds to identify whether the current computer control behavior is a biological control behavior or a program control behavior.

[0008] The computer control parameters include any one or more of the following: curvature fluctuation value of the mouse movement trajectory, speed fluctuation rate, target offset rate, and random pause rate.

[0009] Optionally, when the computer control parameters include the curvature fluctuation value of the mouse movement trajectory, comparing the computer control parameters with a corresponding reference threshold to identify whether the current computer control behavior category is a biological control behavior category or a program control behavior category includes:

[0010] Calculate the curvature of each segment of the mouse movement trajectory;

[0011] Calculate the standard deviation of curvature based on all curvatures;

[0012] Determine whether the standard deviation of curvature is greater than a preset standard deviation threshold of curvature. If it is, the current computer control behavior category is considered to be the biological control behavior category. If not, the current computer control behavior category is considered to be the program control behavior category.

[0013] Optionally, calculating the curvature of each segment of the mouse movement trajectory includes:

[0014] For any movement path of the mouse, obtain the sequence of path points;

[0015] In the trajectory point sequence, every three consecutive points are grouped together to obtain multiple groups of consecutive points;

[0016] Extract the vector formed by two consecutive points in each group of consecutive points to obtain vectors a and b;

[0017] Calculate the angle between each pair of vectors a and b to obtain the angle corresponding to each pair of consecutive points, which is used as the curvature of each trajectory segment.

[0018] The method of calculating the standard deviation of curvature based on all curvatures includes: calculating the standard deviation of the included angle sequence and using this standard deviation as the standard deviation of curvature.

[0019] Optionally, when the computer control parameters include the speed fluctuation rate of the mouse movement trajectory, comparing the computer control parameters with a corresponding reference threshold to identify whether the current computer control behavior is a biological control behavior or a program control behavior includes:

[0020] Get the pixel distance the mouse moves within each consecutive time interval;

[0021] The speed corresponding to each time interval is calculated based on the pixel distance corresponding to each time interval;

[0022] Calculate the standard deviation of velocity based on the velocity corresponding to all time intervals;

[0023] Determine whether the speed standard deviation is greater than a preset speed standard deviation threshold. If it is, the current computer control behavior category is considered to be the biological control behavior category; otherwise, the current computer control behavior category is considered to be the program control behavior category.

[0024] Optionally, when the computer control parameters include the target offset rate of the mouse movement trajectory, comparing the computer control parameters with a corresponding reference threshold to identify whether the current computer control behavior category is a biological control behavior category or a program control behavior category includes:

[0025] Get the mouse coordinates corresponding to each click when clicking the same target multiple times consecutively;

[0026] Calculate the deviation of each mouse coordinate from the center point of the same target;

[0027] Calculate the standard deviation of the deviation based on all deviations;

[0028] Determine whether the standard deviation of the deviation is greater than a preset standard deviation threshold. If it is, the current computer control behavior category is considered to be a biological control behavior category. If not, the current computer control behavior category is considered to be a program control behavior category.

[0029] Optionally, obtaining the mouse coordinates corresponding to each click when clicking the same target multiple times consecutively includes:

[0030] Get the pixel coordinates of any mouse click;

[0031] Based on the pixel coordinates, determine the target to which the pixel coordinates belong and obtain the target range;

[0032] Determine whether the pixel coordinates of the next adjacent mouse click are within the target range. If they are, obtain the pixel coordinates of the next adjacent mouse click until the pixel coordinates of the next adjacent mouse click are no longer within the target range.

[0033] Get the mouse coordinates of each of the multiple consecutive clicks within the target area.

[0034] Optionally, when the computer control parameters include the number of random pauses in the mouse movement trajectory, comparing the computer control parameters with a corresponding reference threshold to identify whether the current computer control behavior is a biological control behavior or a program control behavior includes:

[0035] For any mouse movement path, identify the pauses in that movement path;

[0036] Based on the identified pauses, calculate the number of pauses in any given movement trajectory;

[0037] Determine whether the number of pauses is greater than a preset pause threshold. If it is, the current computer control behavior is considered to be a biological control behavior; otherwise, the current computer control behavior is considered to be a program control behavior.

[0038] Optionally, identifying pauses in any mouse movement trajectory includes: determining whether, during the movement of any mouse movement trajectory, the time spent within any preset coordinate range threshold exceeds a preset time period threshold; if so, it is considered that a pause exists.

[0039] Optionally, the computer control parameters include any and multiple of the following: curvature fluctuation value, speed fluctuation rate, target offset rate, and random pause rate of the mouse movement trajectory. If any one or more of these computer control parameters meet the threshold range of the biological control behavior category, then the current computer control behavior is considered to be a biological control behavior category; otherwise, the current computer control behavior category is considered to be a program control behavior category.

[0040] In a second aspect, one embodiment provides a computer-readable storage medium storing a program that can be loaded by a processor and executed by the computer control behavior category recognition method described in any one of the above embodiments.

[0041] The beneficial effects of this invention are:

[0042] By identifying the current computer control parameters and comparing them with corresponding reference thresholds, it is possible to determine whether the current computer control behavior is a biological control behavior or a program control behavior. In this way, it can be determined whether a human is operating the computer or the RPA is operating the computer. If it is necessary to give priority to human control, the RPA can be stopped based on this judgment to avoid affecting human operation. Attached Figure Description

[0043] Figure 1 is a schematic flowchart of a computer control behavior category recognition method according to an embodiment of this application;

[0044] Figure 2 is a schematic flowchart of a method for calculating the curvature fluctuation value of a mouse movement trajectory according to an embodiment of this application;

[0045] Figure 3 is a schematic flowchart of a method for calculating the speed fluctuation rate of a mouse movement trajectory according to an embodiment of this application.

[0046] Figure 4 is a schematic flowchart of a method for calculating the target offset rate of a mouse movement trajectory according to an embodiment of this application.

[0047] Figure 5 is a schematic flowchart of a method for calculating the number of random pauses in mouse movement trajectory according to an embodiment of this application. Detailed Implementation

[0048] The present invention will now be described in further detail with reference to specific embodiments and accompanying drawings. Similar elements in different embodiments are referred to by associated similar element reference numerals. In the following embodiments, many details are described to facilitate a better understanding of this application. However, those skilled in the art will readily recognize that some features may be omitted in different situations, or may be replaced by other elements, materials, or methods. In some cases, certain operations related to this application are not shown or described in the specification. This is to avoid obscuring the core parts of this application with excessive description. For those skilled in the art, detailed description of these related operations is not necessary; they can fully understand the related operations based on the description in the specification and general technical knowledge in the art.

[0049] Furthermore, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can be rearranged or adjusted in a manner obvious to those skilled in the art. Therefore, the various orders in the specification and drawings are only for the clear description of a particular embodiment and do not imply a necessary order, unless otherwise stated that a particular order must be followed.

[0050] The serial numbers assigned to components in this article, such as "first" and "second", are used only to distinguish the objects being described and have no sequential or technical meaning.

[0051] To facilitate the explanation of the inventive concept of this application, the computer control technology will be briefly described below.

[0052] The applicant discovered in their research that when a human needs to manually operate a computer, if an RPA program is running at the same time, a phenomenon occurs where the human and the RPA compete for the computer's use. This necessitates providing a method to determine whether a human or the RPA is operating the computer. If the RPA is operating the computer, then when human intervention occurs, the RPA can be stopped as needed to avoid affecting human operation.

[0053] In view of this, embodiments of this application provide a method and medium for identifying computer control behavior categories. By identifying the current computer control parameters and comparing them with the corresponding reference thresholds, the method identifies whether the current computer control behavior category is a biological control behavior category or a program control behavior category. In this way, it can be determined whether a human is operating the computer or RPA is operating the computer. If it is necessary to give priority to human control, RPA can be stopped based on this judgment result to avoid affecting human operation.

[0054] This application provides a method for recognizing computer control behavior categories, as shown in Figure 1, including:

[0055] Step S10: Identify the current computer control parameters.

[0056] In some embodiments, computer control parameters may include any one or more of the following: curvature fluctuation value of mouse movement trajectory, speed fluctuation rate, target offset rate, and number of random pauses.

[0057] For mouse control behavior, the coordinates and timestamps of mouse movement can be recorded in real time to form a trajectory sequence. Based on these trajectory sequences, the curvature fluctuation value, speed fluctuation rate, target offset rate, and number of random pauses of the mouse movement trajectory can be obtained.

[0058] For the curvature fluctuation value of the mouse movement trajectory, please refer to Figure 2. The calculation method may include:

[0059] Step S111: Calculate the curvature of each segment of the mouse movement trajectory.

[0060] Curvature can be used to describe the degree of curvature of a curve. The curvature of a straight line is 0, and the greater the degree of curvature of a curve, the greater its curvature.

[0061] In this embodiment, for a discrete sequence of mouse trajectory points, the curvature is approximated by the change in the angle formed by three adjacent points, and the fluctuation of these curvature values ​​is measured by the standard deviation of curvature. Therefore, in one embodiment, step S111 may include:

[0062] Step S1111: For any movement trajectory of the mouse, obtain the trajectory point sequence.

[0063] Suppose the mouse movement trajectory is recorded as a discrete point set D (including coordinates and timestamps). Then this discrete point set can be represented as D = [(x1, y1, t1), (x2, y2, t2), ..., (x... n y n , t n ), ......, (x N y N , t N ]], where x and y represent the horizontal and vertical coordinates respectively, t represents time, n represents the index of the discrete point, N represents the total number of discrete points, and 1≤n≤N.

[0064] Those skilled in the art will understand that all coordinates in this application can be pixel coordinates. Those skilled in the art will also understand that the identification of any mouse movement trajectory can be based on two adjacent clicks. If the target areas of two adjacent clicks do not belong to the same target area (for example, one is a save button target, and the other is another button target outside the save button), then the trajectory between the two clicks can be identified as a single movement trajectory. Alternatively, other movement trajectory identification methods can be set based on specific needs.

[0065] Step S1112: In the trajectory point sequence, group every three consecutive points together to obtain multiple groups of consecutive points.

[0066] Step S1113: Extract the vector formed by two consecutive points in each group of consecutive points to obtain vector a and vector b.

[0067] For example, for any set of continuous points (P) i-1 P i P i+1 If 2≤i≤N-1, then we have: P i-1 =(x i-1 y i-1 ), P i =(x i y i ), P i+1 =(x i+1 y i+1 Then we can obtain:

[0068] Vector a=P i -P i-1 = (x i -x i-1 y i -y i-1 ), vector b=P i+1 -P i = (x i+1 -x i y i+1 -y i ).

[0069] Step S1114: Calculate the angle between each set of vectors a and b to obtain the angle corresponding to each set of continuous points, which is used as the curvature of each trajectory segment.

[0070] The above calculation of the angle between each pair of vectors a and b can be expressed as:

[0071]

[0072] in, This represents the angle between vectors a and b. Let a represent vector a. Let b represent vector. This represents the magnitude of vector a. This represents the magnitude of vector b; This is a local minimum value used to avoid a denominator of 0. When =0° represents a straight line (curvature of 0). When the angle is 90°, it indicates that the trajectory makes a right-angle turn at this point (large curvature).

[0073] Step S112: Calculate the standard deviation of curvature based on all curvatures, and use the standard deviation of curvature as the curvature fluctuation value.

[0074] In one embodiment, the standard deviation of the included angle sequence is calculated and used as the standard deviation of curvature.

[0075] For the speed fluctuation rate of the mouse movement trajectory, please refer to Figure 3. The calculation method may include:

[0076] Step S121: Obtain the pixel distance the mouse moves in each time interval within a continuous time interval.

[0077] This time interval can be set based on actual needs, such as 200ms or 300ms.

[0078] Step S122: Calculate the speed corresponding to each time interval based on the pixel distance corresponding to each time interval.

[0079] Step S123: Calculate the speed standard deviation based on the speed corresponding to all time intervals, and use the speed standard deviation as the speed volatility.

[0080] In one embodiment, the target offset rate of the mouse movement trajectory, as shown in Figure 4, can be calculated using the following method:

[0081] Step S131: Obtain the mouse coordinates corresponding to each click when clicking the same target multiple times consecutively.

[0082] In one embodiment, step S131 may include:

[0083] Step S1311: Obtain the pixel coordinates of any mouse click.

[0084] The click operation is identified, and the pixel coordinates of the mouse cursor are obtained for the identified click operation.

[0085] Step S1312: Based on the above pixel coordinates, determine the target to which the pixel coordinates belong and obtain the target range.

[0086] For example, if the target of the obtained pixel coordinates is the save button of a file, then the pixel coordinate range of the save button can be obtained.

[0087] Step S1313: Determine whether the pixel coordinates of the next adjacent mouse click are within the target range. If so, obtain the pixel coordinates of the next adjacent mouse click until the pixel coordinates of the next adjacent mouse click are no longer within the target range.

[0088] For example, if the current click is detected to be within the pixel coordinate range of the save button, this coordinate range can be recorded first. Then, the next click is detected, and it's determined whether the mouse's pixel coordinates are within the save button's pixel coordinate range. If not, the trajectory between the two clicks can be identified as a movement trajectory. If it is, the pixel coordinates of the next adjacent click are obtained, and so on, until the pixel coordinates of the next adjacent click are no longer within the save button's pixel coordinate range.

[0089] Step S1314: Obtain multiple consecutive clicks within the target range, and the mouse coordinates corresponding to each click.

[0090] For example, if the pixel coordinates of the mouse clicks are all within the pixel coordinate range of the save button after three consecutive clicks, then the mouse coordinates corresponding to each of the three clicks are obtained.

[0091] Step S132: Calculate the deviation of each mouse coordinate from the center point of the same target.

[0092] For example, if the pixel coordinates of the mouse clicks are all within the pixel coordinate range of the save button after three consecutive clicks, then calculate the deviation of each mouse click coordinate from the center point of the save button.

[0093] Step S133: Calculate the standard deviation of the deviation based on all deviations, and use the standard deviation of the deviation as the target offset rate.

[0094] According to the above embodiment, three deviations can be calculated, and the standard deviations of the three deviations can be obtained, thereby obtaining the target offset rate.

[0095] In one embodiment, the number of random pauses in the mouse movement trajectory, as shown in Figure 5, can be calculated using the following method:

[0096] Step S141: For any movement trajectory of the mouse, identify the pauses in that movement trajectory.

[0097] In one embodiment, step S141 includes: for any movement trajectory of the mouse, determining whether the time spent within any preset coordinate range threshold exceeds a preset time period threshold during the movement process; if so, it is considered that there is a pause.

[0098] For example, if the dwell time exceeds 50ms within a 10*10 pixel grid, it is considered that there is a pause at that location; if the dwell time exceeds 50ms within another 10*10 pixel grid, it is considered that there is another pause.

[0099] Step S142: Based on the identified pauses, calculate the number of pauses in any given movement trajectory.

[0100] Step S20: Compare the identified computer control parameters with the corresponding reference thresholds to identify whether the current computational control behavior category is a biological control behavior category or a program control behavior category.

[0101] Regarding the curvature fluctuation value of the mouse movement trajectory, a larger curvature fluctuation value indicates a more irregular mouse movement trajectory, and the more likely the current computer control behavior is of the biological control type. Conversely, a smaller curvature fluctuation value indicates a more regular mouse movement trajectory, and the more likely the current computer control behavior is of the program control type. Therefore, a pre-set curvature standard deviation threshold can be set, and the calculated curvature standard deviation can be judged to be greater than the pre-set threshold. If it is greater, the current computer control behavior is considered to be of the biological control type; otherwise, it is considered to be of the program control type.

[0102] Regarding the speed fluctuation rate of mouse movement, a larger fluctuation rate indicates a more unstable mouse movement speed, making the current computer control behavior more likely to be classified as biological control behavior. Conversely, a smaller fluctuation rate indicates a more stable mouse movement speed, making the current computer control behavior more likely to be classified as program control behavior. Therefore, a preset speed standard deviation threshold can be used to determine whether the calculated speed standard deviation exceeds the preset threshold. If it does, the current computer control behavior is considered to be biological control behavior; otherwise, it is considered to be program control behavior.

[0103] For the target offset rate of mouse movement trajectory, a larger standard deviation indicates more dispersed mouse click positions, making the current computer control behavior more likely to be biological control behavior. A smaller standard deviation indicates more concentrated mouse click positions, making the current computer control behavior more likely to be program control behavior. Therefore, a preset standard deviation threshold can be set. The calculated standard deviation is then checked against this threshold. If it is greater than the preset threshold, the current computer control behavior is considered biological control behavior; otherwise, it is considered program control behavior.

[0104] Regarding the number of random pauses in the mouse movement trajectory, a higher number of pauses indicates more irregular mouse movement, making the current computer control behavior more likely to be biological control behavior. Conversely, fewer random pauses indicate more regular mouse movement, making the current computer control behavior more likely to be program control behavior. Therefore, a preset threshold for the number of random pauses can be set. The system checks if the calculated number of pauses exceeds this threshold. If it does, the current computer control behavior is considered biological control behavior; otherwise, it is considered program control behavior.

[0105] Those skilled in the art will understand that the aforementioned standard deviation thresholds for curvature, velocity, deviation, and random pauses can all be set based on actual needs.

[0106] In one embodiment, the computer control parameters include any combination of the curvature fluctuation value, speed fluctuation rate, target offset rate, and random pause rate of the mouse movement trajectory. If any one or more of these computer control parameters meet the threshold range of the biological control behavior category, then the current computer control behavior is considered to be a biological control behavior category; otherwise, the current computer control behavior is considered to be a program control behavior category.

[0107] Those skilled in the art will understand that when there are multiple computer control parameters, a setting interface can be opened to allow the setting of the categories and logical relationships contained in the control parameters.

[0108] One embodiment of this application provides a computer-readable storage medium storing a program, the stored program including methods that can be loaded by a processor and processed in any of the above embodiments.

[0109] Those skilled in the art will understand that all or part of the functions of the various methods in the above embodiments can be implemented by hardware or by computer programs. When all or part of the functions in the above embodiments are implemented by computer programs, the program can be stored in a computer-readable storage medium, which may include: read-only memory, random access memory, disk, optical disk, hard disk, etc., and the program is executed by a computer to achieve the above functions. For example, the program can be stored in the memory of a device, and when the program in the memory is executed by the processor, all or part of the above functions can be achieved. In addition, when all or part of the functions in the above embodiments are implemented by computer programs, the program can also be stored in a server, another computer, disk, optical disk, flash drive, or external hard drive, etc., and can be downloaded or copied to the memory of a local device, or the system of the local device can be updated. When the program in the memory is executed by the processor, all or part of the functions in the above embodiments can be achieved.

[0110] The above examples illustrate the present invention only to aid in understanding it and are not intended to limit the scope of the invention. Those skilled in the art can make various simple deductions, modifications, or substitutions based on the principles of this invention.

Claims

1. A method for identifying RPA robot control behavior categories, characterized in that, include: The current computer control parameters are identified and compared with corresponding reference thresholds to identify whether the current computer control behavior is a biological control behavior or a program control behavior; the computer control parameters include any one or more of the curvature fluctuation value of the mouse movement trajectory, the target offset rate, and the random pause rate; the curvature fluctuation value is obtained by calculating the curvature of each segment of the mouse movement trajectory. Calculate the standard deviation of curvature based on all curvatures; The standard deviation of curvature is used as the curvature fluctuation value; the target offset rate is obtained by: obtaining the mouse coordinates corresponding to each click when clicking the same target multiple times consecutively; calculating the deviation between each mouse coordinate and the center point of the same target; calculating the standard deviation of the deviation based on all deviations; and using the standard deviation of the deviation as the target offset rate; the random pause rate is obtained by: identifying pauses in any movement trajectory of the mouse; and calculating the number of pauses in any movement trajectory based on the identified pauses. The number of pauses is taken as the random pause rate; and when the curvature fluctuation value is greater than a first preset threshold, and / or the target offset rate is greater than a second preset threshold, and / or the random pause rate is greater than a third preset threshold, the current computer control behavior is determined to be a biological control behavior category; when the current computer control behavior is identified as a biological control behavior category, the RPA robot stops running; the acquisition of the mouse coordinates corresponding to each click when clicking the same target multiple times in a row includes: acquiring the pixel coordinates of the mouse at any click; Based on the pixel coordinates, determine the target to which the pixel coordinates belong and obtain the target range; determine whether the pixel coordinates of the next adjacent mouse click belong to the target range. If so, obtain the pixel coordinates of the next adjacent mouse click until the pixel coordinates of the next adjacent mouse click do not belong to the target range; obtain multiple consecutive clicks that belong to the target range, and the mouse coordinates corresponding to each click.

2. The RPA robot control behavior category recognition method as described in claim 1, characterized in that, The calculation of the curvature of each segment of the mouse movement trajectory includes: for any movement trajectory of the mouse, obtaining a sequence of trajectory points; grouping every three consecutive points in the trajectory point sequence into multiple groups of consecutive points; extracting the vector formed by two consecutive points in each group of consecutive points to obtain vector a and vector b; calculating the angle between vector a and vector b in each group to obtain the angle corresponding to each group of consecutive points, which is used as the curvature of each segment of the trajectory; the calculation of the standard deviation of curvature based on all curvatures includes: calculating the standard deviation of the angle sequence and using this standard deviation as the standard deviation of curvature.

3. The RPA robot control behavior category recognition method as described in claim 1, characterized in that, The method of identifying pauses in any movement trajectory of the mouse includes: for any movement trajectory of the mouse, determining whether the time spent within any preset coordinate range threshold exceeds a preset time period threshold during the movement process; if so, it is considered that there is a pause.

4. A computer-readable storage medium, characterized in that, The medium stores a program that can be loaded by a processor and executed as described in any one of claims 1 to 3, the RPA robot control behavior category recognition method.

Images