An AI-based keyboard and mouse trajectory recognition and operation intention analysis method

Abstract

This application provides an AI-based method for keyboard and mouse trajectory recognition and operation intent analysis, relating to the field of human-computer interaction technology. It acquires multi-source interaction data from a keyboard and mouse terminal and state images of the application interface; performs time-series analysis on the multi-source interaction data to obtain keyboard and mouse trajectory features; collects the positions and interaction states of controls on the application interface; identifies control operations based on the control positions, interaction states, and state images to obtain operation event sequences and candidate operations; identifies trajectory anomalies based on the keyboard and mouse trajectory features; performs continuous correction on the trajectory anomalies using candidate operations and control interaction states to obtain corrected trajectories; and performs time-series matching and behavior correction on the corrected trajectories and operation event sequences to obtain the keyboard and mouse operation intent. This application can achieve collaborative recognition of keyboard and mouse trajectories and interface operation states, improving the accuracy of keyboard and mouse operation intent recognition.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and more specifically, to an AI-based method for recognizing keyboard and mouse trajectories and analyzing operational intentions. Background Technology

[0002] Human-computer interaction is an important technical means for computer systems to realize information transmission and operation control between users and computer terminals. Users can input operation commands to the computer terminal through input devices such as keyboards and mice, and the computer terminal changes the display state of the application interface and executes corresponding functions according to the input operation.

[0003] With the increasing diversity of computer terminal applications, user operations are becoming more continuous, complex, and involve multi-step interactions, placing higher demands on the identification and analysis of user behavior. Existing AI-based keyboard and mouse trajectory recognition and operation intent analysis technologies primarily rely on keyboard and mouse interaction data for trajectory feature extraction and behavior recognition, with limited integration of the control states within the application interface to constrain the keyboard and mouse trajectories. When users perform rapid movements, continuous clicks, or drags, the keyboard and mouse trajectories are prone to sudden changes or deviations, making it difficult to accurately distinguish between normal operations and abnormal trajectories based solely on trajectory changes. Furthermore, the lack of collaborative analysis between keyboard and mouse trajectories and interface operation states leads to insufficient accuracy in identifying operation events and determining operation intent in complex interactive scenarios. Therefore, achieving collaborative recognition of keyboard and mouse trajectories and interface operation states to improve the accuracy of keyboard and mouse operation intent recognition remains a challenging problem for the industry. Summary of the Invention

[0004] This application provides an AI-based method for keyboard and mouse trajectory recognition and operation intent analysis, which can achieve collaborative recognition of keyboard and mouse trajectories and interface operation states, thereby improving the accuracy of keyboard and mouse operation intent recognition.

[0005] In a first aspect, this application provides an AI-based method for keyboard and mouse trajectory recognition and operation intent analysis, the analysis method comprising the following steps: Acquire multi-source interaction data and application interface status images of the keyboard and mouse terminal, perform time-series analysis on the multi-source interaction data, and obtain keyboard and mouse trajectory features; Collect the position and interaction state of controls on the application interface, and perform control operation recognition based on the control position, the control interaction state and the state image to obtain the operation event sequence and candidate operations; Based on the keyboard and mouse trajectory features, abnormal trajectory points are identified, and the abnormal trajectory points are continuously corrected through the candidate operations and the control interaction state to obtain a corrected trajectory. The correction trajectory and the operation event sequence are matched in time and corrected in behavior to obtain the keyboard and mouse operation intention.

[0006] In this embodiment, acquiring multi-source interaction data and application interface status images of the keyboard and mouse terminal specifically includes: Collect interaction records during the operation of the keyboard and mouse terminal, and perform event parsing and time stamping on the interaction records to obtain multi-source interaction data of the keyboard and mouse terminal; The interface images are collected during the continuous change of the application interface, and the interface images are segmented into states based on the time stamps of the multi-source interaction data to obtain the state images of the application interface.

[0007] In this embodiment, performing time-series analysis on the multi-source interaction data to obtain keyboard and mouse trajectory features specifically includes: Based on the mouse position and interaction time in the multi-source interaction data, the data is sorted in time sequence, and the position change and time interval between adjacent interaction positions are calculated to obtain trajectory change characteristics. The continuous interactive positions are divided into segments based on the trajectory change characteristics, and the trajectory continuity characteristics are determined based on the position change and direction change between adjacent interactive segments. By performing a time-series mapping between the trajectory change features and the trajectory continuity features, the keyboard and mouse trajectory features are obtained.

[0008] In this embodiment, collecting the position and interaction state of controls on the application interface specifically includes: Obtain the layout information of the application interface, divide the layout information into control areas, and obtain the control positions; Based on the position of the control, the region of control state change in the state image is extracted to obtain the state change segment; The interaction states of the control are classified according to the state change segments.

[0009] In this embodiment, the control operation recognition based on the control position, the control interaction state, and the state image to obtain the operation event sequence and candidate operations specifically includes: Spatial mapping is performed between the state image and the control position to obtain the control area; Operation states are identified based on the control area and the control interaction state to obtain control operation events, and these events are sorted according to their occurrence time to obtain an operation event sequence. Candidate operations are obtained by continuously filtering operations based on the operation sequence in the operation event sequence and the spatial transfer between the control areas.

[0010] In this embodiment, identifying trajectory anomalies based on the keyboard and mouse trajectory features specifically includes: The trajectory change features and trajectory continuity features in the keyboard and mouse trajectory features are input into the trained trajectory anomaly discrimination model. The trajectory anomaly discrimination model is used to judge the degree of change in the position of continuous trajectory to obtain the trajectory abrupt change segment. Local continuity analysis is performed on the trajectory abrupt change segment, and the positional and directional changes of the continuous trajectory positions before and after the trajectory abrupt change segment are correlated to obtain the local continuous segment; Based on the differences in position and direction of the trajectory abrupt change segments and the local continuous segments, the trajectory positions in the trajectory abrupt change segments are sorted by the degree of abnormality, and then trajectory anomaly points are screened out.

[0011] In this embodiment, the continuity correction of the trajectory anomaly points is performed through the candidate operation and the control interaction state to obtain the corrected trajectory, specifically including: Operation constraints are constructed based on the operation order of the candidate operations and the interaction state of the control; The abnormal trajectory segments are obtained by judging the trajectory state of the abnormal points through the operational constraints. The trajectory connection direction is determined based on the trajectory change characteristics at both ends of the abnormal trajectory segment and the operational constraints; The abnormal trajectory segment is continuously reconstructed according to the trajectory connection direction to obtain the corrected trajectory.

[0012] In this embodiment, performing time-series matching and behavior correction on the correction trajectory and the operation event sequence to obtain the keyboard and mouse operation intent specifically includes: The corrected trajectory is segmented, and the position change, movement direction and pause duration of each trajectory segment are extracted to obtain the keyboard and mouse behavior characteristics; Based on the keyboard and mouse behavior characteristics, the sequence of operation events is time-series matched to obtain a behavior matching sequence; The operation state is corrected based on the differences in behavioral characteristics between operation events in the behavior matching sequence to obtain the keyboard and mouse operation state, and then the keyboard and mouse operation intention is determined based on the keyboard and mouse operation state.

[0013] In this embodiment, the keyboard and mouse terminal refers to a keyboard input device and a mouse input device for inputting operation commands to a computer terminal; the application interface refers to a graphical user interface that runs on the computer terminal and responds to the keyboard and mouse terminal operations.

[0014] In this embodiment, determining the keyboard and mouse operation intent based on the keyboard and mouse operation state specifically includes: Based on the operation sequence, operation position, and behavioral characteristics in the keyboard and mouse operation states, the operation process is simulated to obtain the operation state simulation results; The simulation results of the operation state and the keyboard and mouse operation state are analyzed to obtain the intent discrimination features. The keyboard and mouse operation states are filtered based on the intent discrimination features to obtain the keyboard and mouse operation intent.

[0015] The technical solutions provided by the embodiments disclosed in this application have the following beneficial effects: The technical solution provided in this application enables collaborative recognition of keyboard and mouse trajectories and interface operation states. First, by acquiring multi-source interaction data from the keyboard and mouse terminal and state images of the application interface, and performing time-series analysis on the multi-source interaction data, keyboard and mouse trajectory features are obtained, achieving a trajectory-based representation of the user's continuous keyboard and mouse operation process. Second, by collecting the control positions and interaction states of the application interface, and combining this with the state images for control operation recognition, operation event sequences and candidate operations are obtained, establishing a correlation between keyboard and mouse trajectory analysis and the actual interface control operation states, improving the accuracy of operation event recognition in complex interaction scenarios. Then, by continuously correcting trajectory anomalies using the candidate operations and control interaction states, a corrected trajectory is obtained. The interface operation states are used to constrain and correct the keyboard and mouse trajectory, reducing the impact of local trajectory anomalies caused by rapid movement, continuous clicking, and dragging on the recognition results. Finally, by performing time-series matching and behavior correction on the corrected trajectory and the operation event sequences, the keyboard and mouse operation intent is obtained, achieving collaborative analysis of keyboard and mouse trajectories and interface operation states, reducing misjudgments of intent caused by single trajectory features, and improving the accuracy and reliability of keyboard and mouse operation intent recognition.

[0016] In summary, the technical solution adopted in this application can achieve collaborative recognition of keyboard and mouse trajectories and interface operation status, thereby improving the accuracy of keyboard and mouse operation intent recognition. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of an AI-based method for keyboard and mouse trajectory recognition and operation intent analysis provided in this application; Figure 2 This is a schematic diagram of keyboard and mouse interaction and application interface changes provided in this application; Figure 3 This is an exemplary flowchart of determining the correction trajectory provided in this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Example 1: To better understand the technical solution of this application, the above technical solution will be described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 As shown in the figure, this is a flowchart of an AI-based keyboard and mouse trajectory recognition and operation intent analysis method according to this embodiment of the present application. The analysis method includes the following steps: In step S1, multi-source interaction data of the keyboard and mouse terminal and the status image of the application interface are acquired, and time-series analysis is performed on the multi-source interaction data to obtain keyboard and mouse trajectory features.

[0021] It should be noted that the keyboard and mouse terminal refers to the keyboard input device and mouse input device used to input operation commands into the computer terminal; the application interface refers to the graphical user interface running on the computer terminal and responding to the keyboard and mouse terminal operations; preferably, refer to Figure 2 As shown in the figure, this is a schematic diagram of a keyboard and mouse interaction and application interface change scenario provided in this application. The keyboard and mouse terminal includes a keyboard input device and a mouse input device. The mouse input device includes a left button and a right button for inputting mouse click, move, and other operation commands. The keyboard input device is used to input keyboard key operation commands. These operation commands represent the operation information input from the keyboard and mouse terminal to the computer terminal. Status feedback indicates the interface state changes generated after the computer terminal responds to the application interface according to the operation commands. The dashed elliptical area in the figure represents the trajectory area formed during mouse movement. The computer terminal changes the control interaction state of the application interface according to the received operation commands and... The changed interface state is fed back to the display device for display. For example, the application interface can be a file management interface, which has file controls such as "document.txt", "photo.jpg", "installation package", and "music.mp3" to illustrate different types of interactive interface controls. When the user moves the mouse to the file control corresponding to the installation package (i.e., the selected state) and performs a mouse click operation, the keyboard and mouse terminal generates a corresponding mouse position change and mouse click event. After receiving the mouse click event, the computer terminal switches the "installation package" file control from the unselected state to the selected state, thus forming an interface state change corresponding to the mouse operation, and the state feedback is sent through the display device.

[0022] In this embodiment, the acquisition of multi-source interaction data and application interface status images of the keyboard and mouse terminal can be achieved in the following ways: Collect interaction records during the operation of the keyboard and mouse terminal, and perform event parsing and time stamping on the interaction records to obtain multi-source interaction data of the keyboard and mouse terminal; The interface images are collected during the continuous change of the application interface, and the interface images are segmented into states based on the time stamps of the multi-source interaction data to obtain the state images of the application interface.

[0023] In practice, the system first records keyboard key presses, mouse movements, mouse presses, mouse releases, and scroll wheel operations using a keyboard driver and a mouse event listener. Each record is then written with the operation type, occurrence time, and operation location. The keyboard driver receives keyboard input signals and transmits key information to the computer terminal, while the mouse event listener records mouse movements, key presses, and scroll wheel operations. The records are then arranged in chronological order of occurrence, and keyboard and mouse operation records are stored separately according to their operation type. For keyboard and mouse operation records occurring within the same time frame, the operation type, operation location, and occurrence time are merged into the same interactive data record, and these consecutively arranged interactive data records are treated as multi-source interactive data. Then, the screen capture program continuously reads the currently displayed content of the computer terminal at a fixed acquisition cycle to obtain interface images arranged by time. The acquisition time of each interface image is correlated with the operation time in the multi-source interactive data. The operation time of each interactive data record is used as a time boundary point, and interface images between two adjacent time boundary points are classified into the same time segment. For time segments containing mouse movement, mouse button presses, or keyboard input, the display values ​​of corresponding pixel positions in adjacent interface images are compared frame by frame. Pixel positions where display changes occur are recorded as changed pixel positions, and they are merged according to the spatial range of the changed pixel positions in the interface images. When the same spatial range in consecutive interface images continuously shows display changes, the interface image corresponding to that spatial range and its acquisition time are retained as a state image. The state images corresponding to each time segment are arranged according to the occurrence time of the interactive data records to obtain a set of application interface state images.

[0024] It should be noted that multi-source interactive data is a collection of interactive data records containing operation type, operation location, and operation time. Operation type includes keyboard key presses, mouse movement, mouse press, mouse release, and scroll wheel operations. Operation location refers to the screen coordinates when the mouse operation occurs. If no mouse position is recorded in the keyboard operation record, the corresponding keyboard operation type and operation time are retained. The state image is the interface image corresponding to the keyboard and mouse operation time, which includes at least the interface display content before and after the keyboard and mouse operation. By mapping the operation time of the interactive data records to the acquisition time of the interface images, keyboard and mouse operation records and interface display changes within the same time period can be placed in the same time series.

[0025] In this embodiment, the timing analysis of the multi-source interaction data to obtain the keyboard and mouse trajectory features can be performed in the following manner: Based on the mouse position and interaction time in the multi-source interaction data, the data is sorted in time sequence, and the position change and time interval between adjacent interaction positions are calculated to obtain trajectory change characteristics. The continuous interactive positions are divided into segments based on the trajectory change characteristics, and the trajectory continuity characteristics are determined based on the position change and direction change between adjacent interactive segments. By performing a time-series mapping between the trajectory change features and the trajectory continuity features, the keyboard and mouse trajectory features are obtained.

[0026] In specific implementation, firstly, mouse operations are arranged sequentially according to their occurrence time in the multi-source interactive data, and the position coordinates of each mouse operation are read. The horizontal and vertical coordinate differences between adjacent mouse operations are calculated, and the square root of the sum of the squares of these differences is used to obtain the position change between adjacent interactive positions. Simultaneously, the occurrence time of the subsequent mouse operation is subtracted from the occurrence time of the preceding mouse operation to obtain the time interval between adjacent interactive positions. The position change and time interval are used as trajectory change features. Then, continuous interactive positions are divided into segments according to the position change and time interval of consecutive mouse operations. Interactive positions with continuous position changes and adjacent time intervals within the same time range are grouped into the same interactive segment. The horizontal and vertical coordinate differences between the start and end positions of adjacent interactive segments are calculated to obtain the position change between adjacent interactive segments. Finally, the coordinate change directions of adjacent interactive positions within the same interactive segment are compared to obtain the direction change. The segment position change and direction change are used as trajectory continuity features. Finally, the interaction times corresponding to the trajectory change features and trajectory continuity features are arranged in sequence, and the position change, time interval, segment position change and direction change within the same interaction time range are combined into trajectory records, and the trajectory records are combined into keyboard and mouse trajectory features.

[0027] It should be noted that, in this embodiment, the trajectory change feature describes the change of adjacent operation positions over time in continuous mouse operations, where the position change amount corresponds to the movement amplitude between adjacent mouse positions, and the time interval corresponds to the time span between adjacent mouse operations; the trajectory continuity feature describes the change at the connection position of adjacent trajectory segments, where the segment position change corresponds to the position offset between preceding and subsequent trajectory segments, and the direction change amount corresponds to the change in the movement direction of adjacent trajectory segments; the keyboard and mouse trajectory feature associates continuous position changes with segment connection changes according to the interaction time, so that trajectory changes and trajectory continuity data within the same time range correspond to the same trajectory record.

[0028] In step S2, the positions and interaction states of controls on the application interface are collected. Based on the control positions, interaction states, and state images, control operation recognition is performed to obtain operation event sequences and candidate operations.

[0029] In this embodiment, the position and interaction state of controls on the application interface can be collected in the following ways: Obtain the layout information of the application interface, divide the layout information into control areas, and obtain the control positions; Based on the position of the control, the region of control state change in the state image is extracted to obtain the state change segment; The interaction states of the control are classified according to the state change segments.

[0030] In practice, the process begins by reading the application's layout file or interface control tree to obtain the control types and boundary coordinates corresponding to buttons, input boxes, menus, lists, and scrolling areas. The top-left and bottom-right coordinates of each control are used as its area boundary, and these are arranged according to their horizontal and vertical positions within the application interface to form control position data. Then, the corresponding area in the status image is extracted based on the boundary coordinates of the control area. Adjacent images of the same control area are read in chronological order, and the brightness or color values ​​of corresponding pixels in adjacent images are subtracted point by point. The number of pixels whose difference exceeds an image change threshold is counted. When this number reaches a set threshold, adjacent status images are marked as having undergone a display change, and the time intervals of consecutively marked adjacent status images constitute a status change segment.

[0031] Finally, the start and end times of the state change segment are read, and the mouse position, mouse press time, mouse release time, and keyboard input time within the corresponding time range are read from the multi-source interaction data. When the mouse position first enters the control area and no mouse press event occurs within the control area, the control interaction state corresponding to that time segment is recorded as mouse hover; when the mouse press time is between the start and end times of the state change segment, the corresponding control interaction state is recorded as pressed; when the mouse release time is between the start and end times of the state change segment, the corresponding control interaction state is recorded as released; when the control area is an input box, and the text pixels in the area change after a keyboard input event, the corresponding control interaction state is recorded as content changed; when there are no mouse or keyboard operation records within the state change segment, the corresponding control interaction state is recorded as no interaction. The control area, the start and end times of the state change segment, and the control interaction state corresponding to each control are combined into a control interaction state record.

[0032] It should be noted that in this embodiment, the control position represents the spatial area of ​​the control in the application interface, represented by the boundary coordinates of the control area, so that image changes in continuous state images can be mapped to specific controls; the state change segment represents the time period in which the same control area continuously shows display differences, obtained by comparing pixels between adjacent state images; the control interaction state represents the temporal correspondence between image changes and keyboard and mouse operations, distinguishing the control state before, during, and after the operation based on the sequential relationship between image changes and keyboard and mouse operations, and forming a corresponding interaction state record in combination with the specific control position.

[0033] In this embodiment, the control operation recognition based on the control position, the control interaction state, and the state image to obtain the operation event sequence and candidate operations can be performed in the following manner: Spatial mapping is performed between the state image and the control position to obtain the control area; Operation states are identified based on the control area and the control interaction state to obtain control operation events, and these events are sorted according to their occurrence time to obtain an operation event sequence. Candidate operations are obtained by continuously filtering operations based on the operation sequence in the operation event sequence and the spatial transfer between the control areas.

[0034] In practice, firstly, the image coordinates of the state image are transformed to match the interface coordinates of the control positions, ensuring that the boundary coordinates of the control positions fall within the pixel coordinate range of the state image. Then, the state image is divided into regions according to the transformed boundary coordinates, with the image content within the boundary range of each control serving as the control area. Next, the state records of each control are arranged according to the occurrence time of its interaction. For mouse press and release records occurring sequentially within the same control area, the time from mouse press to mouse release is combined into a mouse click event. When the mouse press occurs followed by continuous mouse position changes within the control area, and a mouse release record appears, the time from mouse press to mouse release is combined into a drag event. For input box controls, when a text pixel change occurs after keyboard input, the keyboard input time and the corresponding text change time are combined into an input event. Mouse hover records, mouse click events, drag events, and input events are written with the operation type, operation position, and occurrence time respectively to form control operation events, which are then arranged in chronological order to form an operation event sequence. Finally, the control areas corresponding to adjacent control operation events are read according to the occurrence time in the operation event sequence. The Euclidean distance between the center positions of adjacent control areas is calculated, and the movement direction angle is calculated based on the difference in horizontal and vertical coordinates between the center positions of the two controls. When the time interval between adjacent operation events does not exceed the continuous operation time threshold, the Euclidean distance between the center positions of the controls does not exceed the continuous operation distance threshold, and the difference in the adjacent movement direction angles does not exceed the continuous operation angle threshold, the two operation events are classified into the same continuous operation segment. The above conditions are repeated to connect operation events until adjacent events do not meet any of the conditions, and the operation events in the same continuous operation segment are taken as candidate operations.

[0035] It should be noted that a control area refers to the image area defined by the control boundaries in the state image, used to represent the interface space range where keyboard and mouse operations actually take place; a control operation event refers to an event record consisting of a control state change caused by a keyboard or mouse operation, as well as the operation time and operation position, used to describe a specific interface interaction action; an operation event sequence refers to a continuous set of events arranged according to the occurrence time of control operation events, used to describe the sequential process of keyboard and mouse operations in the application interface; and a candidate operation refers to a continuous operation process consisting of control operation events with consecutive spatial positions and operation sequences in the operation event sequence, used to represent keyboard and mouse operations with complete operation process characteristics.

[0036] In step S3, abnormal points in the trajectory are identified based on the keyboard and mouse trajectory features. The abnormal points in the trajectory are then continuously corrected using the candidate operations and the control interaction state to obtain the corrected trajectory.

[0037] In this embodiment, the identification of trajectory anomalies based on the keyboard and mouse trajectory features can be carried out in the following manner: The trajectory change features and trajectory continuity features in the keyboard and mouse trajectory features are input into the trained trajectory anomaly discrimination model. The trajectory anomaly discrimination model is used to judge the degree of change in the position of continuous trajectory to obtain the trajectory abrupt change segment. Local continuity analysis is performed on the trajectory abrupt change segment, and the positional and directional changes of the continuous trajectory positions before and after the trajectory abrupt change segment are correlated to obtain the local continuous segment; Based on the differences in position and direction of the trajectory abrupt change segments and the local continuous segments, the trajectory positions in the trajectory abrupt change segments are sorted by the degree of abnormality, and then trajectory anomaly points are screened out.

[0038] In practice, firstly, historical mouse trajectories are divided into normal trajectory samples and abnormal trajectory samples according to manual annotation results. The position change, time interval, segment position change, and direction change corresponding to each trajectory position are used as sample features. Normal and abnormal trajectories are used as category labels. A decision tree binary classification method is used for training. At each classification node, trajectory samples are divided according to the value range of the sample features, so that trajectory samples of the same category enter the same branch as much as possible. After training, the above four features in the current keyboard and mouse trajectory features are sequentially fed into the trajectory anomaly discrimination model. The model outputs the probability that the corresponding trajectory position belongs to an abnormal trajectory. According to the time sequence of the trajectory positions, when the abnormal probability of consecutive trajectory positions reaches the anomaly discrimination threshold, all consecutive trajectory positions are classified into the trajectory change segment. The anomaly discrimination threshold can be taken as the probability value corresponding to the relatively balanced classification results of normal and abnormal trajectories in the training samples, so as to avoid classifying the trajectory change segment based on only a single trajectory position.

[0039] Then, using the continuous trajectory positions not included in the trajectory change section on the front and back sides of the trajectory change section as reference positions, adjacent reference positions are arranged in chronological order. The sum of the absolute values ​​of the difference in the horizontal and vertical coordinates of adjacent reference positions is taken as the position change, and the absolute value of the difference in the movement direction angle of adjacent reference positions is taken as the direction change. The average value of the two types of changes is taken as the local position change threshold and the local direction change threshold, respectively. Adjacent trajectory positions are compared segment by segment from the front and back sides of the trajectory change section towards the trajectory change section. When the position change does not exceed the local position change threshold and the direction change does not exceed the local direction change threshold, the corresponding continuous trajectory position is included in the local continuous section. When any change exceeds the corresponding threshold, the continuous division in the current direction is stopped.

[0040] Finally, the trajectory positions in the trajectory abrupt change section are matched with the temporally adjacent reference positions in the local continuous sections on both sides according to the time sequence. The sum of the absolute values ​​of the difference in horizontal and vertical coordinates between the corresponding positions is taken as the position change difference, and the absolute angle difference between the corresponding movement direction angles is taken as the direction change difference. These are then divided by the local position change threshold and the local direction change threshold, respectively, and the two ratios are added together to determine the degree of anomaly. The trajectory positions in the trajectory abrupt change section are arranged from high to low according to the degree of anomaly, and the trajectory position with the highest degree of anomaly is taken as the trajectory anomaly point. When the local position change threshold or the local direction change threshold is zero, the maximum change in the corresponding reference position is used to replace the zero value.

[0041] It should be noted that the trajectory anomaly discrimination model in this embodiment is a binary classification tool used to distinguish between normal and abnormal mouse trajectory positions. Its input consists of position change, time interval, segment position change, and direction change, and the output is the probability that the trajectory position belongs to an abnormal trajectory. The trajectory mutation segment refers to the range of continuous trajectories where the probability of anomaly in continuous trajectory positions reaches the anomaly discrimination threshold. The local continuous segment refers to the range of continuous trajectories located before or after the trajectory mutation segment, where the position change and direction change of adjacent trajectory positions do not exceed the corresponding local change threshold. Position change difference refers to the degree of offset of the horizontal and vertical coordinates between the trajectory position in the trajectory mutation segment and the reference position. Direction change difference refers to the degree of deflection of the movement direction between the trajectory position in the trajectory mutation segment and the reference position. Anomaly degree is the sum of the ratio of position change difference to the local position change threshold and the ratio of direction change difference to the local direction change threshold, used to characterize the degree of deviation of the trajectory position from the continuous trajectory. The trajectory anomaly point is the trajectory position with the highest anomaly degree in the trajectory mutation segment.

[0042] Preferred, Reference Figure 3 As shown, this figure is an exemplary flowchart for determining a correction trajectory according to the present application. In this embodiment, the trajectory anomaly points are continuously corrected through the candidate operation and the control interaction state to obtain the correction trajectory. This can be achieved through the following steps: In step S31, operation constraints are constructed based on the operation order of the candidate operations and the interaction state of the control; In step S32, the trajectory anomaly points are judged by the operation constraints to obtain the abnormal trajectory segments; In step S33, the trajectory connection direction is determined based on the trajectory change characteristics at both ends of the abnormal trajectory segment and the operation constraints; In step S34, the abnormal trajectory segment is reconstructed continuously according to the trajectory connection direction to obtain the corrected trajectory.

[0043] In practical implementation, firstly, the positions of the controls corresponding to the candidate operations are read according to their order of occurrence. The operation order between adjacent candidate operations is then mapped to the interaction state of the corresponding controls. The order in which the previous control completes its operation and enters the next control, the change in the control's state, and whether the mouse trajectory passes through the corresponding control area are used as operation constraints. Secondly, the positions of trajectory anomalies are mapped point by point to the operation constraints. When a trajectory anomaly is located between the controls corresponding to adjacent candidate operations, and the trajectory change at that position does not correspond to the operation order and interaction state of the preceding and following controls, the continuous trajectory positions containing the trajectory anomaly are divided into an abnormal trajectory segment. Then, the trajectory positions at both ends of the abnormal trajectory segment are read, along with the center positions of the control areas corresponding to the preceding and following candidate operations. The direction angles pointing from the trajectory positions at both ends to the center of the corresponding control area are calculated, and the average direction of the two direction angles is taken as the trajectory connection direction. Alternatively, the direction can be normalized according to the periodicity of the angles to determine the trajectory connection direction. Finally, retain the original trajectory positions at both ends of the abnormal trajectory segment, and establish a connecting line along the trajectory connection direction with the two trajectory positions as the endpoints; according to the time interval of the original trajectory points in the abnormal trajectory segment, insert the trajectory coordinates of the corresponding time positions in sequence on the connecting line, and delete the trajectory coordinates in the abnormal trajectory segment that deviate from the connecting line; rearrange the original trajectory coordinates before and after the abnormal trajectory segment and the replacement trajectory coordinates on the connecting line according to the occurrence time to form the correction trajectory.

[0044] It should be noted that operation constraints refer to the trajectory connection conditions jointly defined by the order of candidate operations, the position of the corresponding control, and the interaction state of the control. They are used to describe the spatial and temporal relationships that should be satisfied between adjacent operations during a continuous keyboard and mouse operation. Abnormal trajectory segments refer to the continuous trajectory range that contains trajectory abnormal points and cannot directly form a continuous correspondence with adjacent candidate operations. The trajectory connection direction refers to the trajectory extension direction jointly defined by the trajectory changes at both ends of the abnormal trajectory segment and the positions of the corresponding controls of the preceding and following candidate operations. It is used to describe the continuous connection direction between the two ends of the abnormal trajectory segment. Correction trajectory refers to the continuous mouse trajectory that can correspond to the order of candidate operations and the interaction state of the control.

[0045] In step S4, the correction trajectory and the operation event sequence are time-series matched and behavior-corrected to obtain the keyboard and mouse operation intention.

[0046] In this embodiment, the timing matching and behavior correction of the correction trajectory and the operation event sequence to obtain the keyboard and mouse operation intention can be carried out in the following way: The corrected trajectory is segmented, and the position change, movement direction and pause duration of each trajectory segment are extracted to obtain the keyboard and mouse behavior characteristics; Based on the keyboard and mouse behavior characteristics, the sequence of operation events is time-series matched to obtain a behavior matching sequence; The operation state is corrected based on the differences in behavioral characteristics between operation events in the behavior matching sequence to obtain the keyboard and mouse operation state, and then the keyboard and mouse operation intention is determined based on the keyboard and mouse operation state.

[0047] In practice, firstly, the continuous mouse positions in the correction trajectory are arranged in order of occurrence time; the time interval, position change, and movement direction between adjacent positions are read; when the time interval between adjacent positions is less than the pause time threshold and the position change is not zero, the next position is added to the current trajectory segment; when the time interval between adjacent positions reaches the pause time threshold, or the difference in movement direction between consecutive positions exceeds the trajectory segment angle threshold, the current trajectory segment ends and the next trajectory segment begins; for the ended trajectory segment, the starting position, ending position, starting time, ending time, movement direction, and pause duration between the ending position and the starting position of the next trajectory segment are recorded, and the above data are combined to form keyboard and mouse behavior characteristics. Then, the occurrence time, operation position, and corresponding control area of ​​each operation event in the operation event sequence are read. The time difference is obtained by subtracting the end time of the trajectory segment from the occurrence time of the operation event. The Euclidean distance between the end position of the trajectory segment and the center position of the control area is used as the position distance. When the absolute value of the time difference does not exceed the event matching time threshold and the position distance does not exceed the event matching distance threshold, the trajectory segment and the operation event are written into the same matching record. When a trajectory segment meets the matching conditions of multiple operation events, the operation event with the smallest absolute value of the time difference is retained. When an operation event corresponds to multiple trajectory segments, the trajectory segment whose end position is closest to the center of the control corresponding to the operation event is retained. The matching records are then arranged in order of the occurrence time of the operation events to form a behavior matching sequence. Finally, the movement direction, position change, and pause duration of adjacent trajectory segments in the behavior matching sequence are read, and the control interaction state of the corresponding operation event is read. When the end position of a trajectory segment is within the corresponding control area, and the difference between the movement direction and the direction of the control corresponding to the previous operation event and the current control does not exceed the behavior direction threshold, the correspondence between the current trajectory segment and the operation event is retained. When the end position of a trajectory segment is outside the corresponding control area, the correspondence between the trajectory segment and the current operation event is released, and the trajectory segment is re-corresponded to the operation event closest to its end position with a time difference not exceeding the event matching time threshold. When the time difference between the end time of the trajectory segment and the occurrence time of the operation event exceeds the event matching time threshold, the current correspondence is released. When the difference between the movement direction and the direction of the adjacent control exceeds the behavior direction threshold, the current trajectory segment is marked as an unmatched trajectory segment, and its position and time are re-corresponded to the operation event adjacent to the next time. The trajectory segments, operation events, operation positions, movement directions, and pause durations with retained correspondences are arranged according to the operation time to form the keyboard and mouse operation state.

[0048] Preferably, in this embodiment, determining the keyboard and mouse operation intention based on the keyboard and mouse operation state can be achieved in the following ways: Based on the operation sequence, operation position, and behavioral characteristics in the keyboard and mouse operation states, the operation process is simulated to obtain the operation state simulation results; The simulation results of the operation state and the keyboard and mouse operation state are analyzed to obtain the intent discrimination features. The keyboard and mouse operation states are filtered based on the intent discrimination features to obtain the keyboard and mouse operation intent.

[0049] In practice, the process begins by reading the operation sequence, position, and behavioral characteristics from the keyboard and mouse operation states. The corresponding mouse movements, mouse button presses, and keyboard inputs are then executed sequentially through the computer terminal's virtual input interface. The application interface state after each operation is captured by a screen capture program. The operation sequence, position, time, and interface state are combined to form the operation state simulation result. Next, the operation state simulation result is compared item by item with the corresponding keyboard and mouse operation states. The absolute values ​​of the horizontal and vertical coordinate differences between the simulated and actual operation positions are added to obtain the position deviation. The absolute difference between the simulated and actual operation times is used to obtain the time deviation. The angular difference between the simulated and actual movement directions is used to obtain the direction difference. The control states after the simulated operation are compared item by item with the actual control states to obtain the control state differences. Finally, the position deviation, time deviation, direction difference, and control state difference are divided by the maximum value in their respective data sets to obtain the normalized position deviation, normalized time deviation, normalized direction difference, and normalized control state difference. The four normalized results are then added together to obtain the state deviation value. The keyboard and mouse operation state with the smallest state deviation value is taken as the keyboard and mouse operation intention.

[0050] It should be noted that, in this embodiment, the behavior matching sequence refers to a continuous data sequence formed by mapping the trajectory segments in the correction trajectory to the operation events in the operation event sequence according to the chronological order. Each operation event corresponds to a corresponding trajectory segment. The keyboard and mouse operation state refers to the state data formed between the operation sequence, operation position, movement direction, and pause duration during the keyboard and mouse operation process, which is used to describe the actual operation process of a continuous keyboard and mouse operation. The operation state simulation result refers to the interface state and operation process data obtained after continuously executing the corresponding operation in the application interface according to the operation sequence, operation position, and behavioral characteristics in the keyboard and mouse operation state. The intent discrimination feature refers to the positional deviation, time deviation, direction difference, and control state difference between the operation state simulation result and the keyboard and mouse operation state, which is used to describe the degree of deviation between the actual keyboard and mouse operation and the corresponding interface operation process. The keyboard and mouse operation intent refers to the operation state that can be mapped to the same interface operation process by the operation sequence, operation position, and behavioral characteristics.

[0051] In summary, the technical solution adopted in this application can achieve collaborative recognition of keyboard and mouse trajectories and interface operation status, thereby improving the accuracy of keyboard and mouse operation intent recognition.

[0052] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

Claims

1. A method for keyboard and mouse trajectory recognition and operation intent analysis based on AI, characterized in that, The analytical method includes the following steps: Acquire multi-source interaction data and application interface status images of the keyboard and mouse terminal, perform time-series analysis on the multi-source interaction data, and obtain keyboard and mouse trajectory features; Collect the position and interaction state of controls on the application interface, and perform control operation recognition based on the control position, the control interaction state and the state image to obtain the operation event sequence and candidate operations; Based on the keyboard and mouse trajectory features, abnormal trajectory points are identified, and the abnormal trajectory points are continuously corrected through the candidate operations and the control interaction state to obtain a corrected trajectory. The correction trajectory and the operation event sequence are matched in time and corrected in behavior to obtain the keyboard and mouse operation intention.

2. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 1, characterized in that, Acquiring multi-source interaction data and application interface status images from keyboard and mouse terminals specifically includes: Collect interaction records during the operation of the keyboard and mouse terminal, and perform event parsing and time stamping on the interaction records to obtain multi-source interaction data of the keyboard and mouse terminal; The interface images are collected during the continuous change of the application interface, and the interface images are segmented into states based on the time stamps of the multi-source interaction data to obtain the state images of the application interface.

3. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 1, characterized in that, The time-series analysis of the multi-source interaction data yields the following keyboard and mouse trajectory features: Based on the mouse position and interaction time in the multi-source interaction data, the data is sorted in time sequence, and the position change and time interval between adjacent interaction positions are calculated to obtain trajectory change characteristics. The continuous interactive positions are divided into segments based on the trajectory change characteristics, and the trajectory continuity characteristics are determined based on the position change and direction change between adjacent interactive segments. By performing a time-series mapping between the trajectory change features and the trajectory continuity features, the keyboard and mouse trajectory features are obtained.

4. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 1, characterized in that, The collection of control positions and interaction states on the application interface specifically includes: Obtain the layout information of the application interface, divide the layout information into control areas, and obtain the control positions; Based on the position of the control, the region of control state change in the state image is extracted to obtain the state change segment; The interaction states of the control are classified according to the state change segments.

5. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 1, characterized in that, Based on the control's position, interaction state, and state image, control operation recognition is performed to obtain an operation event sequence and candidate operations, specifically including: Spatial mapping is performed between the state image and the control position to obtain the control area; Operation states are identified based on the control area and the control interaction state to obtain control operation events, and these events are sorted according to their occurrence time to obtain an operation event sequence. Candidate operations are obtained by continuously filtering operations based on the operation sequence in the operation event sequence and the spatial transfer between the control areas.

6. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 1, characterized in that, Identifying trajectory anomalies based on the aforementioned keyboard and mouse trajectory features specifically includes: The trajectory change features and trajectory continuity features in the keyboard and mouse trajectory features are input into the trained trajectory anomaly discrimination model. The trajectory anomaly discrimination model is used to judge the degree of change in the position of continuous trajectory to obtain the trajectory abrupt change segment. Local continuity analysis is performed on the trajectory abrupt change segment, and the positional and directional changes of the continuous trajectory positions before and after the trajectory abrupt change segment are correlated to obtain the local continuous segment; Based on the differences in position and direction of the trajectory abrupt change segments and the local continuous segments, the trajectory positions in the trajectory abrupt change segments are sorted by the degree of abnormality, and then trajectory anomaly points are screened out.

7. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 1, characterized in that, The trajectory anomaly points are continuously corrected by the candidate operation and the control interaction state to obtain the corrected trajectory, which specifically includes: Operation constraints are constructed based on the operation order of the candidate operations and the interaction state of the control; The abnormal trajectory segments are obtained by judging the trajectory state of the abnormal points through the operational constraints. The trajectory connection direction is determined based on the trajectory change characteristics at both ends of the abnormal trajectory segment and the operational constraints; The abnormal trajectory segment is continuously reconstructed according to the trajectory connection direction to obtain the corrected trajectory.

8. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 1, characterized in that, The timing and behavior correction of the correction trajectory and the operation event sequence are used to obtain the keyboard and mouse operation intentions, specifically including: The corrected trajectory is segmented, and the position change, movement direction and pause duration of each trajectory segment are extracted to obtain the keyboard and mouse behavior characteristics; Based on the keyboard and mouse behavior characteristics, the sequence of operation events is time-series matched to obtain a behavior matching sequence; The operation state is corrected based on the differences in behavioral characteristics between operation events in the behavior matching sequence to obtain the keyboard and mouse operation state, and then the keyboard and mouse operation intention is determined based on the keyboard and mouse operation state.

9. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 1, characterized in that, The keyboard and mouse terminal refers to the keyboard input device and mouse input device that input operation commands to the computer terminal; the application interface refers to the graphical user interface that runs on the computer terminal and responds to the keyboard and mouse terminal operations.

10. The AI-based keyboard and mouse trajectory recognition and operation intent analysis method as described in claim 8, characterized in that, Determining the keyboard and mouse operation intent based on the keyboard and mouse operation status specifically includes: Based on the operation sequence, operation position, and behavioral characteristics in the keyboard and mouse operation states, the operation process is simulated to obtain the operation state simulation results; The simulation results of the operation state and the keyboard and mouse operation state are analyzed to obtain the intent discrimination features. The keyboard and mouse operation states are filtered based on the intent discrimination features to obtain the keyboard and mouse operation intent.

Images