A user operation behavior monitoring method, system, device and medium

Abstract

The present application relates to a user operation behavior monitoring method, system, device and medium, and the monitoring method includes: obtaining all user operation windows and window attribute data in the target device; detecting and determining the current active window; intercepting the window content image of the current active window and associating it with the current timestamp to obtain the current reference frame image; obtaining user operation behavior data related to the current active window and associating it with the current reference frame image; if there is a window change event in the window content image of the current active window, intercepting the window content image to update the current reference frame image, and obtaining new user operation behavior data and associating it with the updated reference frame image; based on the time series, recording the user operation behavior data as a user operation log, and generating a monitoring video file based on the continuously updated reference frame image. While reducing storage pressure, the present application retains the dynamics and details of user operations.

Description

Technical Field

[0001] The present application relates to the field of computer monitoring technology, and in particular to a method, system, device and medium for monitoring user operation behavior. Background Art

[0002] In technical support and IT operation and maintenance, helping users troubleshoot and solve problems in software operation is a common and important task. With the continuous development of information technology, users and enterprises are increasingly dependent on software systems, and the stability of the system and the smoothness of user operations have become particularly important. However, when faced with operational failures or technical problems, how to accurately locate the root cause of the problem has become a challenge. In practice, a common way to solve this problem is to record the user's operation process for easy tracing and analysis.

[0003] Current solutions mainly rely on screen recording technology. By recording screen videos of user operations, technical support personnel can trace back the user's specific operation steps and analyze the cause of the problem. However, when recording the complete video screen content, especially long-term, high-frequency operation recording, a large amount of file data will be generated. For long-term operation monitoring, the storage demand for video files will increase dramatically, which will put a lot of pressure on storage devices and may cause a waste of system storage resources.

[0004] In addition, traditional video recording methods often record all user operations, most of which are irrelevant to the problem. For example, when troubleshooting software operation problems, users' browsing between interfaces, idle waiting, or other irrelevant operations will also be recorded. This not only makes the video file huge, but also reduces the effectiveness of the video content, wasting storage space and computing resources.

[0005] Therefore, how to reduce the storage of redundant data and improve resource utilization while ensuring efficient recording of user operations has become a problem that needs to be solved urgently. Summary of the invention

[0006] In order to achieve efficient and accurate operation recording and playback, the present application provides a user operation behavior monitoring method, system, device and medium.

[0007] In the first aspect, the present application provides a user operation behavior monitoring method, system, device and medium, which adopts the following technical solutions: A method for monitoring user operation behavior, the monitoring method comprising: Get all user operation windows and window attribute data in the target device; Detect and determine the current active window based on the user's current input focus; Intercepting a window content image of the current active window, and associating it with a current timestamp to obtain a current reference frame image; Acquire user operation behavior data related to the current active window, and associate the user operation behavior data with the current reference frame image; If there is a window change event in the window content image of the currently active window, the window content image is intercepted to update the current reference frame image, and new user operation behavior data is obtained to be associated with the updated reference frame image; wherein the window change event includes a window switching event, a window enlargement event, a window reduction event, and a window closing event; Based on the time series, the acquired user operation behavior data is recorded as a user operation log, and a monitoring video file is generated according to the continuously updated reference frame image; wherein the monitoring video file includes the continuously updated reference frame image and the user operation log associated with the reference frame image.

[0008] By adopting the above technical solution, the target device window information is obtained, the user operation behavior is captured, the reference frame is dynamically updated, and the operation trajectory monitoring video is generated, thereby realizing an efficient, accurate and storage-optimized user operation behavior monitoring method. This application greatly reduces the storage pressure through the "reference frame + operation log" structure, while retaining the dynamics and details of user operations. By establishing an association between timestamps and operation ranges, the storage requirements of redundant reference frame images are greatly reduced. New frames are generated only when the window changes significantly, while ensuring that each operation behavior log can be associated with the correct window content, so that the dynamic operation trajectory of the playback is consistent with the actual operation scenario.

[0009] Optionally, the user operation behavior data includes keyboard input events, mouse click events, mouse drag events and wheel scrolling events.

[0010] Optionally, the step of determining whether there is a window change event for the window content image of the currently active window includes: Get the window attribute data of the currently active window in real time; Based on the current reference frame image, determine whether the window ID in the window attribute data has changed. If so, there is a window switching event / window closing event; If not, it is determined whether the window size in the window attribute data has changed based on the current reference frame image. If so, a window enlargement event / window reduction event exists.

[0011] By adopting the above technical solutions, based on real-time window attribute monitoring, combined with the dual detection logic of window ID and window size changes, we can ensure fast and accurate judgment of window change events. Whether it is window switching, closing or resizing, it can trigger the update of the base frame in an efficient and accurate manner, which not only optimizes resource utilization, but also provides a strong guarantee for the integrity of monitoring records.

[0012] Optionally, based on the time series, the steps of recording the acquired user operation behavior data as a user operation log, and generating a monitoring video file according to the continuously updated reference frame image include: Initialize the user operation log list; The acquired user operation behavior data is structuredly stored based on a time series, and the user operation behavior data is appended to a user operation log associated with a current reference frame image; Marking the continuously updated reference frame images by time segment based on the time series, embedding the user operation log into the corresponding time period, and generating a time-series operation track; Loading the time-series operation trajectory into a video generation module; The reference frame image is used as the background of the video frame, and the user operation log is superimposed on the video frame in the form of animation to generate a dynamic and visualized monitoring video file.

[0013] By adopting the above technical solution, user operation behaviors are organized based on time series, ensuring that each user operation log corresponds to the reference frame image, and time segmentation marking and log embedding realize the complete operation track, ensuring the rigor of temporal logic; using the reference frame image as the background, combined with the animation effect of the user operation log, an intuitive monitoring video is generated, which is convenient for users to trace back key operation points. At the same time, the separate storage and time serialization of the reference frame image and the user operation log provide strong technical support for complex monitoring of multiple windows and multiple scenes.

[0014] Optionally, the step of marking the continuously updated reference frame images by time segments based on the time sequence includes: Traversing continuously updated reference frame images, sorting the reference frame images based on a time sequence, and determining a window change event type corresponding to each reference frame image; Locating a generation timestamp of each of the reference frame images, and dividing a time period corresponding to each reference frame image; The time period corresponding to each reference frame image is marked according to the window change event type to obtain a timeline summary of each time period; wherein the timeline summary includes the start and end time of each time period and the window change event type.

[0015] By adopting the above technical solution, through time series sorting, time period division and event type marking, refined time management of reference frames and clear display of operation changes are achieved. This technical solution is particularly suitable for scenarios such as timeline generation, operation playback and behavior analysis of monitoring video files, providing users with efficient and intuitive analysis tools and data support.

[0016] Optionally, after the step of generating the surveillance video file, the following steps are further included: Classifying the generated surveillance video files according to the window attribute data; Determine the target storage path of each surveillance video file based on the classification result; The surveillance video file is stored in the target storage path and the classification index is updated.

[0017] By adopting the above technical solution, with window attribute data as the core, the full process management of surveillance video files is realized through classification, path generation, storage and index update, which not only greatly improves the organization of storage and retrieval efficiency, but also provides solid technical support for the long-term stable operation of the system.

[0018] In the second aspect, the present application provides a user operation behavior monitoring system, which adopts the following technical solution: A user operation behavior monitoring system, the monitoring system comprising: A window acquisition module is used to acquire all user operation windows and window attribute data in the target device; An active window determination module is used to detect and determine the current active window according to the user's current input focus; A reference frame generation module, used to capture the window content image of the current active window and associate it with the current timestamp to obtain a current reference frame image; An associating module, used to obtain user operation behavior data related to the current active window, and associate the user operation behavior data with the current reference frame image; A data processing module, configured to intercept the window content image to update the current reference frame image when a window change event occurs in the window content image of the current active window, and obtain new user operation behavior data to associate with the updated reference frame image; wherein the window change event includes a window switching event, a window enlargement event, a window reduction event, and a window closing event; A monitoring video file generation module is used to record the acquired user operation behavior data as a user operation log based on a time series, and generate a monitoring video file according to a continuously updated reference frame image; wherein the monitoring video file includes a continuously updated reference frame image and a user operation log associated with the reference frame image.

[0019] Optionally, the monitoring system further includes: A classification module, used for classifying the generated surveillance video file according to the window attribute data; A storage path determination module, used to determine a target storage path for each surveillance video file based on the classification result; The file storage module is used to store the monitoring video file to the target storage path and update the classification index.

[0020] In a third aspect, the present application provides a computer device, which adopts the following technical solution: A computer device comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method as described in the first aspect.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the methods in the first aspect.

[0022] To summarize, the present application includes at least one of the following beneficial technical effects: by comprehensively capturing window change events such as window switching, zooming in, zooming out and closing, and highly correlating user operation logs with reference frame images, the monitoring video file can intuitively display the user's operating behavior in different time periods. The entire monitoring process is structured, automated and highly dynamic, meeting the user behavior monitoring needs in complex operation scenarios with multiple windows and multiple tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a first flow chart of a method for monitoring user operation behavior according to one of the embodiments of the present application.

[0024] Figure 2 This is a second flow chart of a method for monitoring user operation behavior according to one of the embodiments of the present application.

[0025] Figure 3 This is a third flow chart of a method for monitoring user operation behavior according to one of the embodiments of the present application.

[0026] Figure 4 This is a fourth flow chart of a method for monitoring user operation behavior according to one of the embodiments of the present application.

[0027] Figure 5 This is the fifth flow chart of a method for monitoring user operation behavior according to one of the embodiments of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-5 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0029] The embodiment of the present application discloses a method for monitoring user operation behavior.

[0030] Reference Figure 1 , a user operation behavior monitoring method, the monitoring method comprising: Step S101, obtaining all user operation windows and window attribute data in the target device; When the monitoring system is started, the system API of the target device is first called to obtain the information and attribute data of all currently running user operation windows, including window title, window ID, window size and position, etc. By obtaining all window attributes, the foundation is laid for the subsequent positioning of active windows; Step S102, detecting and determining the current active window according to the user's current input focus; Among them, the active window is the window that the user is currently operating. By detecting the user's input focus (such as the window clicked by the mouse or the target window of the keyboard input), it is determined which window is in the "active" state, so as to achieve accurate monitoring of user operations and ensure that only the windows with which the user actually interacts are recorded, avoiding interference with other windows or redundant data recording.

[0031] Step S103, capturing the window content image of the current active window and associating it with the current timestamp to obtain the current reference frame image; Specifically, the image capture technology (such as the screenshot function) is used to capture the content area of ​​the current active window and extract the static image as the "reference frame image". The reference frame image is used as the starting point of the user operation trajectory and the current timestamp is recorded for subsequent data association and reconstruction.

[0032] The screenshot process may use a screen capture API provided by the operating system, or directly obtain window pixel data through a video memory copy technique.

[0033] Step S104, obtaining user operation behavior data related to the current active window, and associating the user operation behavior data with the current reference frame image; Among them, user operation behavior data includes keyboard input events, mouse click events, mouse drag events and wheel scrolling events; keyboard input events include the input characters, key types and corresponding positions (such as input boxes or specific controls); mouse click events include the screen coordinates of the click, the target control and the triggered event (such as button clicks); mouse drag events include the starting and end point coordinates, as well as the content affected during the dragging process (such as selecting text or dragging icons); wheel scrolling includes the wheel direction and scrolling amount, reflecting the user's scrolling operation.

[0034] Specifically, by capturing the user's operating behaviors on the active window, including keyboard input (such as text input), mouse clicks (such as button clicks) and other events, the captured user behaviors are associated with the reference frame image to form a preliminary operation record, thereby achieving accurate recording of user operating behaviors and ensuring the integrity of the operation trajectory.

[0035] Step S105, if there is a window change event in the window content image of the current active window, the window content image is intercepted to update the current reference frame image, and new user operation behavior data is obtained and associated with the updated reference frame image; Among them, the window change event includes the window switching event, the window enlargement event, the window reduction event and the window closing event; Specifically, it detects changes in window content in real time to determine whether window switching, enlargement, reduction, or closing occurs; if changes occur, it captures the new window content as the updated reference frame and associates the new operation behavior. By dynamically updating the reference frame, it ensures accurate capture of window changes and provides consistency for the playback of subsequent operation logs.

[0036] It is understandable that if the window content image has not changed or has only slight changes (such as scrolling the page or changes in part of the area), for example, the user operation is a local change such as "add, delete, modify, and query", the current reference frame image can continue to be used, and the new user operation behavior can continue to be associated with the current reference frame image.

[0037] Step S106, based on the time series, recording the acquired user operation behavior data as a user operation log, and generating a monitoring video file according to the continuously updated reference frame image; The monitoring video file includes a continuously updated reference frame image and a user operation log associated with the reference frame image.

[0038] Specifically, the reference frame image and the user operation log are integrated in time series to generate a visual monitoring video file, which contains multiple reference frame images and operation behaviors, and can replay the user's complete operation process according to time points. Among them, the reference frame image is used as the static background of the user's operation, and the operation behavior log is used as the dynamic operation track, and the two together constitute a complete operation record.

[0039] It can be understood that video files realize dynamic visualization of user operation trajectories, which is convenient for subsequent backtracking and analysis. The time-serialized recording method optimizes retrieval efficiency and supports rapid positioning of specific operation points.

[0040] It should be noted that each reference frame image can correspond to multiple user operation logs (all operations within the time period). For example, reference frame image A can correspond to the following logs within the time period [10:00:00,10:05:00]: 10:00:05: click the login button with the mouse; 10:01:30: enter "username123" on the keyboard. However, each operation behavior log can only correspond to one reference frame image (that is, the effective reference frame when the operation occurs) to ensure that the operation behavior log will not be associated with multiple frames at the same time to avoid confusion during playback. Only when the reference frame image is updated will the new operation behavior log automatically switch to the new reference frame image. For example, when the window is switched, the reference frame image is updated from A to B, and the subsequent operation behavior will be associated with the reference frame image B.

[0041] In the above implementation, the target device window information is obtained, the user operation behavior is captured, the reference frame is dynamically updated, and the operation trajectory monitoring video is generated, thereby realizing an efficient, accurate and storage-optimized user operation behavior monitoring method. This application greatly reduces the storage pressure through the "reference frame + operation log" structure, while retaining the dynamics and details of user operations. By establishing an association between timestamps and operation ranges, the storage requirements of redundant reference frame images are greatly reduced, and new frames are generated only when the window changes significantly. At the same time, it ensures that each operation behavior log can be associated with the correct window content, so that the dynamic operation trajectory of the playback is consistent with the actual operation scenario, providing users with efficient and intuitive analysis tools and data support.

[0042] In some embodiments, the tool uses the MQTT transmission protocol, which is a lightweight message transmission protocol with a relatively small number of transmission bytes. It can effectively improve data transmission efficiency and reduce the load on the network and server, especially in resource-constrained environments.

[0043] In practical applications, the technical solution of this application can cope with complex operation scenarios with multiple windows and multiple changes, ensure the integrity and consistency of operation records, and the generated video files can intuitively reflect the user's operation process, providing an accurate, low-cost and efficient solution for the effective monitoring of user operation behavior.

[0044] Reference Figure 2 As an implementation of step S105, the step of determining whether there is a window change event for the window content image of the current active window includes: Step S201, obtaining window attribute data of the current active window in real time; Among them, the system obtains the attribute data of the currently active window by calling the window management API of the operating system. The currently active window is the window that the user is currently operating based on the user's current input focus; the window attribute data includes window ID, window title, window size and position, etc.

[0045] Step S202, judging whether the window ID in the window attribute data has changed based on the current reference frame image, if so, jumping to step S203; if not, jumping to step S204; Step S203: There is a window switching event / window closing event; Specifically, the window ID is an identifier that uniquely identifies a window and is used to distinguish different windows. By comparing the window ID currently acquired in real time with the window ID associated with the reference frame image, it is determined whether the window has been switched or closed. When the window ID changes to another ID, it means that the user has switched to another window, that is, there is a window switching event; when the window ID associated with the reference frame image cannot be detected, it means that the user has closed the window, that is, there is a window closing event. By quickly determining whether the user has switched to a new window or closed the current window, the dynamic update of the reference frame image is ensured.

[0046] Step S204, judging whether the window size in the window attribute data has changed based on the current reference frame image, if so, jumping to step S205; if not, jumping to step S206; Step S205: There is a window enlargement event / window reduction event; Specifically, if the window ID has not changed, the window size (width, height) in the real-time window attribute data is further compared with the window size associated with the reference frame to determine whether the window has been enlarged or reduced. If the change in width or height exceeds the set threshold, it is determined that a window enlargement or reduction event has occurred. By effectively sensing the user's operation of adjusting the window size, the reference frame image is ensured to be updated in time to match the new state of the window content.

[0047] Step S206: No window change event.

[0048] In the above implementation, based on real-time window attribute monitoring, combined with dual detection logic of window ID and window size change, fast and accurate judgment of window change events is ensured. Whether it is window switching, closing or resizing, the reference frame update can be triggered in an efficient and accurate manner, which not only optimizes resource utilization, but also provides a strong guarantee for the integrity of monitoring records.

[0049] Reference Figure 3 As an implementation of step S106, based on the time series, the acquired user operation behavior data is recorded as a user operation log, and the steps of generating a monitoring video file according to the continuously updated reference frame image include: Step S301, initializing the user operation log list; When the system starts, the user operation log list is initialized to record the user operation behavior data associated with each reference frame. Each log list uniquely corresponds to a reference frame image to ensure structured management of log data.

[0050] Specifically, an empty list (or data structure) can be created to store user operation behaviors and record the metadata of the current reference frame (such as timestamp, window ID, window title) to ensure that the operation behavior of each reference frame has an independent management unit for easy classification and backtracking.

[0051] Step S302, performing structured storage on the acquired user operation behavior data based on the time series, and appending the user operation behavior data to the user operation log associated with the current reference frame image; The user operation behavior data captured in real time is stored in a structured manner, including: operation type (such as mouse click, keyboard input), operation target (such as control name, screen coordinates) and timestamp of the operation. The user operation behavior data can be appended to the log associated with the current reference frame image through the timestamp to ensure the timing of the operation behavior.

[0052] Step S302, based on the time sequence, the continuously updated reference frame image is time segmented and marked, and the user operation log is embedded into the corresponding time period to generate a time-series operation track; The reference frame images are segmented and marked according to the generation time to ensure that each reference frame has a clear time range. By embedding the user operation log into the corresponding time period, a complete time-serialized operation trajectory is formed.

[0053] Specifically, the recorded reference frame images and user operation logs are traversed, and a time period is associated with each reference frame, including the start time (the timestamp generated by the current reference frame) and the end time (the timestamp generated by the next reference frame or the effective end time of the current reference frame). The user operation log is then embedded into the corresponding time period to form an operation track with a complete time sequence, providing a clear time node for video generation.

[0054] Step S303, loading the time-series operation trajectory into the video generation module; Among them, the time-serialized operation trajectory data is input into the video generation module. The operation trajectory data includes the reference frame image file path and the operation log data of each period of time. The video generation module is used to map the operation trajectory to the time range corresponding to the reference frame to achieve seamless connection between the operation behavior data and video generation.

[0055] Step S304 , using the reference frame image as the background of the video frame, superimposing the user operation log on the video frame in the form of animation, and generating a dynamic visual monitoring video file.

[0056] Among them, the reference frame image is loaded into the video track as a static background, and the operation behavior is superimposed and displayed in the form of animation within the corresponding time period according to the timestamp of the operation log, including the visual effects of mouse clicks (such as cursor highlighting, click animation) and dynamic display of keyboard input content.

[0057] Specifically, a reference frame image is rendered for each time range as a static background, and then the operation logs are read in sequence to generate animation effects. Finally, all time periods are merged to generate a complete video file.

[0058] In the above implementation, user operation behaviors are organized based on time series to ensure that each user operation log corresponds to the reference frame image. Time segmentation marking and log embedding realize the complete operation track and ensure the rigor of temporal logic. The reference frame image is used as the background, combined with the animation effect of the user operation log, to generate an intuitive monitoring video, which is convenient for users to trace back key operation points. At the same time, the separate storage and time serialization of the reference frame image and the user operation log provide powerful technical support for complex monitoring of multiple windows and multiple scenes.

[0059] Reference Figure 4 As an implementation of step S302, the step of marking the continuously updated reference frame images by time segments based on the time sequence includes: Step S401, traversing continuously updated reference frame images, sorting the reference frame images based on a time sequence, and determining a window change event type corresponding to each reference frame image; Among them, the continuously generated reference frame images are traversed, the generation timestamp and associated window attribute data (such as window ID, window title, window size and position) of each reference frame image are extracted, and the reference frame images are sorted by timestamp to ensure that the reference frames are arranged in the generation order. By analyzing the difference in window attributes between each reference frame image and the previous reference frame image, the window change event type of the reference frame update is determined, such as window switching event, window enlargement event, window reduction event and window closing event.

[0060] Step S402, locating the generation timestamp of each reference frame image, and dividing the time period corresponding to each reference frame image; Among them, the generation timestamp of each reference frame is extracted to determine the effective time period of the frame; the time period of the reference frame image is divided into: start time (generation timestamp of the current reference frame) to end time (generation timestamp of the next reference frame or effective end time of the current frame); it should be noted that if the reference frame is the last frame, the end time is the monitoring end time.

[0061] Step S403, marking the time period corresponding to each reference frame image according to the window change event type to obtain a timeline summary of each time period; wherein the timeline summary includes the start and end time of each time period and the window change event type.

[0062] The time period of each reference frame image is associated with the window change event type to generate a timeline summary, which helps users quickly understand the main operation changes within each period of time.

[0063] In the above implementation, through time series sorting, time period division and event type marking, refined time management of the reference frame and clear display of operation changes are achieved. This technical solution is particularly suitable for scenarios such as timeline generation, operation playback and behavior analysis of monitoring video files, and provides users with efficient and intuitive analysis tools and data support.

[0064] Reference Figure 5 As a further implementation of the monitoring method, after the step of generating a monitoring video file, the method further includes: Step S501, classifying the generated surveillance video files according to the window attribute data; Specifically, after the surveillance video file is generated, the system can classify the video file according to the window attribute data, and the classification basis includes program name, window title or window ID. Among them, the program name identifies the application that generates the video, the window title indicates the specific window content, and the window ID is used to distinguish multiple windows of the same program.

[0065] In some embodiments, priority may be given to classifying by program name to facilitate distinguishing video files by application, and further subdivided according to window title under program name classification; if the same program generates monitoring files for multiple windows, a window ID is attached under the window title for distinction.

[0066] Step S502, determining a target storage path for each surveillance video file based on the classification result; Among them, according to the classification results, a file path for classified storage is constructed, and the specific path structure includes a root directory and a subdirectory; the root directory is a unified root path for storing all surveillance videos, and the subdirectory is a classified subdirectory generated according to the window title or program name.

[0067] In the embodiment of the present application, the automatic generation logic of the storage path is clear and flexible, and can effectively adapt to the scenarios of different window attribute data combinations.

[0068] Step S503, storing the surveillance video file to the target storage path and updating the classification index.

[0069] Among them, the generated surveillance video files are moved to the target storage path to ensure the classification results are implemented. The naming rules of the storage files can include a brief description of the window title and the generation timestamp. In addition, in order to improve the efficiency of file retrieval, a corresponding classification index can be generated to record the storage path, reference frame time period and window attribute data of each video file. After each new file is stored, the index data structure is updated immediately to reflect the latest storage information.

[0070] In the above implementation, with window attribute data as the core, the whole process management of monitoring video files is realized through classification, path generation, storage and index update, which not only greatly improves the organization of storage and retrieval efficiency, but also provides solid technical support for the long-term stable operation of the system.

[0071] The technical solution of this application can fully capture window change events such as window switching, zooming in, zooming out, and closing, and highly correlate the user operation log with the reference frame image, so that the monitoring video file can intuitively display the user's operation behavior in different time periods. The time series-based recording method improves the retrieval efficiency of the operation log, supports rapid positioning of specific operation points and accurate backtracking. The entire monitoring process is structured, automated, and highly dynamic, meeting the monitoring needs in complex operation scenarios with multiple windows and multiple tasks.

[0072] The embodiment of the present application also discloses a user operation behavior monitoring system.

[0073] A user operation behavior monitoring system, the monitoring system comprising: A window acquisition module is used to acquire all user operation windows and window attribute data in the target device; An active window determination module is used to detect and determine the current active window according to the user's current input focus; A reference frame generation module is used to capture the window content image of the current active window and associate it with the current timestamp to obtain the current reference frame image; An association module, used to obtain user operation behavior data related to the current active window, and associate the user operation behavior data with the current reference frame image; A data processing module, used for capturing the window content image to update the current reference frame image when a window change event occurs in the window content image of the current active window, and obtaining new user operation behavior data to associate with the updated reference frame image; Among them, the window change event includes the window switching event, the window enlargement event, the window reduction event and the window closing event; The monitoring video file generation module is used to record the acquired user operation behavior data as a user operation log based on a time series, and generate a monitoring video file according to the continuously updated reference frame image; wherein the monitoring video file includes the continuously updated reference frame image and the user operation log associated with the reference frame image.

[0074] In the above implementation, by accurately locking the active window of the user's current operation and generating a high-precision window snapshot in combination with a timestamp, a static background support is provided for the dynamic association and recording of the operation behavior, the user's operation behavior data is captured in real time and the reference frame image is dynamically updated to ensure the consistency of the operation log and the window content. Through the time-serialized operation trajectory, the continuously updated reference frame image is integrated with the user operation log to generate a monitoring video file, thereby realizing the dynamic visualization and record playback of the user's operation trajectory.

[0075] As a further embodiment of the monitoring system, the monitoring system further includes: A classification module, used to classify the generated surveillance video files according to the window attribute data; A storage path determination module, used to determine a target storage path for each surveillance video file based on the classification result; The file storage module is used to store surveillance video files to the target storage path and update the classification index.

[0076] A user operation behavior monitoring system of an embodiment of the present application can implement any of the above-mentioned user operation behavior monitoring methods, and the specific working process of each module in the user operation behavior monitoring system can refer to the corresponding process in the above-mentioned method embodiment.

[0077] In the several embodiments provided in this application, it should be understood that the provided methods and systems can be implemented in other ways. For example, the system embodiments described above are only illustrative; for example, the division of a certain module is only a logical function division, and there may be other division methods in actual implementation, such as multiple modules can be combined or integrated into another system, or some features can be ignored or not executed.

[0078] The embodiment of the present application also discloses a computer device.

[0079] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, a user operation behavior monitoring method as described above is implemented.

[0080] The embodiment of the present application also discloses a computer-readable storage medium.

[0081] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute any one of the above-mentioned user operation behavior monitoring methods.

[0082] Among them, computer-readable storage media can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, apparatus or device; the program code contained on the computer-readable medium can be transmitted using any appropriate medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0083] It should be noted that in the above embodiments, the description of each embodiment has different emphases, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0084] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A method for monitoring user operation behavior, characterized in that: The monitoring method comprises: Get all user operation windows and window attribute data in the target device; Detect and determine the current active window based on the user's current input focus; Intercepting the window content image of the current active window, and associating it with the current timestamp to obtain a current reference frame image; Acquire user operation behavior data related to the current active window, and associate the user operation behavior data with the current reference frame image; If there is a window change event in the window content image of the currently active window, the window content image is intercepted to update the current reference frame image, and new user operation behavior data is obtained to be associated with the updated reference frame image; wherein the window change event includes a window switching event, a window enlargement event, a window reduction event, and a window closing event; Based on the time series, the acquired user operation behavior data is recorded as a user operation log, and a monitoring video file is generated according to the continuously updated reference frame image; wherein the monitoring video file includes the continuously updated reference frame image and the user operation log associated with the reference frame image.

2. A user operation behavior monitoring method according to claim 1, characterized in that: The user operation behavior data includes keyboard input events, mouse click events, mouse drag events and wheel scroll events.

3. A user operation behavior monitoring method according to claim 1, characterized in that: The step of determining whether a window change event occurs in the window content image of the currently active window comprises: Get the window attribute data of the currently active window in real time; Based on the current reference frame image, determine whether the window ID in the window attribute data has changed. If so, there is a window switching event / window closing event; If not, it is determined whether the window size in the window attribute data has changed based on the current reference frame image. If so, a window enlargement event / window reduction event exists.

4. A user operation behavior monitoring method according to claim 1, characterized in that: Based on the time series, the steps of recording the acquired user operation behavior data as a user operation log, and generating a monitoring video file according to the continuously updated reference frame image include: Initialize the user operation log list; The acquired user operation behavior data is structuredly stored based on a time series, and the user operation behavior data is appended to a user operation log associated with a current reference frame image; Marking the continuously updated reference frame images by time segments based on the time series, embedding the user operation log into the corresponding time period, and generating a time-series operation track; Loading the time-series operation trajectory into a video generation module; The reference frame image is used as the background of the video frame, and the user operation log is superimposed on the video frame in the form of animation to generate a dynamic and visualized monitoring video file.

5. A user operation behavior monitoring method according to claim 4, characterized in that: The step of marking time segments of the continuously updated reference frame images based on the time sequence includes: Traversing continuously updated reference frame images, sorting the reference frame images based on a time sequence, and determining a window change event type corresponding to each reference frame image; Locating a generation timestamp of each of the reference frame images, and dividing a time period corresponding to each of the reference frame images; The time period corresponding to each reference frame image is marked according to the window change event type to obtain a timeline summary of each time period; wherein the timeline summary includes the start and end time of each time period and the window change event type.

6. A user operation behavior monitoring method according to any one of claims 1 to 5, characterized in that: After the step of generating the surveillance video file, the following steps are also included: Classifying the generated surveillance video files according to the window attribute data; Determine the target storage path of each surveillance video file based on the classification result; The surveillance video file is stored in the target storage path and the classification index is updated.

7. A user operation behavior monitoring system, characterized in that: The monitoring system comprises: A window acquisition module is used to acquire all user operation windows and window attribute data in the target device; An active window determination module is used to detect and determine the current active window according to the user's current input focus; A reference frame generation module, used to capture the window content image of the current active window and associate it with the current timestamp to obtain a current reference frame image; An associating module, configured to obtain user operation behavior data related to the current active window, and associate the user operation behavior data with the current reference frame image; A data processing module, configured to intercept the window content image to update the current reference frame image when a window change event occurs in the window content image of the current active window, and obtain new user operation behavior data to associate with the updated reference frame image; wherein the window change event includes a window switching event, a window enlargement event, a window reduction event, and a window closing event; A monitoring video file generation module is used to record the acquired user operation behavior data as a user operation log based on a time series, and generate a monitoring video file according to a continuously updated reference frame image; wherein the monitoring video file includes a continuously updated reference frame image and a user operation log associated with the reference frame image.

8. A user operation behavior monitoring system according to claim 7, characterized in that: The monitoring system also includes: A classification module, used for classifying the generated surveillance video file according to the window attribute data; A storage path determination module, used to determine a target storage path for each surveillance video file based on the classification result; The file storage module is used to store the monitoring video file to the target storage path and update the classification index.

9. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored which can be loaded by a processor and execute the method according to any one of claims 1 to 6.

Images