Method and device for identifying input operation

Abstract

The present application provides a method for identifying input operations, which obtains the action time, location information, action content and operation video of the input operation, obtains the action video frame, and then matches the input operation location information with the location information of the icon and window in the action video frame according to the location information of the icon and window, determines the target window and target icon of the input operation, and obtains the attribute information of the target window and target icon. Then, according to the input operation action time, action content, and the attribute information of the corresponding target window and target icon, the input operation is subjected to a timing action logic judgment, and the action information of the related input operation is identified. In this way, the auditor can replace the process of confirming the operation and maintenance operation by sampling the video in the prior art by reviewing the action information of the operation, thereby achieving the technical effect of being able to audit all operation and maintenance videos and greatly reducing the time-consuming audit work.

Description

Technical Field

[0001] The present application relates to the field of computer application technology, and in particular to a method and device for identifying an input operation. Background Art

[0002] In traditional operation and maintenance audits, auditors can only obtain screen recording information and mouse click (time / coordinate) information of the operation and maintenance process. Auditors need to watch the recording to confirm the operation content and further determine whether there are any illegal operation and maintenance behaviors. This process makes the audit of graphic operation and maintenance very time-consuming, and due to time costs, only random inspections can be carried out and all operation and maintenance recordings cannot be audited. Summary of the invention

[0003] This application is applied to intelligent operation and maintenance technology, and performs image recognition and text recognition processing on screen recordings of graphic operation and maintenance, identifies and extracts the mouse operation content and converts it into readable text description information. It is a method for identifying and extracting the operation content of OMA (IT operation and maintenance management audit system) operation and maintenance recordings.

[0004] One purpose of the present application is to provide an input operation recognition method and device to solve the problem that the audit work of operation and maintenance in the prior art is very time-consuming, and due to time cost, only random inspections can be carried out and all operation and maintenance videos cannot be audited.

[0005] To achieve the above objectives, the present application provides an input operation recognition method and device including the following contents:

[0006] Acquire the operation time, operation position information and operation content of the input operation, and record the display interface during the execution of the input operation to obtain an operation video of the input operation;

[0007] Extracting, from the operation video, an action video frame corresponding to the operation action time according to the operation action time of the input operation;

[0008] Extracting the position information of icons and windows from the action video frame, matching the operation position information of the input operation with the position information of icons and windows in the corresponding action video frame, determining the target window and target icon of the input operation, and acquiring the attribute information of the target window and target icon;

[0009] According to the operation action time and operation action content of the relevant input operation, and the attribute information of the target window and the target icon in the corresponding action video frame, the relevant input operation is subjected to a timing action logic judgment, and the action information of the relevant input operation is identified.

[0010] Further, according to the operation action time of the input operation, extracting an action video frame corresponding to the operation action time from the operation video includes:

[0011] Decomposing the operation video into a plurality of continuous video frames according to the time sequence;

[0012] According to the operation action time, an action video frame at the operation action time point is extracted.

[0013] Further, extracting the position information of the icon and the window from the action video frame, matching the operation position information of the input operation with the position information of the icon and the window in the corresponding action video frame, and determining the target window and the target icon of the input operation, including:

[0014] Extracting window position information from the action video frame, matching the operation position information of the input operation with the window position information in the corresponding action video frame, and determining a target window for the input operation;

[0015] The position information of the icon is extracted from the target window of the action video frame, the operation position information of the input operation is matched with the position information of the window in the target window, and the target icon of the input operation is determined.

[0016] Furthermore, obtaining the attribute information of the target icon includes:

[0017] Determining text associated with the target icon in the action video frame;

[0018] The content of the text is extracted and determined as the attribute information of the target icon.

[0019] Further, determining the text associated with the target icon in the action video frame includes:

[0020] Acquire text position information corresponding to the target icon position information in the action video frame;

[0021] Corresponding text content is extracted from the text position information as text associated with the target icon.

[0022] Furthermore, obtaining text position information corresponding to the target icon position information in the action video frame includes:

[0023] Acquire an image of a target icon, and determine a first length threshold according to the image of the target icon;

[0024] Acquire a range of a first length threshold extending outward from the center of the image as a base point;

[0025] According to the range and the text position information, the text position information corresponding to the target icon position information is acquired.

[0026] Further, determining a first length threshold according to the image of the target icon includes:

[0027] Get the length and height of the image of the target icon;

[0028] Numerically comparing the length value with the height value;

[0029] Two times the smaller value between the two is used as the first length threshold.

[0030] Furthermore, obtaining the attribute information of the target window includes:

[0031] Determining the text position information corresponding to the target window according to the text position information in the action video frame and the position information of the target window;

[0032] Determine whether the character length of the text corresponding to the text position information corresponding to the target window is greater than the first character length;

[0033] If so, the text content corresponding to the text position information corresponding to the target window is determined as the attribute information of the target window.

[0034] Further, identifying the action information of the related input operation includes:

[0035] The action information of the related input operation is identified through an input operation action information library, wherein the input operation action information library is created according to different window types.

[0036] A device for identifying input operations, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any one of the methods described above.

[0037] Compared with the prior art, the present application provides a method for identifying input operations, which obtains the action time, position information, action content and operation video of the input operation, obtains the action video frame, and then matches the input operation position information with the position information of the icon and window in the action video frame according to the position information of the icon and window, determines the target window and target icon of the input operation, and obtains the attribute information of the target window and target icon. Then, according to the input operation action time, action content, and the attribute information of the corresponding target window and target icon, the input operation is subjected to a timing action logic judgment, and the action information of the related input operation is identified. In this way, the auditor can replace the process of confirming the operation and maintenance operation by sampling the video in the prior art by reviewing the action information of the operation, thereby achieving the technical effect of being able to audit all the operation and maintenance videos and greatly reducing the time-consuming audit work. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 The present invention is a flowchart of one embodiment of the method described in the present application.

[0040] Figure 2 A schematic diagram of one embodiment of the method described in this application.

[0041] Figure 3 A schematic diagram of one embodiment of the method described in this application.

[0042] Figure 4 A schematic diagram of one embodiment of the method described in this application.

[0043] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0044] The present application is described in further detail below in conjunction with the accompanying drawings.

[0045] In a typical configuration of the present application, the terminal, the device of the service network and the trusted party each include one or more processors (CPU), input / output interfaces, network interfaces and memory. The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0046] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include non-transitory media such as modulated data signals and carrier waves. The present application is further described in detail below in conjunction with the accompanying drawings.

[0047] The present application provides a method for identifying an input operation, comprising the following steps:

[0048] Step S101: obtaining the operation time, operation position information and operation content of the input operation, and recording the display interface during the execution of the input operation to obtain an operation video about the input operation.

[0049] Step S102: extracting an action video frame corresponding to the operation action time from the operation video according to the operation action time of the input operation.

[0050] Step S103: extract the position information of icons and windows from the action video frame, match the operation position information of the input operation with the position information of icons and windows in the corresponding action video frame, determine the target window and target icon of the input operation, and obtain the attribute information of the target window and target icon.

[0051] Step S104: Based on the operation time and content of the relevant input operation, and the attribute information of the target window and the target icon in the corresponding action video frame, a timing action logic judgment is performed on the relevant input operation to identify the action information of the relevant input operation.

[0052] Combine the following Figure 1 The methods and embodiments described in this application are further described.

[0053] Step S101: obtaining the operation time, operation position information and operation content of the input operation, and recording the display interface during the execution of the input operation to obtain an operation video about the input operation.

[0054] First, the operation time of the input operation is obtained. The input operation may be a mouse single click or double click, a touch selection operation of a stylus, etc. The operation time may be a time node during the operation, such as when a mouse clicks an icon, the time node when the single click action is performed. The operation position information may be the position information of the operation point relative to the window during the operation. When determining the operation position information, a window on the screen may be used as a reference to establish a unified position coordinate, and when calculating the operation position information, the coordinate range may be defined according to this coordinate, so as to better determine the operation position information. The operation action content may include an action description of each operation, such as single click, double click, etc.

[0055] When obtaining the above-mentioned operation action time, operation location information and operation action content, the entire screen or operation interface can be recorded, and the content of the screen recording includes all relevant operation processes. If necessary, only a local window can be recorded. The video recorded during the operation process can be used as the operation video. In this application, OMA (IT operation and maintenance management audit system) can be used to record the process of input operation to obtain the operation video.

[0056] Step S102: extracting an action video frame corresponding to the operation action time from the operation video according to the operation action time of the input operation.

[0057] According to the recorded operation video, an action video frame corresponding to the operation action time is extracted from the operation video. The number of video frames can be set as needed, for example, 12 frames per second, the purpose of which is to obtain the picture during the input operation. The time point of this picture should correspond to the operation action time, and the action video frame may include the operation picture corresponding to the operation action time in the operation video. The corresponding to the operation action time may include being the same as the operation action time point. Therefore, the action video frame with the same time point as the operation action time can be obtained in the operation video according to the operation action time.

[0058] In some embodiments of the present application, according to the operation action time of the input operation, an action video frame corresponding to the operation action time is extracted from the operation video. The operation video can be first decomposed into multiple continuous video frames according to the time sequence, and then according to the operation action time, the action video frame at the operation action time point is extracted.

[0059] When extracting action video frames corresponding to the operation action time from the operation video through inputting the operation action time, the operation video can first be decomposed into multiple video frames according to the time sequence to obtain video pictures arranged in the time sequence, and then the video pictures corresponding to the operation action time nodes are extracted from the multiple video pictures according to the operation action time, and these video pictures can be used as action video frames.

[0060] Step S103: extract the position information of icons and windows from the action video frame, match the operation position information of the input operation with the position information of icons and windows in the corresponding action video frame, determine the target window and target icon of the input operation, and obtain the attribute information of the target window and target icon.

[0061] The position information of each icon and window is extracted from the action video frame. The establishment of this position information should adopt the same standard as the establishment of the above-mentioned operation position information. For example, the outline of the icon and window can be first extracted through edge recognition and outline extraction, and then a unified coordinate system is established in the action video frame. In the coordinate system, the coordinate range of the icon and window is calibrated according to the outline of the icon and window. The operation position information is uniformly calibrated in the same coordinate system through the respective coordinate ranges, and each position information is described by the coordinate range. For the establishment of the coordinate system in the screen, it can be divided according to the pixel points of the screen. For example, if the screen resolution is 1920*1080, the upper left corner is (0,0) and the lower right corner is (1920,1080).

[0062] After obtaining the operation position information and the position information of the icon and the window, they can be matched. The purpose of the matching is to obtain the icon or window targeted by the operation action, and the behavior object of the operation action has been obtained. The behavior object of the operation action includes the target window and target icon of the input operation. After obtaining the target window and the target icon, relevant attribute information can be obtained based on the target window and the target icon, wherein the attribute information of the target icon can be the name of the target icon, function description, etc., and the attribute information of the target window can be the path information of the window, the application to which the dialog box belongs, etc.

[0063] For example Figure 2As shown, window 1 is the desktop of the operating system. In the present application, the desktop of the operating system is also treated as a window. In the desktop window 1, there are icons 23 (icon image 2 and text information 3), icons 45 (icon image 4 and text information 5), and a session window 6, wherein the session window 6 includes the path information 7 of the session window and icons 89 (icon image 8 and text information 9). The position information of icons 23, 45, session window 6, and icon 89 can be extracted by image recognition technology. The position information of the mouse 10 has been obtained in advance. If the mouse 10 clicks on icon 23, then according to the position of the mouse 10, the position of the desktop window 1, and the position of icon 23, the operation target of the mouse 10 is matched to icon 23, and the target icon 23 in the target window 1 is obtained.

[0064] In some embodiments of the present application, the position information of icons and windows is extracted from the action video frame, the operation position information of the input operation is matched with the position information of icons and windows in the corresponding action video frame, and the target window and target icon of the input operation are determined. The window position information can be first extracted from the action video frame, the operation position information of the input operation is matched with the position information of the window in the corresponding action video frame, the target window of the input operation is determined, and then the icon position information is extracted from the target window of the action video frame, the operation position information of the input operation is matched with the position information of the window in the target window, and the target icon of the input operation is determined.

[0065] When acquiring the target icon and the target window, the window position information can be first acquired from the action video frame, the position information of the target window can be acquired by matching the operation position information with the window position information, and then the icon position information can be extracted within the range of the target window position information, and the operation position information can be matched with the icon position information to acquire the target icon position information. In the case of establishing a unified coordinate system, the matching of the operation position information with the window position information and the matching of the operation position information with the icon position information can be determined by the coordinate range.

[0066] For example Figure 2 As shown, if the mouse 10 clicks on icon 89, the position information of session window 6 and window 1 can be extracted first, and the target window can be determined to be session window 6 based on the acquired position information of session window 6 and the position information of mouse 10, and then the position information of icon 89 can be extracted from session window 6, and the target icon can be determined to be icon 89 based on the position information of icon 89 and the position information of mouse 10.

[0067] In some embodiments of the present application, acquiring the attribute information of the target icon may be to determine text associated with the target icon in the action video frame, and then extract the content of the text to determine it as the attribute information of the target icon.

[0068] The attribute information of the target icon may be the name, function description, etc. of the target icon. The text information related to the target icon may be determined in the action video frame. After extracting the text related to the target icon, it may be used as the name, function description, etc. of the target icon, which may be used as the attribute information of the target icon.

[0069] like Figure 2 The icon 45 in the figure is taken as the target icon, wherein the icon 45 is divided into an image 4 and text information 5. The position information of the image 4 and the image 2, the text information 3 and the text information 5 can be extracted first, and then the attribute information of the text information 5 as the icon 45 can be obtained by matching.

[0070] In some embodiments of the present application, to determine the text associated with the target icon in the action video frame, the text position information corresponding to the target icon position information in the action video frame can be first obtained, and then the corresponding text content can be extracted from the text position information as the text associated with the target icon.

[0071] When obtaining the text associated with the target icon, it can be judged by the position information of the target icon and the text position information. The text position information can be obtained in the action video frame through text recognition technology, such as OCR, and then the corresponding text position information is determined through the position information of the target icon, and finally the text content is extracted in the position area of ​​the text position information. This text content can be used as the text associated with the target icon.

[0072] In some embodiments of the present application, to obtain text position information corresponding to the target icon position information in the action video frame, it is possible to first obtain an image of the target icon, determine a first length threshold based on the image of the target icon, then obtain a range that extends the first length threshold outward from the center of the image, and finally obtain the text position information corresponding to the target icon position information based on the range and the text position information.

[0073] To obtain the text position information corresponding to the target icon position information, the image of the target icon can be obtained first, and then a first length threshold can be determined based on the target image. The determination of the first length threshold can be preset. Its function is to use the center of the image as a base point and expand the range of the first length threshold outward from the image with the base point as the center. The position of the text within this range can be used as the text position information corresponding to the target icon position information.

[0074] In a specific application scenario, the range of the first length threshold that is extended outward from the center of the image can be first extracted, and then OCR processing can be performed within this range. At this time, there are fewer interference factors in the target image for OCR processing, and the text recognition accuracy is high, which facilitates the acquisition of text location information content.

[0075] In some embodiments of the present application, the first length threshold is determined based on the image of the target icon. The length value and height value of the image of the target icon are first obtained, and then the length value is numerically compared with the height value, and finally several times the smaller value between the two is used as the first length threshold.

[0076] In determining the first length threshold based on the image of the target icon, this method can be used to first obtain the length value and height value of the target image, then compare the two values, and then use the smaller value as the base, and use several times the length of the base as the first length threshold.

[0077] For example Figure 3 As shown, the target image 4 is obtained with a length of h and a width of w. The two values ​​are compared. If h>w, w is used as the base. The first length threshold can be w, 2 times w, 2.5 times w, 3 times w, etc. When the first length threshold is 2w, the text information 5 can be obtained by expanding the range of the first length threshold outward from the center of the image 4.

[0078] This method can also determine whether the text information can be used as the attribute of the target icon, for example Figure 3 After obtaining the position information of the text information 5 and the image 4 of the target icon, it is first found that within the range of the first length threshold extending outward from the center of the image 4, the text information 5 and the image 4 of the target icon are closest, and the text information 5 is directly below the image 4 of the target icon, and the size is closer. If text information that is farther away from the text information 5 is obtained, the text information 5 can be used as the main attribute information of the icon 45, and the text information that is farther away can be used as secondary attribute information, or it can be ignored.

[0079] In other embodiments of the present application, the attribute information of the target window is obtained by determining the text position information corresponding to the target window based on the text position information in the action video frame and the position information of the target window, and then determining whether the length of text characters corresponding to the text position information corresponding to the target window is greater than the first character length. If so, the text content corresponding to the text position information corresponding to the target window is determined as the attribute information of the target window.

[0080] The attribute information of the target window may be path information of the target window. When determining the attribute information of the target window, it may be obtained through the position information and text position information of the target window. First, the text position information corresponding to the target window is determined through the position information of the target window. The text position information may be the text position within the position range of the target window. After obtaining the text position information, the text character length may be obtained to determine whether the text character length is greater than the first character length. If greater, the text content of the text position information may be used as the attribute information of the target window.

[0081] For example Figure 2 As shown, the text information 7 can be determined as the text information of the session window 6 according to the position information of the session window 6 and the position information of the text information 7, and the text of the text information 7 can be extracted to determine whether it is greater than the first character length. If it is greater, the text information 7 is the attribute information of the session window 6, wherein the text information 7 can be the path information of the session window 6. For example, the text content of the text information 7 is: 192.168.211.201 / # / Wb5M5v1rE6LWLVp2Lr075ShpWNPCeMGwr, and the preset first character length is 8. If the characters of the text content of the text information 7 are greater than the first character length, the content of the text information 7 can be used as the path information of the session window 6. When the path information is used as the attribute information of the session window 6, whether the content of the text information 7 contains a " / " symbol can also be added to determine whether it is the path information.

[0082] Step S104: Based on the operation time and content of the relevant input operation, and the attribute information of the target window and the target icon in the corresponding action video frame, a timing action logic judgment is performed on the relevant input operation to identify the action information of the relevant input operation.

[0083] The operation time, operation content, target window and target icon of the input operation are comprehensively judged, the target window and target icon of the operation are integrated according to the timing of the action, and the behavior logic of the operation is judged. The description of this behavior logic can be used as the action information of the input operation. For example, the operation time is 10:45, the operation content, target window and target icon are double-clicking the target icon with the left mouse button, the attribute information of this icon is My Document, right-clicking another target icon in the opened dialog box, the attribute information of this target icon is My Picture, and clicking the Delete option with the mouse in the pop-up menu. The behavior logic of the judgment operation is "Open My Document, Delete My Picture in the Document". The input operation action information can be: 10:45, open My Document, delete My Picture in the Document.

[0084] When performing sequential action logic judgment, the judgment of behavior scenarios can also be added for different windows, and the different windows can be the desktop, a folder dialog box, a page, a menu, etc. The behavior purpose of the operation action can be more completely described in the output input operation action information. For example, a file deletion operation completed at one time includes: sliding the left button to select several target files, right-clicking one of the selected files to open the menu bar, left-clicking the delete menu, and left-clicking the confirmation button in the confirmation window.

[0085] In some other embodiments of the present application, identifying the action information of the related input operation may be identifying the action information of the related input operation through an input operation action information library, wherein the input operation action information library is created according to different window types.

[0086] After judging the behavioral logic of the operation, the judgment result and / or judgment method and / or judgment result of the behavioral scenario can be stored in the established input operation action information library, so that when encountering similar operations in the future, there is no need to make judgments again, and the data in the input operation action information library can be directly called to describe them. This can save the time used in the timing logic judgment process of the operation behavior, so that it can generate input operation action information more efficiently.

[0087] The establishment of input operation action information library can be based on different windows. For example, an input operation action information library can be established in the desktop window, and an input operation action information library can be established in the window or dialog box commonly used for operation and maintenance. In the determination of operation behavior scenarios, the same mouse operation means different behaviors in different windows. For example, double-clicking a folder icon on the desktop is an open behavior, while single-clicking it in the start menu is an open behavior. Therefore, different input operation action information libraries can be created according to different window types to convert the original mouse operation.

[0088] Figure 4 This is a schematic diagram of one of the embodiments of the present application. First, the operation action time, operation position information, operation action content, and video are obtained. The action video frame is obtained from the operation video through the time node of the operation action time, and the icon position information and window position information are obtained through edge recognition and contour extraction technology. The target icon and target window are extracted from the icon position information and window position information according to the operation position information. The attribute information of the target icon and the attribute information of the target window are obtained from the target icon and the target window. The operation action time, operation action content, target icon, target icon attribute information, target window, and target window attribute information are comprehensively judged for the timing action logic or combined with the input operation action information library to obtain the action information of the input operation.

[0089] The present application also provides a device for identifying mouse and keyboard operations, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute any one of the methods or embodiments described above.

[0090] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0091] It should be noted that the present invention can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In one embodiment, the software program of the present invention can be executed by a processor to implement the steps or functions described above. Similarly, the software program of the present invention (including related data structures) can be stored in a computer readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and the like. In addition, some steps or functions of the present invention can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0092] In addition, a part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. The program instruction for calling the method of the present invention may be stored in a fixed or removable recording medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium, and / or stored in a working memory of a computer device that runs according to the program instruction. Here, according to an embodiment of the present invention, a device is included, the device including a memory for storing computer program instructions and a processor for executing program instructions, wherein, when the computer program instruction is executed by the processor, the device is triggered to run the method and / or technical solution based on the aforementioned multiple embodiments of the present invention.

[0093] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for identifying an input operation, characterized in that: include: Acquire the operation time, operation position information and operation content of the input operation, and record the display interface during the execution of the input operation to obtain an operation video of the input operation; Extracting, from the operation video, an action video frame corresponding to the operation action time according to the operation action time of the input operation; Extracting the position information of icons and windows from the action video frame, matching the operation position information of the input operation with the position information of icons and windows in the corresponding action video frame, determining the target window and target icon of the input operation, and acquiring the attribute information of the target window and target icon; According to the operation action time and operation action content of the relevant input operation, and the attribute information of the target window and the target icon in the corresponding action video frame, the relevant input operation is subjected to a timing action logic judgment, and the action information of the relevant input operation is identified.

2. The method according to claim 1, characterized in that: Extracting an action video frame corresponding to the operation action time from the operation video according to the operation action time of the input operation includes: Decomposing the operation video into a plurality of continuous video frames according to the time sequence; According to the operation action time, an action video frame at the operation action time point is extracted.

3. The method according to claim 1, characterized in that: Extracting the position information of the icon and the window from the action video frame, matching the operation position information of the input operation with the position information of the icon and the window in the corresponding action video frame, and determining the target window and the target icon of the input operation, including: Extracting window position information from the action video frame, matching the operation position information of the input operation with the window position information in the corresponding action video frame, and determining a target window for the input operation; The position information of the icon is extracted from the target window of the action video frame, the operation position information of the input operation is matched with the position information of the window in the target window, and the target icon of the input operation is determined.

4. The method according to claim 1, characterized in that: Obtaining the attribute information of the target icon includes: Determining text associated with the target icon in the action video frame; The content of the text is extracted and determined as the attribute information of the target icon.

5. The method according to claim 4, characterized in that: Determining text associated with the target icon in the action video frame includes: Acquire text position information corresponding to the target icon position information in the action video frame; Corresponding text content is extracted from the text position information as text associated with the target icon.

6. The method according to claim 5, characterized in that: Acquiring text position information corresponding to the target icon position information in the action video frame, including: Acquire an image of a target icon, and determine a first length threshold according to the image of the target icon; Acquire a range of a first length threshold extending outward from the center of the image as a base point; According to the range and the text position information, the text position information corresponding to the target icon position information is acquired.

7. The method according to claim 6, characterized in that: Determining a first length threshold according to the image of the target icon includes: Get the length and height of the image of the target icon; Numerically comparing the length value with the height value; Two times the smaller value between the two is used as the first length threshold.

8. The method according to claim 1, characterized in that: Obtaining the property information of the target window, including: Determining the text position information corresponding to the target window according to the text position information in the action video frame and the position information of the target window; Determine whether the character length of the text corresponding to the text position information corresponding to the target window is greater than the first character length; If so, the text content corresponding to the text position information corresponding to the target window is determined as the attribute information of the target window.

9. The method according to claim 1, characterized in that: Identifying the action information of the related input operation, including: The action information of the related input operation is identified through an input operation action information library, wherein the input operation action information library is created according to different window types.

10. A device for identifying mouse and keyboard operations, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein: When the computer program instructions are executed by the processor, the device is triggered to execute the method according to any one of claims 1 to 9.

Images