Computer information monitoring method, system, device, medium and program product

Abstract

The present disclosure provides a computer information monitoring method, system, device, medium and program product. The monitoring method comprises: acquiring the image information and the operation information; time aligning the image information and the operation information; identifying a target recognition object in the image information; analyzing the operation information, image position information and image semantic information to determine operation event information of the target recognition object; and in response to the operation event information meeting an abnormal event condition, generating operation process evidence information based on the operation event information. Through a monitoring device independent of a computer host, image information and operation information corresponding to display output and input device information are intercepted at a physical layer, avoiding being bypassed or damaged from a software layer. Based on a preset analysis engine in the monitoring device, multi-modal fusion analysis is performed on the image information and the operation information, real-time monitoring, analysis, recording and alarm of computer operation are realized, and the high reliability of computer information is improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer information technology, and in particular to a method, system, device, medium, and program product for monitoring computer information. Background Technology

[0002] In modern industrial control and other sensitive environments, computer operations require strict monitoring and auditing to prevent unauthorized operations by internal personnel or external intruders, such as stealing sensitive data, engaging in illegal transactions, or damaging the system. However, traditional monitoring software, such as screen recording and keyboard / mouse logging software, has many shortcomings, specifically:

[0003] Software running on top of an operating system is easily terminated, damaged, or bypassed by viruses, Trojans, or users with high privileges; continuous screen recording generates a huge amount of data, making storage and analysis costly; it only has recording functions and cannot perform real-time analysis and instant alerts, requiring a lot of post-event auditing work; special display outputs or direct hardware access may not be captured by the software layer. Summary of the Invention

[0004] The technical problem to be solved by this disclosure is to overcome the shortcomings of the outdated operation and rudimentary functions of traditional monitoring methods in the prior art, and to provide a method, system, device, medium and program product for monitoring computer information.

[0005] This disclosure solves the above-mentioned technical problems through the following technical solution:

[0006] A first aspect provides a method for monitoring computer information. The computer equipment includes a monitoring device and a computer host. The monitoring device is connected to both an image information output interface and an operation information input interface of the computer host. The monitoring device is used to monitor the image information and operation information of the computer. The monitoring method includes:

[0007] Obtain the image information and the operation information;

[0008] The image information and the operation information are time-aligned;

[0009] Identify the target object in the image information;

[0010] The target recognition object includes image location information and image semantic information;

[0011] Analyze the operation information, the image location information, and the image semantic information to determine the operation event information of the target recognition object;

[0012] If the operation event information meets the abnormal event conditions, operation process evidence information is generated based on the operation event information.

[0013] Optionally, the monitoring device includes a monitoring clock unit, which provides a reference time source, and the step of aligning the image information with the operation information in time includes:

[0014] Obtain the first generation time based on the image information, and the second generation time of the operation information;

[0015] A first timestamp corresponding to the first generation time and a second timestamp corresponding to the second generation time are generated based on the reference time source;

[0016] Align the first timestamp with the second timestamp.

[0017] Optionally, the step of identifying the target object in the image information includes:

[0018] The image information is used to obtain text information from the image information based on optical character recognition, and the first image position information corresponding to the text information is obtained.

[0019] The image information is input into a preset convolutional neural network model to identify the interface elements in the image information and obtain the second image position information corresponding to the interface elements.

[0020] The text information is associated with the interface elements based on the first image location information and the second image location information;

[0021] The image semantic information of the target recognition object is obtained based on the associated text information and the interface elements.

[0022] Optionally, the step of analyzing the operation information, the image location information, and the image semantic information to determine the operation event information of the target recognition object includes:

[0023] A sliding time window of a preset duration is constructed based on the reference time source;

[0024] Based on the first timestamp and the second timestamp, the operation information and the image information within the sliding time window are sequentially associated.

[0025] The associated operation information, image location information, and image semantic information are input into a preset behavior analysis engine to determine the operation event information that occurs within the sliding time window.

[0026] Optionally, the step of generating operation process evidence information based on the operation event information includes:

[0027] Determine the monitoring event type, monitoring event level, and event occurrence time of the operation event information;

[0028] The operation information and image information within the sliding time window corresponding to the time of the event are used to generate the operation process evidence information.

[0029] Optionally, the computer stores target monitoring files, and the abnormal event conditions include at least one of the following events:

[0030] The image information shows that the number of incorrect password inputs is greater than the preset number of inputs.

[0031] The target monitoring file was subjected to an illegal operation;

[0032] The illegal operations include deletion operations, access operations performed at a preset illegal time, or copy operations performed more than a preset number of times.

[0033] The monitoring method also includes:

[0034] In response to the operation event information meeting the abnormal event conditions, the operation process evidence information is encrypted and stored; and / or, an alarm message including the operation process evidence information is sent to a remote server.

[0035] Secondly, a computer information monitoring system is provided, applied to computer equipment. The computer equipment includes a monitoring device and a computer host. The monitoring device is connected to the image information output interface and the operation information input interface of the computer host, respectively. The monitoring device is used to monitor the image information and operation information of the computer. The monitoring system includes an information acquisition module, a time sequence alignment module, a target recognition module, an operation event analysis module, and an abnormal event response module.

[0036] The information acquisition module is used to acquire the image information and the operation information;

[0037] The timing alignment module is used to align the image information with the operation information in a timing sequence.

[0038] The target recognition module is used to identify target objects in the image information;

[0039] The target recognition object includes image location information and image semantic information;

[0040] The operation event analysis module is used to analyze the operation information, the image location information, and the image semantic information to determine the operation event information of the target recognition object;

[0041] The abnormal event response module is used to generate operation process evidence information based on the operation event information if the operation event information meets the abnormal event conditions.

[0042] Thirdly, a computer device is provided, including a computer host, a display device, and an operation input device, and also includes a monitoring device, wherein the display device and the operation input device are respectively connected to the monitoring device, and the monitoring device is respectively connected to the image information output interface and the operation information input interface of the computer.

[0043] The monitoring device includes a neural processing unit and a memory, as well as a computer program stored in the memory and used to run on the neural processing unit.

[0044] When the neural processing unit executes the computer program, it implements the computer information monitoring method as described in the first aspect.

[0045] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the computer information monitoring method as described in the first aspect.

[0046] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the computer information monitoring method as described in the first aspect.

[0047] Based on common knowledge in the field, the above-mentioned preferred conditions can be combined arbitrarily to obtain various preferred embodiments of this disclosure.

[0048] The positive and progressive effects of this disclosure are as follows: by using a monitoring device independent of the computer host, image information and operation information corresponding to the display output and input device information are intercepted at the physical layer, and the image information and operation information are subjected to multimodal fusion analysis to realize real-time monitoring, analysis, recording and alarm of computer information, thereby improving reliability. Attached Figure Description

[0049] Figure 1 A flowchart illustrating a method for monitoring computer information provided as an exemplary embodiment of this disclosure;

[0050] Figure 2 A flowchart of step S102 in a computer information monitoring method provided as an exemplary embodiment of this disclosure;

[0051] Figure 3A flowchart of step S103 in a computer information monitoring method provided as an exemplary embodiment of this disclosure;

[0052] Figure 4 A flowchart of step S104 in a computer information monitoring method provided as an exemplary embodiment of this disclosure;

[0053] Figure 5 A flowchart of step S105 in a computer information monitoring method provided as an exemplary embodiment of this disclosure;

[0054] Figure 6 A schematic diagram of a computer information monitoring system provided as an exemplary embodiment of this disclosure;

[0055] Figure 7 This is a schematic diagram of the structure of a computer device provided for an exemplary embodiment of the present disclosure. Detailed Implementation

[0056] The present disclosure is further illustrated below by way of embodiments, but the present disclosure is not limited to the scope of the embodiments described herein.

[0057] The prefixes such as "first" and "second" used in this disclosure are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this disclosure does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not be construed as an unnecessary limitation. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0058] Example 1

[0059] Figure 1 A flowchart illustrating a method for monitoring computer information provided as an exemplary embodiment of this disclosure.

[0060] This embodiment provides a method for monitoring computer information. The computer equipment includes a monitoring device and a computer host. The monitoring device is connected to the image information output interface and the operation information input interface of the computer host, respectively. The monitoring device is used to monitor the image information and operation information of the computer. The monitoring method includes:

[0061] S101. Obtain the image information and the operation information;

[0062] Specifically, computer equipment includes a computer host, which is equipped with DVI (Digital Visual Interface), HDMI (High Definition Multimedia Interface) and / or DP (DisplayPort) image signal interfaces for connecting display devices such as monitors; and USB (Universal Serial Bus) and / or PS / 2 (Personal System / 2) operating device interfaces for connecting input devices such as mice and keyboards.

[0063] In this embodiment, the monitoring device is equipped with DP, HDMI, and / or DVI image signal input and output interfaces matching the computer host, as well as USB and / or PS / 2 input and output interfaces for operating devices. The computer host's first image signal output interface is connected to the monitoring device's first image signal input interface to send the real-time image signal and corresponding image information output by the computer host to the monitoring device. Simultaneously, the monitoring device connects to the display device's second image signal input interface via a configured second image signal output interface to distribute the image signal and corresponding image information output by the computer host to the display device for display. The monitoring device also connects to the first operation signal output interface corresponding to the keyboard and mouse input device via a configured first operation signal input interface to receive operation information from the keyboard and mouse input device. Simultaneously, the monitoring device's second operation signal output interface connects to the computer host's second operation signal input interface to distribute the received operation information to the computer host. This allows the monitoring device to monitor the operation information input and output image information from the computer host without affecting the user's normal operation of the computer host.

[0064] The monitoring device also includes a storage unit, which caches image and operation information for real-time analysis.

[0065] S102. Align the image information with the operation information in a time sequence;

[0066] Specifically, the monitoring device decodes received image signals, such as video streams, by using a built-in VPU (Visual / Video Processing Unit) and / or pre-stored software libraries (such as FFmpeg, Fast Forward Moving Picture Experts Group) in the storage unit to decode the raw video stream into a continuous sequence of RGB (primary colors) or YUV (luminance and chrominance encoded) image frames. For received operation signals, it reads the data stream sent by the PS / 2 capture device via interrupt or polling, and parses the scan code according to the PS / 2 protocol standard; and / or, uses a standard USB keyboard and mouse driver, such as the built-in hid-generic (Generic Human Interface Device Driver), to parse the meaning of each bit in the raw data stream sent by the keyboard and mouse device. The image signals and operation signals are standardized, and the image information and operation information corresponding to each frame are time-aligned according to the timing of the image information.

[0067] S103. Identify the target object in the image information;

[0068] The target recognition object includes image location information and image semantic information;

[0069] Specifically, based on image information in a single frame, pre-defined monitoring objects are identified within the image information. These include application windows, dialog boxes, icons, and / or interface controls containing pre-defined monitoring text such as "confirm," "delete," "confidential," or "confidential," as well as the mouse pointer. The position of the mouse pointer in the image represents the result of the operation, such as the selection of a pre-defined monitoring object by the mouse pointer and / or a selection box formed by dragging the mouse pointer.

[0070] In one embodiment, by analyzing the first image position information of a preset monitored object in each frame of image information and the second image position information of the corresponding mouse pointer, and based on the first and second image position information identified in at least one frame of image information, the user's operational intent towards the preset monitored object is understood, and image semantic information of the corresponding frame image is obtained. For example, in adjacent frames of image information, the mouse pointer selects the icon of the preset monitored object in the first frame of image information, a delete dialog box for the preset monitored object appears in the second frame of image information, and the mouse pointer selects the "delete" button in the delete dialog box in the third frame of image information, or the image information shows that the "delete" button is selected and pressed (e.g., the user presses the Enter key on the keyboard, triggering the selection and pressing of the "delete" button). Image semantic information such as the first frame image representing the selection operation of the preset monitored object, the second frame image representing the deletion trigger operation of the preset monitored object, and the third frame image representing the deletion confirmation operation of the preset monitored object can be generated.

[0071] S104. Analyze the operation information, the image location information, and the image semantic information to determine the operation event information of the target recognition object;

[0072] Specifically, by associating image information, image semantic information, and operation information corresponding to at least one frame of image based on time sequence, an operation dataset of the target recognition object is obtained. By analyzing the operation dataset, the corresponding operation event information is obtained. As mentioned in the previous embodiment, based solely on the image information of the last frame, it can be analyzed that a deletion operation event of a preset monitoring object has occurred. Alternatively, based on the selection, deletion triggering, and deletion confirmation of the preset monitoring object occurring in the time sequence of multiple frames of image information, the deletion operation event of the preset monitoring object can be analyzed from the movement speed and trajectory of the mouse pointer to determine whether the deletion operation event of the preset monitoring object is a deletion operation event triggered by human operation or a deletion operation event triggered by script operation.

[0073] S105. In response to the operation event information meeting the abnormal event conditions, operation process evidence information is generated based on the operation event information.

[0074] Specifically, based on the judgment results of the operation event information, for operation event information that meets the abnormal event conditions, according to the severity of the abnormal event, the image information and operation information within a preset time before and / or after the abnormal occurrence time of the abnormal operation event information are obtained as associated process information based on the time of the abnormal occurrence. The operation event information and associated process information are combined to form operation process evidence information, wherein the preset time is positively correlated with the severity of the abnormal event.

[0075] In this solution, a monitoring device independent of the computer host intercepts image and operation information corresponding to the display output and input device information at the physical layer, preventing it from being bypassed or destroyed at the software layer. Based on the preset analysis engine in the monitoring device, multimodal fusion analysis is performed on the image and operation information to achieve real-time monitoring, analysis, recording, and alarm of computer operations, thereby improving the high reliability of computer information.

[0076] As one possible implementation, the monitoring device includes a monitoring clock unit, which provides a reference time source, such as... Figure 2 As shown, step S102 includes:

[0077] S1021. Obtain the first generation time based on the image information and the second generation time of the operation information;

[0078] S1022. Generate a first timestamp corresponding to the first generation time and a second timestamp corresponding to the second generation time based on the reference time source;

[0079] S1023. Align the first timestamp with the second timestamp.

[0080] Specifically, in order to generate high-precision timestamps and align information from different signal sources in time, a stable and accurate reference clock is needed. In this embodiment, a monitoring clock unit independent of the computer host is set in the monitoring device as the reference clock to provide a reference time source for aligning image information and operation information.

[0081] In one embodiment, taking an HDMI video input interface as an example, the start and end of a frame are identified by receiving image signals according to the HDMI protocol via an HDMI-RX (HDMI receiver) in the monitoring device. The HDMI signal includes a vertical sync signal and a horizontal sync signal, and the rising / falling edge of the vertical sync signal can serve as a marker for the start of a frame. After identifying the start of a frame, timestamp sampling of the image information corresponding to that frame is initiated. The first reference time information corresponding to the start time of the frame is read by the monitoring clock unit and used as the first timestamp of the image information corresponding to that frame. The generated first timestamp is cached and used as a reference. After generating the first timestamp corresponding to the next frame, it is compared with the first timestamp of the previous frame to ensure the monotonically increasing nature of the generated timestamps.

[0082] In one embodiment, for operation information received via PS / 2 and / or USB configured in the monitoring device, such as in PS / 2 transmission, transmission is initiated by the input device (keyboard, mouse), i.e., when a key is pressed, scan code data packets are sent bit by bit; the monitoring device detects the level changes corresponding to the accessed transmission information in real time to obtain complete data packets; after receiving the complete data packet, it immediately triggers a data packet reception completion interrupt, reads the second reference time information corresponding to the interrupt time through the monitoring clock unit, uses the second reference time information as the second timestamp of the operation information, and binds the second timestamp with the corresponding operation information.

[0083] In this solution, a unified system time source is provided by the monitoring clock unit in the monitoring device, ensuring that the timestamp reference of the image information and operation information obtained by different acquisition threads is consistent, effectively eliminating the timing error caused by system call delay, and providing a high-precision, monotonically increasing timestamp.

[0084] As a feasible approach, such as Figure 3 As shown, step S103 includes:

[0085] S1031. Obtain text information from the image information based on optical character recognition, and acquire the first image position information corresponding to the text information;

[0086] S1032. Input the image information into a preset convolutional neural network model to identify the interface elements in the image information and obtain the second image position information corresponding to the interface elements.

[0087] S1033. Associate the text information with the interface elements based on the first image location information and the second image location information;

[0088] S1034. Based on the associated text information and the interface elements, obtain the image semantic information of the target recognition object.

[0089] Specifically, the monitoring device deploys an Optical Character Recognition (OCR) engine to extract text from image information, and simultaneously deploys a lightweight pre-defined convolutional neural network model to perform keyword matching and contextual semantic analysis on the text information identified in the image information to determine the interface elements identified in the image. In one embodiment, the pre-defined convolutional neural network model can be a pre-trained YOLOv5s (a lightweight object detection model) or a pre-trained NanoDet (a deep convolutional neural network model). Based on the first timestamp of the image information and the second timestamp of the operation information, the image information and operation information are sequentially associated and sorted to obtain the image semantic information of the target object from at least one frame of image. The text information and interface elements are associated through the first image position information of the text information and the second image position information of the interface elements, considering the overlap between them, or the matching of coordinate ranges. For example, when the overlap between the first and second image position information exceeds a preset association threshold (e.g., 80%), or when the coordinates of the text information fall entirely within the coordinate range of the interface element, the two are associated.

[0090] In this solution, by using dual-modal extraction of optical character recognition and a pre-defined convolutional neural network model, as well as positional association of interface elements and text information in image information, the solution breaks through the single recognition method of image information, accurately achieves semantic extraction of images, and improves recognition accuracy.

[0091] As a feasible approach, such as Figure 4 As shown, step S104 includes:

[0092] S1041. Construct a sliding time window of a preset duration based on the reference time source;

[0093] S1042. Based on the first timestamp and the second timestamp, the operation information and the image information within the sliding time window are sequentially associated.

[0094] S1043. Input the associated operation information, image location information and image semantic information into a preset behavior analysis engine to determine the operation event information that occurred within the sliding time window.

[0095] Specifically, the monitoring device includes a SoC (System on Chip) that integrates an NPU (Neural Processing Unit), a GPU (Graphics Processing Unit), and a CPU. Through image processing, it uses OpenCV (Open Source Computer Vision Library) libraries (such as template matching and optical flow) combined with simple temporal models to track the mouse cursor and calculate behavioral characteristics within the sliding time window, such as whether the instantaneous movement speed exceeds human limits, and to determine whether it is an automated script. Simple state machines or logical judgments can be implemented using software compilation.

[0096] In one embodiment, a preset behavior analysis engine is configured in the monitoring device to associate visual information and input information, achieving a deep understanding of what is seen is what is done. An algorithm maintains a sliding time window, associating all events within a given time period—including text recognition results, appearing windows, key presses, mouse clicks, etc.—based on their respective first and second timestamps. For example, when event A, "Are you sure you want to delete all data?", is detected on the interface, event A is input into the preset behavior analysis engine. If, within 500 milliseconds after event A, an "Enter key pressed" event B is captured, event B is also input into the preset behavior analysis engine. The preset behavior analysis engine then performs a correlation assessment on events A and B, forming a high-risk candidate event for deleting a preset monitored object.

[0097] In one embodiment, the preset behavior analysis engine includes a security policy rule base. Rules typically employ an "IF-THEN" (conditional statement) structure, with the condition portion capable of calling the results from all the aforementioned analysis engines. Security policy rules include:

[0098] Example rule 1: IF (condition) (OCR detected text containing "delete" AND target detected "confirm" button) AND (mouse click on "confirm" button detected within 500 milliseconds), THEN (result) risk level = high;

[0099] Example rule 2: IF (3 consecutive OCRs detect "password error" text) AND (input behavior analysis detects consecutive rapid password input), THEN Risk level = High;

[0100] Example rule 3: IF (target detects "command terminal" window) AND (input behavior analysis detects "rm -rf" command sequence), THEN Risk level = severe;

[0101] Example rule 4: IF (mouse behavior analysis detects non-human movement patterns, such as instantaneous long-distance precise movement), THEN Risk Level = Medium;

[0102] Example rule 5: IF (OCR detects "confidential" text) AND (USB device inserted and file copied), THEN Risk level = High.

[0103] In addition to pre-defined rules, a lightweight classification model (such as LSTM+Attention, a long short-term memory network with an attention mechanism) can be trained. This model takes a multimodal feature sequence over a specified time period, including text, targets, and keyboard / mouse events, as input, and directly outputs the anomaly probability and / or event classification result. The trained classification model can then more flexibly address unknown and complex attack patterns.

[0104] In this solution, a high-performance SoC is configured in the monitoring device to support a preset behavior analysis engine, which associates visual information with input information to achieve a deep understanding of what you see is what you do. Based on the understanding results and the calculation and analysis of the preset security policy rule base, the operation event information is output quickly and accurately.

[0105] As a feasible approach, such as Figure 5 As shown, step S105, which generates operation process evidence information based on the operation event information, includes:

[0106] S1051. Determine the monitoring event type, monitoring event level, and event occurrence time of the operation event information;

[0107] S1052. Generate the operation process evidence information from the operation information and the image information within the sliding time window corresponding to the time of the event occurrence.

[0108] In this solution, for operation event information that meets the conditions for an abnormal event, the operation process evidence information not only records "an abnormality occurred," but more importantly, records "the complete process of the abnormality occurring." Based on the monitoring event level of the abnormal event, using the event occurrence time of the abnormal operation event information, i.e., the corresponding timestamp information, image information and operation information within a preset time period before and / or after the time of the abnormality occurrence are obtained as associated process information. This includes video clips or keyframe screenshots N seconds before the alarm, complete keyboard and mouse operation sequence logs, and system context (such as a list of application windows running at the time). The operation event information and associated process information are combined to form operation process evidence information, where the preset time period is positively correlated with the severity of the abnormal event.

[0109] If the severity is medium, only image information and operation information within a first preset time period before the time of the anomaly are acquired as associated process information; if the severity is high, image information and operation information within a first preset time period before and after the time of the anomaly are acquired as associated process information; if the severity is severe, image information and operation information within a second preset time period before and after the time of the anomaly are acquired as associated process information, wherein the second preset time period is longer than the first preset time period, and if the associated process information includes other operation event information with a severity of high or higher, the second preset time period is extended until the associated process information includes all operation event information with a severity of high or higher that is adjacent to the operation time information with a severity of severe in the time sequence.

[0110] As one possible approach, the computer stores the target monitoring file, and the abnormal event conditions include at least one of the following events:

[0111] The image information shows that the number of incorrect password inputs is greater than the preset number of inputs.

[0112] The target monitoring file was subjected to an illegal operation;

[0113] The illegal operations include deletion operations, access operations performed at a preset illegal time, or copy operations performed more than a preset number of times.

[0114] Specifically, based on the confidentiality requirements of the computer equipment and its communication, operation, and display conditions, abnormal event conditions are adaptively set. Basic conditions include monitoring the number of password inputs and monitoring unauthorized operations on the target monitored file. Additional conditions may include: accessing unauthorized storage media (such as USB drives, external hard drives, cloud storage folders, etc.); transmitting the target monitored file through unauthorized channels (such as WeChat, QQ, non-confidential email, Bluetooth, NFC, infrared transmission); uploading the target monitored file to an unregistered server via the network; transmitting the target monitored file in plaintext (such as without encryption protocols / software); unauthorized modification of the "confidentiality level identifier" of the target monitored file; and unauthorized use of screenshot or screen recording tools. The preset illegal time includes preset working hours matched with the logged-in user (such as those associated with the logged-in user's attendance time) and / or preset confidentiality management time matched with the monitored object (such as a preset confidentiality management period). The preset number of incorrect password inputs is associated with the confidentiality level of the target monitored file; for example, for the highest confidentiality level file, the preset number of inputs is 1; for the second highest confidentiality level file, the preset number of inputs is 3. In addition to monitoring the number of password entries, it can also monitor preset copying and preset access counts of the target monitoring file.

[0115] The monitoring method also includes:

[0116] In response to the operation event information meeting the abnormal event conditions, the operation process evidence information is encrypted and stored; and / or, an alarm message including the operation process evidence information is sent to a remote server.

[0117] In one embodiment, the evidence information of the operation process is encrypted to ensure its integrity and relevance, such as by using national cryptographic algorithms. All data is encrypted and stored on a local or remote server in both structured (database) and unstructured (video file) formats.

[0118] In one embodiment, different levels of response actions are triggered based on the monitoring event level output by the preset behavior analysis engine. Response actions include: logging, real-time alarms (such as sending network messages, emails, SMS messages, etc. to the management platform), and proactive defense (such as sending instructions to the host or network device through the network interface of the monitoring device to forcibly lock or turn off the display screen).

[0119] In this solution, by encrypting and storing complete operational process evidence information and implementing timely response measures that match the monitoring time level, an adaptive active and passive defense against computer equipment is formed to ensure the information security of the monitored objects.

[0120] The embodiment provided uses a monitoring device independent of the computer host to intercept image information and operation information corresponding to display output and input device information at the physical layer, preventing them from being bypassed or destroyed at the software layer. Based on the preset analysis engine in the monitoring device, multimodal fusion analysis is performed on the image information and operation information to achieve real-time monitoring, analysis, recording and alarm of computer operation, thereby improving the high reliability of computer information.

[0121] Example 2

[0122] Corresponding to the aforementioned embodiments of computer information monitoring methods, this disclosure also provides embodiments of computer information monitoring systems.

[0123] Figure 6 This is a schematic diagram of a computer information monitoring system provided as an exemplary embodiment of the present disclosure. It is applied to a computer device, which includes a monitoring device and a computer host. The monitoring device is connected to the image information output interface and the operation information input interface of the computer host, respectively. The monitoring device is used to monitor the image information and operation information of the computer. The computer information monitoring system 100 includes an information acquisition module 110, a timing alignment module 120, a target recognition module 130, an operation event analysis module 140, and an abnormal event response module 150.

[0124] The information acquisition module 110 is used to acquire the image information and the operation information;

[0125] The timing alignment module 120 is used to align the image information with the operation information in a timing sequence.

[0126] The target recognition module 130 is used to identify the target object in the image information;

[0127] The target recognition object includes image location information and image semantic information;

[0128] The operation event analysis module 140 is used to analyze the operation information, the image location information, and the image semantic information to determine the operation event information of the target recognition object.

[0129] The abnormal event response module 150 is used to generate operation process evidence information based on the operation event information if the operation event information meets the abnormal event conditions.

[0130] In one possible implementation, the monitoring device includes a monitoring clock unit for providing a reference time source, and the timing alignment module 120 includes a time determination unit, a timestamp generation unit, and a timestamp alignment unit.

[0131] A generation time determination unit is used to obtain a first generation time based on the image information and a second generation time based on the operation information;

[0132] A timestamp generation unit is used to generate a first timestamp corresponding to the first generation time and a second timestamp corresponding to the second generation time based on the reference time source;

[0133] A timestamp alignment unit is used to align the first timestamp with the second timestamp.

[0134] As one possible implementation, the target recognition module 130 includes an image location information recognition unit, an information association unit, and an image semantic information analysis unit;

[0135] An image location information recognition unit is used to obtain text information in the image information based on optical character recognition, and to obtain the first image location information corresponding to the text information;

[0136] The image location information recognition unit is also used to input the image information into a preset convolutional neural network model, recognize the interface elements in the image information, and obtain the second image location information corresponding to the interface elements;

[0137] An information association unit is used to associate the text information with the interface elements based on the first image location information and the second image location information;

[0138] The image semantic information analysis unit is used to obtain the image semantic information of the target recognition object based on the associated text information and the interface elements.

[0139] As one possible approach, the operation event analysis module 140 includes a time window generation unit, a time sequence correlation unit, and a behavior analysis unit;

[0140] A time window generation unit is used to construct a sliding time window of a preset duration based on the reference time source;

[0141] The time-series association unit is used to perform time-series association between the operation information and the image information within the sliding time window based on the first timestamp and the second timestamp;

[0142] The behavior analysis unit is used to input the associated operation information, image location information and image semantic information into a preset behavior analysis engine to determine the operation event information that occurs within the sliding time window.

[0143] As one possible implementation, the abnormal event response module 150 includes an evidence information determination unit and an evidence information generation unit;

[0144] An evidence information determination unit is used to determine the monitoring event type, monitoring event level, and event occurrence time of the operation event information;

[0145] An evidence information generation unit is used to generate the operation process evidence information from the operation information and the image information within the sliding time window corresponding to the time of the event.

[0146] As one possible approach, the computer stores the target monitoring file, and the abnormal event conditions include at least one of the following events:

[0147] The image information shows that the number of incorrect password inputs is greater than the preset number of inputs.

[0148] The target monitoring file was subjected to an illegal operation;

[0149] The illegal operations include deletion operations, access operations performed at a preset illegal time, or copy operations performed more than a preset number of times.

[0150] The computer information monitoring system 100 also includes an encrypted storage module and / or an alarm module;

[0151] An encrypted storage module is configured to, in response to the operation event information meeting abnormal event conditions, encrypt and store the operation process evidence information; and / or,

[0152] The alarm module is used to send alarm information, including evidence information of the operation process, to a remote server.

[0153] For the system embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs.

[0154] Example 3

[0155] Figure 7 This is a schematic diagram of the structure of a computer device 900 according to an example embodiment of the present disclosure. The computer device 900 includes a computer host 910, a display device 930, and an operation input device 940, and also includes a monitoring device 920. The display device 930 and the operation input device 940 are respectively connected to the monitoring device 920. The monitoring device 920 is respectively connected to the first image signal output interface 914 and the second operation signal input interface 915 of the computer host 910.

[0156] The monitoring device 920 includes a neural processing unit 9212 and a second memory 922, as well as a computer program stored in the second memory 922 and used to run on the neural processing unit 9212.

[0157] When the neural processing unit 9212 executes the computer program, it implements the computer information monitoring method as described in Embodiment 1.

[0158] Specifically, the computer host 910 is equipped with DVI, HDMI, and / or DP as first image signal output interfaces 914 for connecting display devices 930 such as monitors; and USB and / or PS / 2 as second operation signal input interfaces 915 for connecting operation input devices 940 such as mice and keyboards. It is also equipped with a first memory 911 for pre-setting monitoring files, etc.

[0159] The first memory 911 may also include a first program tool 912 (or utility) having a set (at least one) of first program modules 913, and the second memory 922 includes a second program tool 9221 having a second program module 9222. Such program modules include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0160] The monitoring device 920 also includes a first image signal input interface 923 and a second image signal output interface 925, such as DP, HDMI and / or DVI, which are matched with the computer host, as well as a first operation signal input interface 926 and a first operation signal output interface 924 of an operation input device 940, such as USB and / or PS / 2. The first image signal output interface 914 of the computer host 910 is connected to the first image signal input interface 923 of the monitoring device 920 to send the image signal and corresponding image information output in real time by the computer host 910 to the monitoring device. At the same time, the monitoring device 920 is connected to the display device 930 through the configured second image signal output interface 925 to distribute the image signal and corresponding image information output by the computer host 910 to the display device 930 for display. The monitoring device 920 is connected to the operation input device 940 through the configured first operation signal input interface 926 to receive the operation information of the operation input device 940. At the same time, the first operation signal output interface 924 of the monitoring device 920 is connected to the second operation signal input interface 915 of the computer host 910 to distribute the operation information received by the monitoring device 920 to the computer host 910.

[0161] The memory in the monitoring device 920 is the second memory 922 of the computer device 900. The second memory 922 is used to store received image information and operation information, as well as a computer program for implementing the computer information monitoring method as described in Embodiment 1.

[0162] The monitoring device 920 also includes a built-in SoC chip 921, which integrates a CPU 9211, GPU 9213, VPU 9214, NPU 9212, and system clock 9215. The VPU 9214 and system clock 9215 run computer programs stored in the second memory 922 to synchronize and preprocess image and operational information stored therein. The CPU 9211, GPU 9213, and NPU 9212, by running computer programs stored in the second memory 922, correlate the synchronized information based on time sequence, perform calculations, understanding, and analysis on the correlated multimodal data, output analysis results, and execute corresponding alarm, evidence generation, and proactive defense measures based on the stored computer programs.

[0163] Computer device 900 can also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter. The network adapter is used for network connection. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with computer device 900, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0164] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0165] Example 4

[0166] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the computer information monitoring method provided in any of the above embodiments.

[0167] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0168] Example 5

[0169] This disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the computer information monitoring method described in any of the above embodiments.

[0170] The program code for executing the computer program product of this disclosure can be written in any combination of one or more programming languages, and the program code can be executed entirely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or entirely on a remote device.

[0171] While specific embodiments of this disclosure have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of this disclosure is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of this disclosure, but all such changes and modifications fall within the scope of protection of this disclosure.

Claims

1. A method of monitoring computer information, characterized by, The application is applied to a computer device, the computer device comprises a monitoring device and a computer host, the monitoring device is connected with an image information output interface and an operation information input interface of the computer host respectively, the monitoring device is used for monitoring image information and operation information of the computer device, and the monitoring method comprises: obtaining the image information and the operation information; aligning the image information and the operation information in time sequence; identifying a target identification object in the image information; wherein the target identification object comprises image position information and image semantic information; the identification of the target identification object in the image information comprises: obtaining text information in the image information based on optical character recognition of the image information, and obtaining first image position information corresponding to the text information; inputting the image information into a preset convolutional neural network model to identify interface elements in the image information and obtain second image position information corresponding to the interface elements; associating the text information and the interface elements based on the first image position information and the second image position information; obtaining the image semantic information of the target identification object based on the associated text information and interface elements; analyzing the operation information, the image position information and the image semantic information to determine operation event information of the target identification object; in response to the operation event information meeting an abnormal event condition, generating operation process evidence information based on the operation event information.

2. The method of claim 1, wherein, The monitoring device comprises a monitoring clock unit, which is used to provide a reference time source, and the step of aligning the image information and the operation information in time sequence comprises: obtaining a first generation time based on the image information and a second generation time of the operation information; generating a first timestamp corresponding to the first generation time and a second timestamp corresponding to the second generation time based on the reference time source; aligning the first timestamp and the second timestamp.

3. The method of claim 2, wherein, The step of analyzing the operation information, the image position information and the image semantic information to determine the operation event information of the target identification object comprises: constructing a sliding time window of a preset duration based on the reference time source; based on the first timestamp and the second timestamp, time sequence association is performed on the operation information and the image information in the sliding time window; inputting the associated operation information, image position information and image semantic information into a preset behavior analysis engine to determine the operation event information occurring in the sliding time window.

4. The method of claim 3, wherein, The step of generating operation process evidence information based on the operation event information comprises: determining the monitoring event type, monitoring event level and event occurrence time of the operation event information; generating the operation process evidence information based on the operation information and the image information in the sliding time window corresponding to the event occurrence time.

5. The method of claim 3, wherein, The computer host stores a target monitoring file, and the abnormal event condition comprises at least one of the following events: the number of times of displaying password input errors in the image information is greater than a preset input number; The target monitoring file is executed with an illegal operation; The illegal operation includes a deletion operation, an access operation at a preset illegal time, or a copy operation number greater than a preset copy number; The monitoring method further includes: In response to the operation event information meeting an abnormal event condition, the operation process evidence information is encrypted and stored, and / or alarm information including the operation process evidence information is sent to a remote server.

6. A computer information monitoring system characterized by comprising: The computer device includes a monitoring device and a computer host, the monitoring device is connected with an image information output interface and an operation information input interface of the computer host respectively, the monitoring device is used for monitoring image information and operation information of the computer, and the monitoring system includes an information acquisition module, a time sequence alignment module, a target identification module, an operation event analysis module, and an abnormal event response module. The information acquisition module is used for acquiring the image information and the operation information. The time sequence alignment module is used for time sequence alignment of the image information and the operation information. The target identification module is used for identifying a target identification object in the image information. The target identification object includes image position information and image semantic information. The target identification module includes an image position information identification unit, an information association unit, and an image semantic information analysis unit. The image position information identification unit is used for obtaining text information in the image information based on optical character recognition of the image information, and acquiring first image position information corresponding to the text information. The image position information identification unit is also used for inputting the image information into a preset convolutional neural network model, identifying interface elements in the image information, and acquiring second image position information corresponding to the interface elements. The information association unit is used for associating the text information and the interface elements based on the first image position information and the second image position information. The image semantic information analysis unit is used for obtaining the image semantic information of the target identification object based on the associated text information and the interface elements. The operation event analysis module is used for analyzing the operation information, the image position information, and the image semantic information, and determining operation event information of the target identification object. The abnormal event response module is used for generating operation process evidence information based on the operation event information in response to the operation event information meeting an abnormal event condition.

7. A computer device comprising a computer mainframe, a display device and an operation input device, characterized by comprising: The display device and the operation input device are connected with the monitoring device respectively, and the monitoring device is connected with an image information output interface and an operation information input interface of the computer respectively. The monitoring device includes a neural processing unit and a memory, and a computer program stored in the memory and used for running on the neural processing unit; The neural processing unit implements the computer information monitoring method in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the computer information monitoring method in any one of claims 1 to 5 when executed by a processor.

9. A computer program product comprising a computer program, characterized in that, The computer program, when executed by a processor, implements the computer information monitoring method as claimed in any one of claims 1 to 5.

Images