Method, device and equipment for deduplicating interception logs of terminal protection system

Abstract

The present invention discloses a method, device and equipment for deduplicating interception logs of a terminal protection system, which relates to the field of log deduplication. The method comprises: obtaining an interception log set and determining a feature metadata group set; for each feature metadata group, traversing the historical feature metadata group sets in each node of a circular queue starting from the tail node in time slice order to determine whether there are duplicate feature identifiers; if so, determining whether to discard the target feature metadata group and target interception log, or insert the target feature metadata group into the tail node and record the target interception log based on whether the difference between the traversed node position and the timing identifier exceeds the time window period; if not, inserting the target feature metadata group into the tail node; repeating the aforementioned steps to continuously process new interception log sets. The present invention solves the problem of merging the same audit or interception logs in a sliding time window of a terminal protection system, and solves the problem of possible overlap of fixed window logs.

Description

Technical Field

[0001] The present invention belongs to the technical field of log deduplication, and in particular relates to a method, device and equipment for deduplicating intercepted logs of a terminal protection system. Background Art

[0002] In the Windows endpoint protection system, user-mode software operating on PE files (Portable Executable) files triggers multiple kernel-mode driver events, resulting in multiple audit or interception logs. Fixed-time window deduplication is a simple and effective merging method, but adjacent time windows may still generate duplicate logs, causing some user inconvenience. For example, assume a fixed time window period of 1 second, and t1 and t2 are two consecutive fixed time windows. If log A1 appears in the last 0.1 seconds of the t1 fixed time window, and log A2 appears in the first 0.1 seconds of the t2 fixed time window, if logs A1 and A2 are duplicates, even if the time interval between logs A1 and A2 is 0.2 seconds, due to the fixed time window, log A2 may not be detected when it is retrieved, and deduplication of log A2 cannot be achieved. Therefore, fixed time windows only deduplicate logs within that window. Even if the time interval between logs in two adjacent windows is less than 1 second, deduplication cannot be achieved. Summary of the Invention

[0003] In order to solve the above problems, the present invention provides a method, device and equipment for deduplicating interception logs of a terminal protection system, which solves the problem of merging the same audit or interception logs in a sliding time window of the terminal protection system and solves the problem of possible overlap of fixed window logs.

[0004] In a first aspect, the present invention provides a method for deduplicating interception logs of a terminal protection system, comprising:

[0005] S1, obtaining an interception log set, determining a feature metadata group for each interception log in the interception log set, and obtaining a feature metadata group set corresponding to the interception log set; wherein the feature metadata group includes a feature identifier and a time sequence identifier of the interception log;

[0006] S2, for each feature metadata group in the feature metadata group set, traverse the historical feature metadata group set in each node of the circular queue starting from the tail node in time slice order to determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, determine whether to discard or anchor the target feature metadata group and target interception log corresponding to the duplicate feature identifier based on whether the traversed node position and the timing identifier difference exceeds the time window period; if no duplicate feature identifier is found, anchor the target feature metadata group and target interception log; wherein, the timing identifier difference is the difference between the timing identifier in the target feature metadata group and the timing identifier corresponding to the duplicate feature identifier in the historical feature metadata group set, and each node in the circular queue stores the historical feature metadata group set of the preset time slice span according to the preset first data structure;

[0007] S3, inserting the anchored feature metadata group into the tail node of the circular queue according to a preset first data structure;

[0008] S4, repeat steps S1 to S3, and continue processing the new interception log set.

[0009] In an alternative embodiment, the capacity of the circular queue C for:

[0010] (1)

[0011] In formula (1), C Indicates the number of nodes in the circular queue; Indicates the time slice span value, Represents the time window period value; where, yes An integer multiple of .

[0012] In an optional embodiment, step S2 includes:

[0013] S21, comparing the feature identifier of the target interception log with the historical feature identifier of the historical feature metadata set in the current node to determine whether there is a duplicate feature identifier;

[0014] S22, determining whether the traversal of the circular queue is completed, and if the traversal is not completed, determining whether the historical feature metadata set of the Cth node is traversed;

[0015] S23: If the historical feature metadata group set of the Cth node is traversed, it is determined whether there is a duplicate feature identifier. If so, it is determined whether the time sequence identifier difference exceeds a preset time window period. If it does not exceed the preset time window period, the target feature metadata group and target log are discarded. If it exceeds the preset time window period, the target feature metadata group and target log corresponding to the duplicate feature identifier are anchored.

[0016] S24, if the historical feature metadata group set that is not the C-th node is traversed, determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, discard the target feature metadata group and target log; if there is no duplicate feature identifier, update the existing node and return to step S21 to continue traversal.

[0017] In an optional implementation, the feature identifier is generated using a hash algorithm based on the path, creation time, last modification time, and file size of the interception log.

[0018] In an optional implementation, the hash algorithm is the MD5 algorithm.

[0019] In an optional implementation manner, the time sequence identifier is a timestamp when the interception log is intercepted.

[0020] In an optional implementation, the first data structure is a red-black tree or a hash table.

[0021] In an optional implementation, the target feature metadata group further includes the number of repeated logs.

[0022] In a second aspect, the present invention provides an interception log deduplication device for a terminal protection system, comprising:

[0023] A data acquisition module is used to acquire an interception log set, determine a feature metadata group for each interception log in the interception log set, and obtain a feature metadata group set corresponding to the interception log set; wherein the feature metadata group includes a feature identifier and a time sequence identifier of the interception log;

[0024] A traversal module is used to traverse the historical feature metadata group sets in each node of the circular queue starting from the tail node in time slice order for each feature metadata group in the feature metadata group set, and determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, determine whether to discard or anchor the target feature metadata group and target interception log corresponding to the duplicate feature identifier based on whether the traversed node position and the difference between the timing identifier exceeds the time window period; if no duplicate feature identifier is found, anchor the target feature metadata group and target interception log; wherein the timing identifier difference is the difference between the timing identifier in the target feature metadata group and the timing identifier corresponding to the duplicate feature identifier in the historical feature metadata group set, and each node in the circular queue stores the historical feature metadata group set of a preset time slice span according to a preset first data structure;

[0025] A queue updating module, configured to insert the anchored target feature metadata group into the tail node of the circular queue according to a preset first data structure;

[0026] The repeat module is used to repeat the data acquisition module, traversal module and queue update module to continuously process new interception log sets.

[0027] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in any one of the aforementioned embodiments when executing the computer program.

[0028] In a fourth aspect, the present invention provides a computer-readable medium having a non-volatile program code executable by a processor, wherein the program code enables the processor to execute the method described in any one of the aforementioned embodiments.

[0029] The technical solution provided by the embodiment of the present invention has the following beneficial effects: the interception log deduplication method, device and equipment of the terminal protection system of the present invention first obtains the interception log set, determines the feature metadata group of each log in the interception log set, and obtains the feature metadata group set corresponding to the interception log set; then, for each feature metadata group in the feature metadata group set, traverses the historical feature metadata group sets in each node of the circular queue starting from the tail node in time slice order to determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, it is determined whether to discard or anchor the target feature metadata group and target interception log corresponding to the duplicate feature identifier based on whether the difference between the traversed node position and the timing identifier exceeds the time window period; if no duplicate feature identifier is found, the target feature metadata group and target interception log are anchored; finally, the anchored target feature metadata group is inserted into the tail node of the circular queue according to a preset first data structure; this process is repeated to continuously process new interception log sets; the present invention utilizes the elimination strategy of the circular queue and the dynamic sliding control strategy of the time window to achieve log deduplication, solves the merging of the same audit or interception logs in the terminal protection system under the sliding time window, and solves the problem of possible overlap of fixed window logs. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 A schematic diagram of a flow chart of a method for deduplicating interception logs in a terminal protection system according to an embodiment of the present invention;

[0031] Figure 2 A schematic diagram illustrating the principle of storing MD5 logs in a circular queue according to an embodiment of the present invention;

[0032] Figure 3 A schematic diagram of the principle of traversing a circular queue provided by an embodiment of the present invention;

[0033] Figure 4 Another schematic diagram of the principle of traversing a circular queue provided by an embodiment of the present invention;

[0034] Figure 5A schematic diagram of the process of step S2 provided in an embodiment of the present invention;

[0035] Figure 6 A schematic diagram of the working process of a timer provided in an embodiment of the present invention;

[0036] Figure 7 A schematic diagram of the system principle of the interception log deduplication device of the terminal protection system provided by an embodiment of the present invention;

[0037] Figure 8 A schematic diagram of the system principle of an electronic device provided by an embodiment of the present invention.

[0038] In the figure: 100 - data acquisition module; 200 - traversal module; 300 - queue update module; 400 - repeat module; 1000 - electronic device; 1001 - communication interface; 1002 - processor; 1003 - memory; 1004 - bus. DETAILED DESCRIPTION

[0039] The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0040] Reference Figure 1 A method for deduplicating interception logs of a terminal protection system includes the following steps S1 to S4.

[0041] S1, obtaining an interception log set, determining a feature metadata group for each interception log in the interception log set, and obtaining a feature metadata group set corresponding to the interception log set; wherein the feature metadata group includes a feature identifier and a time sequence identifier of the interception log.

[0042] Specifically, the execution subject of this embodiment is a terminal protection system, and the terminal protection system intercepts an interception log or a batch of interception logs. In this embodiment, the obtained batch of interception logs is called an interception log set.

[0043] This embodiment uses a circular queue as the basic data structure, and each node of the circular queue uses a first data structure to store data. The first data structure can be a search tree. Preferably, the search tree is implemented using a red-black tree. The red-black tree has balanced insertion and search performance, facilitates fast insertion and search, and has flexible memory usage. This embodiment extracts the feature data of the interception log obtained at the beginning, stores it as a feature metadata group, and obtains a historical feature metadata group set. The feature metadata group here includes a feature identifier and a time sequence identifier of the interception log. The feature identifier is generated using a hash algorithm based on four elements: the path, creation time, last modification time, and file size of the interception log. These four elements can uniquely represent the interception log. The hash algorithm is the MD5 algorithm (Message Digest Algorithm 5), and the time sequence identifier is the timestamp when the interception log is intercepted.

[0044] For example, when the interception log is a PE file, the MD5 value of the PE file four-element group (path, creation time, last modification time, and file size) generated within the search tree element time window and the log generation time (the timestamp when the PE file was intercepted) are used.

[0045] In addition, the circular queue does not store MD5 logs (i.e., feature metadata groups) indefinitely, but has limited storage. The C of the circular queue is set to:

[0046] (1)

[0047] In formula (1), C Indicates the number of nodes in the circular queue; Indicates the time slice span value, Represents the time window period value; where, yes An integer multiple of .

[0048] In this embodiment, the time slice is used not only to divide the time window, but also to measure the nodes of the circular queue (one node stores the MD5 log of one time slice).

[0049] For example, if the time window period is 2 seconds and the time slice span is 1 second, the capacity C of the circular queue is 3. Therefore, the circular queue only needs to store the MD5 logs of three nodes, with one node corresponding to one time slice. Therefore, during subsequent log deduplication, when interception logs are obtained at 0.9 seconds, since 3 seconds of logs (one node corresponds to 1 second of logs) are retained, the time window can be used to deduplicate logs between 0.9 seconds and 2.9 seconds.

[0050] S2, for each feature metadata group in the feature metadata group set, traverse the historical feature metadata group sets in each node of the circular queue starting from the tail node in time slice order to determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, determine whether to discard or anchor the target feature metadata group and target interception log corresponding to the duplicate feature identifier based on whether the traversed node position and the timing identifier difference exceeds the time window period; if no duplicate feature identifier is found, anchor the target feature metadata group and target interception log; wherein the timing identifier difference is the difference between the timing identifier in the target feature metadata group and the timing identifier corresponding to the duplicate feature identifier in the historical feature metadata group set, and each node in the circular queue stores the historical feature metadata group set of the preset time slice span according to the preset first data structure.

[0051] Preferably, see Figure 5 , step S2 includes the following steps S21 to S24.

[0052] S21, comparing the feature identifier of the target interception log with the historical feature identifier of the historical feature metadata set in the current node to determine whether there is a duplicate feature identifier.

[0053] S22, determining whether the traversal of the circular queue is completed. If the traversal is not completed, determining whether the historical feature metadata set of the Cth node is traversed.

[0054] S23, if the historical feature metadata set of the Cth node is traversed, go to step S231, otherwise go to step S24.

[0055] S231, determine whether there is a repeated feature identifier, if so, execute the following steps S232 to S233; if not, directly execute step S234.

[0056] Step S232: determine whether the timing identifier difference exceeds a preset time window period.

[0057] Step S233: If the time window period is not exceeded, the target feature metadata group and the target log are discarded.

[0058] Step S234: If the time window period is exceeded, the target feature metadata group and target log corresponding to the repeated feature identifier are anchored to the current time slice, and the log is recorded (inserted into the tail node of the queue).

[0059] S24, if the historical feature metadata group set that is not the C-th node is traversed, determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, discard the target feature metadata group and target log; update the node and return to step S21 to continue traversal; if there is no duplicate feature identifier, update the existing node and return to step S21 to continue traversal.

[0060] Specifically, the nodes of the circular queue represent time windows, and the red-black tree under the window stores the MD5 logs generated within the time window.

[0061] During the traversal, a timer is used to time the time slices, and the time slices are rotated periodically in sequence to always keep the new data in the circular queue and the old data out. In a possible embodiment, the process of the timer is as follows Figure 6 As shown, the process includes the following steps S61 to S64.

[0062] Step S61: The timer starts.

[0063] Step S62, determine whether the time slice has arrived; if so, execute step S63, otherwise sleep.

[0064] Step S63: insert data (ie, target feature metadata group) at the end of the circular queue.

[0065] Step S64, determine whether the circular queue capacity C is reached; if yes, go to step S65; if not, go to step S62.

[0066] Step S65: Pop the first node of the queue. Go to step S62.

[0067] When a new log arrives, the query action is triggered. For example, for the target interception log, each node of the circular queue is queried. If there is no duplicate feature identifier in all nodes, that is, there is no identical interception log, the target interception log is reported to the latest node. If there is a duplicate interception log, the target interception log is discarded.

[0068] The time window is a sliding window at the business level. If the time window period is 2 seconds, that is, the business time window is 2 seconds, and the update period of the circular queue node is 1 second (that is, the time slice span is 1 second), the node with the oldest time slice of 2 seconds must be discarded every 1 second (one time slice). Because the circular queue has nodes for 1-2 seconds and 0-1 seconds, it also needs a node for 2-3 seconds. If the current time is 0.6 seconds, it can traverse the MD5 logs for 2-2.6 seconds. Therefore, the capacity of the circular queue is set to 2 + 1 = 3 nodes.

[0069] For the red-black tree, this embodiment further illustrates the circular queue node update cycle of 1 second. When storing feature metadata groups, the circular queue of this embodiment updates the node once every 1 second. That is, one node in the circular queue manages all interception logs within 1 second. If there are multiple interception logs within this 1 second, then all of them should be attributed to this node.

[0070] The storage structure of the red-black tree is convenient for sorting and searching. When searching and traversing, the direction of traversal is from the end node to the head node. Figure 4 As shown, searching from right to left, the right side contains the node for the time period that just occurred. The node to the left is the node for the previous time slice, and the leftmost node is the node for the furthest time slice. The search traversal here involves checking the red-black tree under a certain node to see if there is an MD5 value identical to the MD5 value of the target interception log. If an identical MD5 value exists, the target interception log is discarded, meaning the feature metadata group corresponding to the target interception log is not inserted into the node in the circular queue. If an identical MD5 value does not exist, the feature metadata group corresponding to the target interception log is inserted into the circular queue node. This ensures that the same target interception log is always available, with information about its existence within the nearest window (0-3 seconds).

[0071] In addition, when traversing, this embodiment handles the same MD5 log found at different nodes differently, that is, it is necessary to determine whether to discard or retain the target feature metadata group and target interception log based on the traversed node position and whether the difference in timing identifier exceeds the time window period. If the same MD5 log is found at the first 1 to C-1 nodes, the target interception log is directly discarded; however, if the same MD5 log is found at the Cth node, it is necessary to determine the difference in timing identifier. If the timing difference is greater than the time window period value, the target interception log is directly retained; if the timing difference is less than or equal to the time window period value, the target interception log is discarded.

[0072] For example, if the target interception log is obtained at 9:15:26 (s), traverse the circular queue from right to left. If no identical MD5 log is found between 0 and 1s, but an identical MD5 log is found between 1 and 2s, the target interception log is discarded. If an identical MD5 log is found between 2 and 3s, and the timestamp of the identical MD5 log is 9:15:06, the time difference is 2s, and the target interception log is discarded.

[0073] The operating principle of the circular queue in this embodiment is to insert (new MD5 log) from one end and pop out (discard old MD5 log) from the other end. On the one hand, the newly intercepted log is judged according to the above steps. When the log needs to be inserted into the circular queue, the log is inserted into the red-black tree of the tail node of the queue and stored in order by timestamp. On the other hand, the system starts an independent timer, triggers the queue rotation at a fixed period, inserts a new node at the tail of the queue, and removes the earliest node that exceeds the time window period. This embodiment inserts from the head node and pops out from the tail node. That is, the retained feature metadata group is inserted from the tail node, and the old feature metadata group set is popped out from the head node. In this way, the capacity of the circular queue is always kept within the capacity value C.

[0074] For example, during traversal, as a node is inserted at the end of the circular queue, a node is popped from the front, ensuring that there are always only three nodes at a given time, each storing all logs within 1 second. This way, the queue capacity is set to less than 3. To obtain interception logs at a given time, we only need to compare the MD5 values ​​of the logs from 0-1s, 1-2s, and 2-3s to see if they have the same MD5 value.

[0075] The term "anchoring" is used to describe a special way of handling duplicate logs, that is, fixing the duplicate feature metadata group and its corresponding interception log in the current circular queue node, so that it remains in the storage state during the subsequent time window sliding process until the preset discard condition is met.

[0076] S3: insert the anchored feature metadata group into the tail node of the circular queue according to the preset first data structure.

[0077] Specifically, when a new log arrives, a query operation is triggered to query the circular queue node. If there is no duplicate feature identifier in the node (red-black tree), the interception log is reported and inserted into the collection; if there is a duplicate feature identifier, the target interception log is discarded.

[0078] S4, repeat steps S1 to S3, and continue processing the new interception log set.

[0079] When a new interception log set is obtained, steps S1-S3 are repeated to traverse the circular queue. First, the first C-1 nodes are searched for the same MD5 value. If so, the target interception log is discarded. If not, the Cth node is searched. If the same MD5 value is found, the timing identifier difference is determined by calculating the timestamp difference. The timing identifier difference is compared with the time window period. If the timing identifier difference is greater than the time window period, the target interception log and its MD5 log are retained. If the timing identifier difference is less than or equal to the time window period, the target interception log and its MD5 log are discarded.

[0080] The following further illustrates the process of establishing and traversing the circular queue through the accompanying figures and examples. The following description is based on an example where the time window period is 2s, the time slice span is 1s, and the capacity C of the circular queue is 3. In the time window of 0 to 1s, a feature metadata group (MD5-1, (t1, cnt1)) is generated at time t1, a feature metadata group (MD5-2, (t2, cnt2)) is generated at time t2, and a feature metadata group (MD5-3, (t3, cnt3)) is generated at time t3. The generated red-black tree is as follows: Figure 2 shown.

[0081] In the time window of 0-2s, the elements of the circular queue are searched in the direction from near to far (from right to left in the queue). If there is a duplicate feature identifier in the red-black tree, it is discarded. If there is no duplicate feature identifier in the red-black tree, it is reported and inserted into the set tree closest to the end of the queue. Figure 3 As shown, the feature metadata group (MD5-1, (t1, cnt1)), feature metadata group (MD5-2, (t2, cnt2)) and feature metadata group (MD5-3, (t3, cnt3)) are generated in 1~2s, and the feature metadata group (MD5-4, (t4, cnt4)) and feature metadata group (MD5-5, (t5, cnt5)) are generated in 0~1s. At the current time, the feature metadata group of the target interception log (MD5-1, (t1', cnt1')) is generated. When traversing the circular queue, if the feature metadata group (MD5-1, (t1', cnt1')) is found within the 1-2s time window, the target interception log and the feature metadata group (MD5-1, (t1', cnt1')) are discarded.

[0082] When the intercept log arrives at the 3rd second, the loop queue is traversed from the nearest to the farthest (from the right to the left) and searched in sequence. If the log exists in the C-1 elements, it is discarded. If not, it is inserted into the rightmost element set. After it is found in the Cth element, the time sequence identifier difference (the difference between the time identifier in the target feature data group and the time identifier of the duplicate log) is compared to see if it exceeds the time window of 2 seconds. If it exceeds, it can be inserted into the 0-1s element. Otherwise, it is discarded. For example, if Figure 4 As shown, feature metadata groups (MD5-1, (t1, cnt1)), (MD5-2, (t2, cnt2)), and (MD5-3, (t3, cnt3)) are generated between 2 and 3 seconds. Feature metadata groups (MD5-4, (t4, cnt4)) and (MD5-5, (t5, cnt5)) are generated between 1 and 2 seconds. Adding the current time, feature metadata groups MD5-1 and MD5-8 exist. If duplicate logs exist for both, and the difference between their timestamps t' and the timestamp t' of the duplicate log is greater than 2 seconds, then feature metadata groups MD5-1 and MD5-8 will be inserted between 0 and 1 seconds.

[0083] The queue is traversed from right to left, meaning the query is from the right (entry direction) to the left (exit direction) of the circular queue. The rightmost node always contains the MD5 log generated within the most recent 1 second, the leftmost node contains the MD5 log generated within the next most recent 1 second, and the leftmost node contains the MD5 log generated within the most recent 1 second. If a duplicate signature is found within 1 second, the current interception log is not recorded. If no duplicate signature is found, the query continues for the next 1 second (next node). Note that each node contains a red-black tree containing logs generated within the current time window (1 second).

[0084] In an optional embodiment, the target feature metadata group further includes the number of repeated logs.

[0085] For example, in (MD5-1, (t1, cnt1)), the cnt1 attribute indicates the number of duplicate logs. That is, each log entry corresponds to a cnt, and the cnt in MD5-1 is cnt1. Subsequent feature metadata groups in the cnt group indicate the number of duplicate hit logs within that time window. The addition of this attribute enables accurate logging of persistent attack behavior. For all of the aforementioned hits requiring logging, an additional action is performed, and the count is incremented by 1. For newly inserted data, the initial cnt value is 1.

[0086] For example, Figure 5In step S234, due to the presence of duplicate feature identifiers, a new step is added to set the hit count to 1 (for newly inserted data, the initial value of cnt is 1). After step S24 (and before step S241), a new step is added: Update log hit count + 1, i.e., increment cnt1 by 1.

[0087] This embodiment uses a circular queue to deduplicate log data, solves the problem of merging the same audit or interception logs in a sliding time window of the terminal protection system, and solves the problem of possible overlap of fixed window logs.

[0088] See also Figure 7 An embodiment of the present invention provides an interception log deduplication device for a terminal protection system, comprising a data acquisition module 100, a traversal module 200, a queue update module 300, and a duplication module 400. The data acquisition module 100 is configured to acquire an interception log set, determine a feature metadata group for each interception log in the interception log set, and obtain a feature metadata group set corresponding to the interception log set; the feature metadata group includes a feature identifier and a time sequence identifier for the interception log. The traversal module 200 is used to traverse the historical feature metadata group sets in each node of the circular queue, starting from the tail node in time slice order, for each feature metadata group in the feature metadata group set, and determine whether there are duplicate feature identifiers. If there are duplicate feature identifiers, the target feature metadata group and target interception log corresponding to the duplicate feature identifier are discarded or anchored based on whether the difference between the traversed node position and the timing identifier exceeds the time window period. If no duplicate feature identifier is found, the target feature metadata group and target interception log are anchored. The timing identifier difference is the difference between the timing identifier in the target feature metadata group and the timing identifier corresponding to the duplicate feature identifier in the historical feature metadata group set. Each node in the circular queue stores the historical feature metadata group set with a preset time slice span according to a preset first data structure. The queue update module 300 is used to insert the anchored target feature metadata group into the tail node of the circular queue according to the preset first data structure. The repetition module 400 is used to repeat the data acquisition module, traversal module, and queue update module to continuously process new interception log sets.

[0089] In an alternative embodiment, the capacity of the circular queue C for:

[0090] (1)

[0091] In formula (1), C Indicates the number of nodes in the circular queue; Indicates the time slice span value, Represents the time window period value; where, yes An integer multiple of .

[0092] In an optional embodiment, the traversal module 200 includes a feature identifier comparison module, a first judgment module, a second judgment module, and a third judgment module. The feature identifier comparison module is used to compare the feature identifier of the target interception log with the historical feature identifier of the historical feature metadata group set in the current node to determine whether there is a duplicate feature identifier. The first judgment module is used to determine whether the traversal of the circular queue has ended. If the traversal has not ended, it is determined whether the traversal has reached the historical feature metadata group set of the Cth node. The second judgment module is used to determine whether there is a duplicate feature identifier if the traversal has reached the historical feature metadata group set of the Cth node. If so, it is determined whether the timing identifier difference exceeds the preset time window period; if it does not exceed the preset time window period, the target feature metadata group and target log are discarded; if it exceeds the preset time window period, the target feature metadata group and target log corresponding to the duplicate feature identifier are anchored. The third judgment module is used to determine whether there is a duplicate feature identifier if the historical feature metadata group set that is not the C-th node is traversed; if there is a duplicate feature identifier, the target feature metadata group and target log are discarded; if there is no duplicate feature identifier, the existing node is updated and the feature identifier is returned to the comparison module to continue traversal.

[0093] In an optional embodiment, the feature identifier is generated using a hash algorithm based on the path, creation time, last modification time, and file size of the interception log.

[0094] In an optional embodiment, the hash algorithm is the MD5 algorithm.

[0095] In an optional embodiment, the time sequence identifier is a timestamp when the interception log is intercepted.

[0096] In an optional embodiment, the first data structure is a red-black tree or a hash table.

[0097] In an optional embodiment, the target feature metadata group further includes the number of repeated logs.

[0098] The device provided in the embodiment of the present application adopts the same inventive concept as the above-mentioned method provided in the embodiment of the present application. On the premise that the method can solve the technical problem, the device can also solve the technical problem, and no further details will be given here.

[0099] Reference Figure 8An embodiment of the present invention further provides an electronic device 1000, comprising a communication interface 1001, a processor 1002, a memory 1003 and a bus 1004, wherein the processor 1002, the communication interface 1001 and the memory 1003 are connected via the bus 1004; the memory 1003 is used to store a computer program that supports the processor 1002 to execute the interception log deduplication method of the terminal protection system, and the processor 1002 is configured to execute the program stored in the memory 1003.

[0100] Optionally, an embodiment of the present invention further provides a computer-readable medium having a non-volatile program code executable by the processor 1002 , wherein the program code enables the processor 1002 to execute the interception log deduplication method of the terminal protection system as in the above embodiment.

[0101] It is understood from common technical knowledge that the present invention may be implemented by other embodiments that do not depart from its spirit or essential features. Therefore, the embodiments disclosed above are, in all respects, merely illustrative and not exclusive. All modifications within the scope of the present invention or equivalent to the scope of the present invention are intended to be encompassed by the present invention.

Claims

1. A method for deduplicating interception logs of a terminal protection system, characterized in that: include: S1, obtaining an interception log set, determining a feature metadata group for each interception log in the interception log set, and obtaining a feature metadata group set corresponding to the interception log set; wherein the feature metadata group includes a feature identifier and a time sequence identifier of the interception log; S2, for each feature metadata group in the feature metadata group set, traverse the historical feature metadata group set in each node of the circular queue starting from the tail node in time slice order to determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, determine whether to discard or anchor the target feature metadata group and target interception log corresponding to the duplicate feature identifier based on whether the traversed node position and the timing identifier difference exceeds the time window period; if no duplicate feature identifier is found, anchor the target feature metadata group and target interception log; wherein, the timing identifier difference is the difference between the timing identifier in the target feature metadata group and the timing identifier corresponding to the duplicate feature identifier in the historical feature metadata group set, and each node in the circular queue stores the historical feature metadata group set of the preset time slice span according to the preset first data structure; S3, inserting the anchored target feature metadata group into the tail node of the circular queue according to a preset first data structure; S4, repeat steps S1 to S3, and continue processing the new interception log set.

2. The method for deduplicating interception logs of a terminal protection system according to claim 1, characterized in that: The capacity of the circular queue C for: （1） In formula (1), C Indicates the number of nodes in the circular queue; Indicates the time slice span value, Represents the time window period value; where, yes An integer multiple of .

3. The method for deduplicating interception logs of a terminal protection system according to claim 2, characterized in that: The S2 includes: S21, comparing the feature identifier of the target interception log with the historical feature identifier of the historical feature metadata set in the current node to determine whether there is a duplicate feature identifier; S22, determining whether the traversal of the circular queue is completed, and if the traversal is not completed, determining whether the historical feature metadata set of the Cth node is traversed; S23: If the historical feature metadata group set of the Cth node is traversed, it is determined whether there is a duplicate feature identifier. If so, it is determined whether the time sequence identifier difference exceeds a preset time window period. If it does not exceed the preset time window period, the target feature metadata group and target log are discarded. If it exceeds the preset time window period, the target feature metadata group and target log corresponding to the duplicate feature identifier are anchored. S24, if the historical feature metadata group set that is not the C-th node is traversed, determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, discard the target feature metadata group and target log; if there is no duplicate feature identifier, update the existing node and return to step S21 to continue traversal.

4. The method for deduplicating interception logs of a terminal protection system according to claim 1, characterized in that: The feature identifier is generated using a hash algorithm based on the path, creation time, last modification time and file size of the interception log.

5. The method for deduplicating interception logs of a terminal protection system according to claim 4, characterized in that: The hash algorithm is the MD5 algorithm.

6. The method for deduplicating interception logs of a terminal protection system according to claim 1, characterized in that: The time sequence identifier is the timestamp of when the interception log is intercepted.

7. The method for deduplicating interception logs of a terminal protection system according to claim 1, characterized in that: The first data structure is a red-black tree or a hash table.

8. The method for deduplicating interception logs of a terminal protection system according to claim 1, characterized in that: The target feature metadata group also includes the number of repeated logs.

9. A device for deduplicating interception logs in a terminal protection system, characterized in that: include: A data acquisition module is used to acquire an interception log set, determine a feature metadata group for each interception log in the interception log set, and obtain a feature metadata group set corresponding to the interception log set; wherein the feature metadata group includes a feature identifier and a time sequence identifier of the interception log; A traversal module is used to traverse the historical feature metadata group sets in each node of the circular queue starting from the tail node in time slice order for each feature metadata group in the feature metadata group set, and determine whether there is a duplicate feature identifier; if there is a duplicate feature identifier, determine whether to discard or anchor the target feature metadata group and target interception log corresponding to the duplicate feature identifier based on whether the traversed node position and the difference between the timing identifier exceeds the time window period; if no duplicate feature identifier is found, anchor the target feature metadata group and target interception log; wherein the timing identifier difference is the difference between the timing identifier in the target feature metadata group and the timing identifier corresponding to the duplicate feature identifier in the historical feature metadata group set, and each node in the circular queue stores the historical feature metadata group set of a preset time slice span according to a preset first data structure; A queue updating module, configured to insert the anchored target feature metadata group into the tail node of the circular queue according to a preset first data structure; The repeat module is used to repeatedly execute the data acquisition module, the traversal module and the queue update module to continuously process new interception log sets.

10. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method according to any one of claims 1 to 8 when executing the computer program.

Images