A method and system for identifying and correcting abnormal flow logs in a multi-export environment

By identifying and correcting exception flow logs in a multi-export environment, using database query and timestamp sorting, the problem of inaccurate flow logs is solved, accurate log generation is achieved, and system cost and resource consumption is reduced.

CN118573598BActive Publication Date: 2024-11-15NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411029828.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-15
Estimated Expiration
2044-07-30

AI Technical Summary

Technical Problem

In a multi-egress environment, the statistical information of the flow log is incomplete due to the incomplete upstream and downstream information of the traffic on multiple exit nodes of the operator, resulting in inaccurate flow logs. It is difficult for existing methods to maintain and process unknown traffic through forwarding rules. Causes flow exception problem.

Method used

By obtaining the flow log to be identified in the form of a specified field, query the number of logs with the same five-tuple in the database, calculate the standard deviation and mean of the time distribution, determine the deviation range, and sort it according to the timestamp, identify and correct the existence of time intervals greater than the threshold value. The segmentation point of the file deletes or gathers the exception logs to generate accurate flow logs.

Benefits of technology

Accurate flow logs can be generated without traffic forwarding and aggregation, which reduces the number of intermediate-layer devices, reduces system server investment, and improves resource utilization and system stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118573598B_ABST
    Figure CN118573598B_ABST
Patent Text Reader

Abstract

The present application discloses a method and system for identifying and correcting abnormal flow logs in a multi-exit environment, which relates to data processing technology, including: obtaining flow logs to be identified that are recorded in the form of specified fields; querying the number of flow logs with the same five-tuple; when the number of queries is greater than a first preset threshold, calculating the standard deviation and mean of the time distribution interval of the queried flow logs, and determining the deviation range; when the determined deviation range is less than the preset deviation threshold, sorting the queried flow logs according to the timestamp to determine whether there are split points with a time interval greater than the time threshold; if there are no split points with a time interval greater than the time threshold, deleting the abnormal flow logs, and correcting the flow logs. The method of the present application is used to check, identify and correct the flow logs generated by distributed devices to generate accurate flow logs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and system for identifying and correcting abnormal flow logs in a multi-export environment. Background Art

[0002] Flow logs record session information of network connections, mainly including the five-tuple of users accessing the network (source IP address, destination IP address, source port, destination port, protocol number), statistics of sent and received messages, and other call log data. They are important network security threat intelligence data and have important data value in host security protection, network attack event detection, and data theft protection.

[0003] In the operator's multi-exit environment, it is impossible to migrate and align traffic to different locations, or the migration cost is very high. Therefore, the form of receiving data at multiple points in different locations at the same time can solve the problem of insufficient data receiving performance. For flow logs that are counted in units of flows, there will be problems such as inaccurate statistical information of the generated flow logs, as the upstream and downstream information of the unified access traffic is in multiple exit nodes of the operator, resulting in incomplete traffic received by the receiving end.

[0004] At present, there are many methods for flow log calibration in the multi-exit environment of operators. One method is to receive traffic through distributed devices, add forwarding rules in each distributed device, and the device forwards the traffic according to the forwarding rules, directing the traffic of the same five-tuple to the same data receiving end. Arrange multiple data receiving ends to receive the traffic forwarded by the forwarding rules, and distinguish different data flows according to the five-tuples at the data receiving end to generate flow logs to complete log calibration.

[0005] Adding forwarding rules to the device can accurately guide traffic. However, as traffic continues to increase, the continuous increase in forwarding rules will lead to the following problems: First, it will become increasingly difficult to maintain forwarding rules. Second, forwarding rules are configured based on known traffic. If a large amount of unknown traffic appears in the environment, the pre-configured forwarding rules will not be able to forward traffic to the same source and destination, which may lead to flow anomalies and other problems. Summary of the invention

[0006] The embodiments of the present application provide a method and system for identifying and correcting abnormal flow logs in a multi-exit environment, which checks, identifies and corrects the flow logs generated by distributed devices, and generates accurate flow logs, without the need to forward and aggregate traffic to obtain accurate flow logs.

[0007] The embodiment of the present application provides a method for identifying and correcting abnormal flow logs in a multi-exit environment, including:

[0008] Get the flow log to be identified that is recorded in the form of specified fields;

[0009] In the database, query the number of flow logs that have the same five-tuple as the log to be identified;

[0010] When the number of flow logs with the same five-tuple found is greater than the first preset threshold, the standard deviation and mean of the time distribution interval of the flow logs found are calculated, and the deviation range is determined according to the standard deviation and the mean;

[0011] When the determined deviation range is less than the preset deviation threshold, the queried flow logs are sorted according to the timestamps according to the set maximum time interval, and the time intervals of adjacent flow logs are calculated to determine whether there is a split point with a time interval greater than the time threshold;

[0012] If there is no split point with a time interval greater than the time threshold, the abnormal flow log is deleted and the flow log is corrected;

[0013] If there is a split point with a time interval greater than the time threshold, determine whether it is a normal log based on the number of flow logs before and after the split point, and correct the abnormal flow log.

[0014] Optionally, before obtaining the flow log to be identified recorded in the form of a specified field, the following is also included:

[0015] The generated flow log is recorded in the form of specified fields, wherein the specified fields include a primary key and statistical items, the primary key is used to describe the basic information of the flow log, and the statistical items are used to describe the upstream and downstream data information.

[0016] Optionally, it also includes determining that the flow log is a normal flow log when the number of flow logs with the same quintuple found is only 1.

[0017] Optionally, determining the deviation range based on the standard deviation and the mean includes: taking the mean as a basis and using a standard deviation of a specified multiple as a deviation amount to determine the deviation range.

[0018] Optionally, when the determined deviation range is less than a preset deviation threshold, the method further includes:

[0019] The mean is used as the maximum time interval, the queried flow logs are sorted from small to large according to the timestamps, and then the time intervals between adjacent flow logs are calculated.

[0020] Optionally, it also includes the case where the number of queried flow logs with the same quintuple is between a first preset threshold and a second preset threshold, where the second preset threshold is greater than 1 and less than the first preset threshold, directly sorting the queried flow logs according to timestamps according to the set maximum time interval, and calculating the time intervals of adjacent flow logs to determine whether there is a split point where the time interval is greater than the time threshold.

[0021] Optionally, if there is a split point whose time interval is greater than the time threshold, judging whether it is a normal log according to the number of flow logs before and after the split point, and correcting the abnormal flow log includes:

[0022] If the number of flow logs between any adjacent split points is greater than 2, the flow logs between the adjacent split points are aggregated;

[0023] If the number of flow logs between any adjacent split points is 1, it is considered a normal log.

[0024] An embodiment of the present application also proposes a system for identifying and correcting abnormal flow logs in a multi-exit environment, including a processor and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the method for identifying and correcting abnormal flow logs in a multi-exit environment as described above are implemented.

[0025] The embodiment of the present application improves the log identification and error correction method, checks, identifies and corrects the flow logs generated by the distributed devices, and generates accurate flow logs, without the need to forward and aggregate the traffic to obtain accurate flow logs.

[0026] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0028] Figure 1 The following is a basic flow diagram of a method for identifying and correcting abnormal flow logs in a multi-exit environment according to an embodiment of the present application;

[0029] Figure 2The overall process diagram of the method for identifying and correcting abnormal flow logs in a multi-exit environment of an embodiment of the present application is shown;

[0030] Figure 3 The following is a logical flow diagram of a method for identifying and correcting abnormal flow logs in a multi-exit environment according to an embodiment of the present application;

[0031] Figure 4 The present invention provides a logical flow diagram of a method for identifying and correcting abnormal flow logs in a multi-exit environment according to an embodiment of the present application. DETAILED DESCRIPTION

[0032] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0033] Assume that in a distributed environment, there are multiple devices A, B, ... N receiving traffic at the same time. There are the following situations for the flow logs generated for different data flows:

[0034] Case 1: Device A receives all data from data flow 1 within the time threshold, and generates a flow log.

[0035] Case 2: Device A receives part of the data flow 1 within the time threshold and generates a flow log log 1. One minute later, device B receives the remaining data flow 1 within the time threshold and generates a flow log log 2, as shown in Table 1.

[0036] Table 1

[0037]

[0038] Case 3: Device A receives all data from data flow 1 within the time threshold, generating a flow log log 1. One hour later, device A receives all data from data flow 2 within the time threshold, generating a flow log log 2. Data flow 1 and data flow 2 have the same quintuple but do not belong to the same data flow, as shown in Table 2.

[0039] Table 2

[0040]

[0041] The judgment criteria for normal flow logs: only one flow log is generated for the same data flow within the specified time range. The flow logs generated in cases 1 and 3 meet the judgment criteria for normal flow logs and are considered normal flow logs. In case 2, because multiple flow logs are generated within the specified time range, these flow logs do not meet the judgment criteria for normal flow logs and are considered abnormal flow logs.

[0042] The present application embodiment provides a method for identifying and correcting abnormal flow logs in a multi-exit environment, such as Figure 1 , Figure 2 , Figure 3 As shown, the following steps are included:

[0043] In step S101, a flow log to be identified recorded in the form of a specified field is obtained. In some embodiments, before obtaining the flow log to be identified recorded in the form of a specified field, the process further includes:

[0044] The generated flow log is recorded in the form of specified fields, where the specified fields include primary keys and statistical items. The primary key is used to describe the basic information of the flow log, and the statistical items are used to describe the upstream and downstream data information. In some specific examples, the database table structure of the specified fields is shown in Table 3.

[0045] Table 3

[0046]

[0047] In step S102, the number of flow logs having the same five-tuple as the log to be identified is queried in the database.

[0048] In step S103, when the number of flow logs with the same five-tuple found is greater than the first preset threshold, the standard deviation and mean of the time distribution interval of the found flow logs are calculated, and the deviation range is determined based on the standard deviation and mean. In some embodiments, determining the deviation range based on the standard deviation and mean includes: taking the mean as the basis, and using the standard deviation of a specified multiple as the deviation amount to determine the deviation range. In some specific examples, the flow logs are sorted, and the standard deviation σ and mean µ of the time distribution interval of all detected flow logs of the flow are calculated, for example, the values ​​of (µ-3σ, µ+3σ) are calculated.

[0049] In step S104, when the determined deviation range is less than the preset deviation threshold, the queried flow logs are sorted according to the timestamps according to the set maximum time interval, and the time intervals of adjacent flow logs are calculated to determine whether there is a split point whose time interval is greater than the time threshold.

[0050] In step S105, if there is no segmentation point whose time interval is greater than the time threshold, the abnormal flow log is deleted, and the flow log is corrected, such as merging the flow log, and the corrected flow log is inserted into the database, and the process ends. Figure 4 As shown, the method of the embodiment of the present application identifies and corrects abnormal flow logs with the help of database retrieval functions and identification and correction methods implemented in cooperation with codes. Identification refers to discovering abnormal flow logs, and correction refers to merging the data of multiple flow logs such as the number of upstream bytes, the number of downstream bytes, the number of upstream data packets, the number of downstream data packets, etc., updating the corrected results to the records with the largest timestamp values ​​and deleting the flow abnormality logs to complete the correction of the abnormal flow logs, which will not be repeated later.

[0051] In step S106, if there is a split point whose time interval is greater than the time threshold, it is determined whether it is a normal log based on the number of flow logs before and after the split point, and the abnormal flow log is corrected.

[0052] The embodiment of the present application improves the log identification and error correction method, checks, identifies and corrects the flow logs generated by the distributed devices, and generates accurate flow logs, without the need to forward and aggregate the traffic to obtain accurate flow logs.

[0053] In some embodiments, it also includes determining that the flow log is a normal flow log when the number of flow logs with the same five-tuple found is only one.

[0054] In some embodiments, when the determined deviation range is less than a preset deviation threshold, it also includes: using the mean µ as the time threshold, and setting the time threshold to the maximum time interval, sorting the queried flow logs from small to large according to the timestamp, and then calculating the time interval between adjacent flow logs.

[0055] In some embodiments, when the number of queried flow logs with the same quintuple is between a first preset threshold and a second preset threshold, wherein the second preset threshold is greater than 1 and less than the first preset threshold, the queried flow logs are directly sorted according to the timestamps according to the set maximum time interval, and the time intervals of adjacent flow logs are calculated to determine whether there is a split point with a time interval greater than the time threshold.

[0056] In some embodiments, if there is a split point whose time interval is greater than the time threshold, judging whether it is a normal log according to the number of flow logs before and after the split point, and correcting the abnormal flow log includes:

[0057] If the number of flow logs between any adjacent split points is greater than 2, the flow logs between the adjacent split points are aggregated;

[0058] If the number of flow logs between any adjacent split points is 1, it is considered a normal log.

[0059] In a specific example, if the number of flow logs before the first split point is greater than 2, the flow logs before the first split point are aggregated. If the number of flow logs between adjacent split points is greater than 2, the flow logs between adjacent split points are aggregated. If the number of flow logs before the last split point is greater than 2, the flow logs after the last split point are aggregated. If the number of flow logs before, after, and in between the split points is 1, it is determined to be a normal flow log, and no corrective action is taken, and the process ends.

[0060] In response to the above-mentioned situations 1, 2 and 3, an application example of the abnormal flow log identification and correction method of the present application is further proposed.

[0061] The three cases dynamically generate the same database query statements, as follows:

[0062] SELECT * FROM {

[0063] SELECT * COUNT(*) OVER(PARTITION BY date(timestamp), source IP, destination IP, source port, destination port, protocol) as cnt

[0064] FROM table name

[0065] }sub

[0066] WHERE cnt > 1;

[0067] 1) Situation 1:

[0068] Database query result: empty, which means that there is no other flow log in the data flow to which the log belongs within the specified time range, and the log does not meet the judgment standard 1 of abnormal flow log, and the log is judged as a normal flow log. The log identification is correct, so the method proposed in this application runs normally in situation 1.

[0069] 2) Situation 2:

[0070] Database query results: log1 and log2, which means that log1 and log2 are flow logs with the same five-tuple within the specified time range. Although log1 and log2 have the same five-tuple, whether they belong to the same data flow needs further determination. The program loads log1 and log2 into memory. After internal inspection of the program, the difference between the timestamps of log1 and log2 is 60, which is less than the time threshold of 120, and log1 and log2 are determined to belong to the same data flow. This determination indicates that there are two flow logs for the same data flow within the specified time range, which does not meet the determination criteria for normal flow logs, and log1 and log2 are identified as abnormal flow logs. The abnormal flow logs log1 and log2 are corrected. After the correction, the flow logs for data flow 1 are only:

[0071]

[0072] Combined with the correction results, it can be shown that the identification and correction of log1 and log2 are correct, so the method proposed in this application operates normally in situation 2.

[0073] 3) Situation 3:

[0074] Execute the query statement, the query results are: log1 and log2, which means that log1 and log2 are flow logs with the same five-tuple within the specified time range. After internal inspection of the program, the difference between the timestamps of log1 and log2 is 3600, which is greater than the time threshold of 120. It is determined that log1 and log2 do not belong to the same data flow and are marked. This judgment indicates that there are no multiple flow logs for the same data flow within the specified time range, which meets the judgment criteria of normal flow logs, and log1 and log2 are identified as normal flow logs. The identification of log1 and log2 is correct, so the method proposed in this application runs normally in situation 3.

[0075] The method of the present application uses the database query function to complete the preliminary identification of abnormal flow logs. After the initial screening, the flow logs are first subjected to a secondary judgment based on the log time threshold and the flow log time interval according to the identification algorithm. At the same time, a discrete judgment of the time interval is performed to identify the same abnormal data flow, and further determine whether the abnormal flow logs belong to the same data flow, thereby further accurately identifying whether the abnormal flow logs belong to different data flows, providing a basis for log merging.

[0076] This application follows the method of identifying abnormal flow logs, performs corrective operations on the flow logs of the same data flow identified as abnormal to generate normal flow logs, and automatically deletes the abnormal flow logs in the database to complete the entire process.

[0077] The method of the present application does not need to go through the middle layer to aggregate and correct the flow logs. The generated flow logs are directly stored from the collection device, and the abnormal flow logs are pre-retrieved from the database. At the same time, the abnormal flow logs are identified and corrected in cooperation with the program. The method of the present application reduces the number of middle-layer devices, reduces the investment in system servers, and improves the system's economic benefits; from the perspective of use effect, data centers are widely used and have advantages such as stability. Therefore, the flow logs are pre-processed using database query, storage and other functions to reduce the processing resource consumption of the correct logs and improve the overall resource utilization of the system.

[0078] An embodiment of the present application also proposes a system for identifying and correcting abnormal flow logs in a multi-exit environment, including a processor and a memory, wherein a computer program is stored on the memory, and when the computer program is executed by the processor, the steps of the method for identifying and correcting abnormal flow logs in a multi-exit environment as described above are implemented.

[0079] It should be noted that in the various embodiments of the present application, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0080] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0081] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0082] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for identifying and correcting abnormal flow logs in a multi-exit environment, characterized in that: include: Get the flow log to be identified that is recorded in the form of specified fields; In the database, query the number of flow logs that have the same five-tuple as the flow log to be identified; When the number of flow logs with the same five-tuple found is greater than the first preset threshold, the standard deviation and mean of the time distribution interval of the flow logs found are calculated, and the deviation range is determined according to the standard deviation and the mean; When the determined deviation range is less than the preset deviation threshold, the queried flow logs are sorted according to the timestamps according to the set maximum time interval, and the time intervals of adjacent flow logs are calculated to determine whether there is a split point with a time interval greater than the time threshold; If there is no split point with a time interval greater than the time threshold, the abnormal flow log is deleted and the flow log is corrected; If there is a split point with a time interval greater than the time threshold, determine whether it is a normal log based on the number of flow logs before and after the split point, and correct the abnormal flow log.

2. A method for identifying and correcting abnormal flow logs in a multi-exit environment as described in claim 1, characterized in that: Before obtaining the flow log to be identified recorded in the form of specified fields, it also includes: The generated flow log is recorded in the form of specified fields, wherein the specified fields include a primary key and statistical items. The primary key is used to describe the basic information of the flow log. The basic information includes source IP, destination IP, source port, destination port, protocol and timestamp. The statistical items are used to describe the upstream and downstream data information.

3. The method for identifying and correcting abnormal flow logs in a multi-exit environment as claimed in claim 1, characterized in that: It also includes determining that the flow log is a normal flow log when the number of flow logs with the same five-tuple found is only one.

4. The method for identifying and correcting abnormal flow logs in a multi-exit environment as claimed in claim 1, characterized in that: Determining the deviation range based on the standard deviation and the mean includes: taking the mean as a basis and using the standard deviation of a specified multiple as the deviation amount to determine the deviation range.

5. A method for identifying and correcting abnormal flow logs in a multi-exit environment as described in claim 4, characterized in that: When the determined deviation range is less than the preset deviation threshold, the method further includes: The mean is used as the maximum time interval, the queried flow logs are sorted from small to large according to the timestamps, and then the time intervals between adjacent flow logs are calculated.

6. A method for identifying and correcting abnormal flow logs in a multi-exit environment as claimed in claim 5, characterized in that: It also includes the case where the number of queried flow logs with the same quintuple is between a first preset threshold and a second preset threshold, where the second preset threshold is greater than 1 and less than the first preset threshold, directly sorting the queried flow logs according to the timestamp according to the set maximum time interval, and calculating the time intervals of adjacent flow logs to determine whether there is a split point with a time interval greater than the time threshold.

7. A method for identifying and correcting abnormal flow logs in a multi-exit environment as claimed in claim 5, characterized in that: If there is a split point with a time interval greater than the time threshold, determine whether it is a normal log based on the number of flow logs before and after the split point, and correct the abnormal flow log including: If the number of flow logs between any adjacent split points is greater than 2, the flow logs between the adjacent split points are aggregated; If the number of flow logs between any adjacent split points is 1, it is considered a normal log.

8. A system for identifying and correcting abnormal flow logs in a multi-export environment, characterized in that: It includes a processor and a memory, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the steps of the method for identifying and correcting abnormal flow logs in a multi-exit environment as described in any one of claims 1 to 7 are implemented.