Malicious traffic detection system based on point cloud analysis false positive cleaning method and device

Abstract

The application discloses a false positive cleaning method and device for a malicious traffic detection system based on point cloud analysis, and the method comprises the following steps: obtaining a point cloud based on a traffic feature vector associated with an alarm signal generated by the malicious traffic detection system; covering the point cloud by using a first plurality of voxels, and determining a second plurality of voxels based on the point cloud covering result; performing an aggregation operation on the second plurality of voxels to construct a voxel group; and respectively identifying alarm signals corresponding to a high-density voxel group and a low-density voxel group obtained based on voxel group clustering to obtain a true positive alarm identification result and a false positive alarm identification result. Through the point cloud analysis method based on voxels, the application automatically divides the alarms generated by the malicious traffic identification system into true positive alarms and false positive alarms. The manual cost of manually distinguishing the true positive alarms from the false positive alarms is significantly reduced, and the application has good real-time performance, accuracy and robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, and in particular to a false positive cleaning method and device for a malicious traffic detection system based on point cloud analysis. BACKGROUND

[0002] In recent years, network security has gradually become an important part of national security. However, a large number of new network attacks are proposed every year. The malicious traffic identification system based on artificial intelligence is a new defense technology, which can automatically detect the traffic generated by these new network attacks, and can protect a large number of legitimate Internet users by intercepting the detected attack traffic.

[0003] However, the malicious traffic detection system will generate a large number of false positive alarms, that is, alarms generated by the false identification of legitimate user traffic as attack traffic. Only by significantly reducing the number of false positive alarms, can the malicious traffic detection system based on artificial intelligence be truly applied to industrial environments. Under the existing research technology, the malicious traffic identification system based on artificial intelligence can automatically identify malicious traffic on the Internet. By intercepting the identified malicious traffic, a large number of Internet users can be protected from network attacks. However, as a new network technology, the malicious traffic identification system will generate a large number of false positive alarms, which seriously affects the deployment and use of the system. SUMMARY

[0004] The present application aims to at least solve one of the problems in the related art.

[0005] To this end, the present application proposes a false positive cleaning method for a malicious traffic detection system based on point cloud analysis, which can significantly reduce the number of false positive alarms of various malicious traffic identification systems, thereby reducing the manual cost caused by manually distinguishing between true positive alarms and false positive alarms, significantly reducing operating costs, and further applying the malicious traffic identification system based on artificial intelligence to real industrial environments.

[0006] Another object of the present application is to propose a false positive cleaning device for a malicious traffic detection system based on point cloud analysis.

[0007] To achieve the above object, the present application proposes a false positive cleaning method for a malicious traffic detection system based on point cloud analysis, comprising:

[0008] obtaining a point cloud based on the traffic feature vectors associated with the alarm signals generated by the malicious traffic detection system;

[0009] covering the point cloud with a first plurality of voxels, and determining a second plurality of voxels based on the point cloud coverage result;

[0010] performing a gathering operation on the second plurality of voxels to construct a voxel group;

[0011] respectively identifying alarm signals corresponding to the high-density voxel group and the low-density voxel group obtained based on clustering of the voxel group to obtain true positive alarm identification results and false positive alarm identification results.

[0012] In addition, the false positive alarm cleaning method of the malicious traffic detection system based on point cloud analysis according to the above-mentioned embodiments of the present application can further have the following additional technical features:

[0013] Further, in an embodiment of the present application, the point cloud obtained based on the traffic feature vectors associated with the alarm signals generated by the malicious traffic detection system comprises:

[0014] obtaining a plurality of alarm signals of the malicious traffic detection system within a preset time;

[0015] associating each alarm signal of the plurality of alarm signals with a traffic feature vector of a preset length, and combining all the traffic feature vectors to obtain a first matrix;

[0016] performing a logarithmic transformation on each element of the first matrix to obtain a second matrix, and performing a maximum and minimum normalization on the second matrix to obtain a third matrix representing a point cloud.

[0017] Further, in an embodiment of the present application, the covering of the point cloud with the first plurality of voxels and the determination of the second plurality of voxels based on the covering result of the point cloud comprise:

[0018] obtaining a first plurality of voxels based on a cube of a preset edge length in a multi-dimensional feature space;

[0019] covering points in the point cloud represented by the third matrix with the first plurality of voxels, and removing voxels in the first plurality of voxels that cover less than a preset number of points in the point cloud represented by the third matrix to obtain a second plurality of voxels.

[0020] Further, in an embodiment of the present application, the performing of a gathering operation on the second plurality of voxels to construct a voxel group comprises:

[0021] obtaining an index corresponding to the second plurality of voxels;

[0022] calculating a Manhattan distance between voxels represented by the index, and obtaining a voxel adjacency relationship based on a comparison result of the Manhattan distance and a first distance threshold;

[0023] representing voxels that are adjacent to each other in the second plurality of voxels as a voxel group based on the voxel adjacency relationship.

[0024] Further, in one embodiment of the present application, the identification of the alarm signals corresponding to the high-density voxel group and the low-density voxel group respectively obtained based on the voxel group clustering is to obtain true positive alarm identification results and false positive alarm identification results, comprising:

[0025] obtaining a density feature vector based on the extracted density features of the voxel group;

[0026] clustering all the density feature vectors using a preset clustering algorithm, so as to identify the voxel group corresponding to the density feature vector with a distance greater than a second distance threshold from the clustering center as a voxel group in a high-density region, and vice versa, as a voxel group in a low-density region;

[0027] identifying the alarm signals corresponding to the points represented by all the voxels in the voxel group in the high-density region and the voxel group in the low-density region respectively to obtain true positive alarm identification results and false positive alarm identification results.

[0028] To achieve the above purpose, another aspect of the present application provides a false positive cleaning device for a malicious traffic detection system based on point cloud analysis, comprising:

[0029] a point cloud acquisition module configured to obtain a point cloud based on a traffic feature vector associated with an alarm signal generated by a malicious traffic detection system;

[0030] a voxel construction module configured to cover the point cloud with a first plurality of voxels and determine a second plurality of voxels based on the point cloud coverage result;

[0031] a voxel aggregation module configured to perform an aggregation operation on the second plurality of voxels to construct a voxel group;

[0032] a density learning module configured to identify the alarm signals corresponding to a high-density voxel group and a low-density voxel group respectively obtained based on the voxel group clustering to obtain true positive alarm identification results and false positive alarm identification results.

[0033] The false positive cleaning method and device for a malicious traffic detection system based on point cloud analysis according to the embodiments of the present application can automatically classify the alarms generated by various malicious traffic identification systems into true positive alarms and false positive alarms, and further filter out false positive alarms. Moreover, the method and device do not require manual labor, can process a large number of alarms in real time, have good robustness and good universality.

[0034] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0035] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description, taken in conjunction with the following drawings of exemplary embodiments of the present application, wherein:

[0036] Figure 1 is a flowchart of a false positive cleaning method of a malicious traffic detection system based on point cloud analysis according to an embodiment of the present application;

[0037] Figure 2 is an architectural diagram of a false positive cleaning method of a malicious traffic detection system based on point cloud analysis according to an embodiment of the present application;

[0038] Figure 3 is a structural schematic diagram of a false positive cleaning device of a malicious traffic detection system based on point cloud analysis according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict. The present application will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0040] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.

[0041] The false positive cleaning method and device of a malicious traffic detection system based on point cloud analysis according to the embodiments of the present application will be described below with reference to the accompanying drawings.

[0042] Figure 1 is a flowchart of a false positive cleaning method of a malicious traffic detection system based on point cloud analysis according to an embodiment of the present application.

[0043] As shown in Figure 1 , the method includes but is not limited to the following steps:

[0044] S1, obtaining a point cloud based on a traffic feature vector associated with an alarm signal generated by a malicious traffic detection system;

[0045] S2, covering the point cloud with a first plurality of voxels, and determining a second plurality of voxels based on the point cloud coverage result;

[0046] S3, performing an aggregation operation on the second plurality of voxels to construct a voxel group;

[0047] S4, identify the alarm signals corresponding to the high-density voxel groups and low-density voxel groups obtained based on voxel group clustering to obtain the true positive alarm identification results and false positive alarm identification results.

[0048] The false positive cleaning method for malicious traffic detection system based on point cloud analysis according to embodiments of the present invention can automatically classify alarms generated by various existing malicious traffic identification systems into true positive alarms and false positive alarms, thereby allowing system users to focus on true positive alarms that actually report attack events, and thus make a rapid and real-time response to network threat events.

[0049] The false positive cleaning method of the malicious traffic detection system based on point cloud analysis according to an embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0050] like Figure 2 The diagram shown is a framework diagram of a false positive cleaning method for a malicious traffic detection system based on point cloud analysis, according to an embodiment of the present invention. Figure 2 As shown:

[0051] In one embodiment of the present invention, S1 can be... Figure 2 The voxel construction module in the system uses point clouds to represent the alarms generated by the malicious traffic identification system, and uses voxels to represent the point clouds.

[0052] Specifically, a series of alerts generated by a malicious traffic identification system are taken as input, and each alert is associated with a traffic feature vector. All traffic feature vectors are normalized and represented as point clouds, and then a series of voxels are constructed to re-represent these points. The steps in this voxel construction module may include:

[0053] S11, the malicious traffic detection system receives N alerts within a time interval Δt. For each alert, it is associated with a vector of length M. All feature vectors are combined into a matrix S, where the element in the i-th row and j-th column represents the j-th feature of the i-th alert.

[0054]

[0055] S12, and then performing a logarithmic transformation on each element of matrix S to obtain the matrix representation Q = [q i,j (1≤j≤M, 1≤i≤N).

[0056] Q = log2(1 + S)

[0057] S13, then perform max-min normalization on the obtained matrix Q to obtain the matrix This will be the output of this step.

[0058]

[0059]

[0060] S14, then construct a voxel to represent the point cloud carried by the matrix P. By definition, the voxel is a small cube with side length ∈ in the M-dimensional feature space, which is defined as wherein is the index of the voxel. Then, all the points covered by the voxel are represented by the voxel.

[0061]

[0062] S15, then remove the voxels that cover less than E points in the point cloud carried by P, and keep the remaining voxels, the index of which is

[0063] In an embodiment of the present application, S2 and S3 can be Figure 2 The voxel aggregation module in S2 and S3: aggregate the remaining voxels, and represent a group of voxels that are adjacent to each other in position by a voxel group. The steps of an embodiment of the voxel aggregation module can include:

[0064] S31, calculate the Manhattan distance between the voxels represented by the indices a i ,a j ∈ A * , if the Manhattan distance is less than D*M, it is determined that the voxels represented by a i ,a j are adjacent to each other, wherein D is a parameter of the system.

[0065] S32, according to the adjacency relationship obtained in the previous step, represent the voxels that are adjacent to each other as a voxel group, which is involved in the next step.

[0066] In an embodiment of the present application, S4 can be Figure 2 The density learning module in S4: use an unsupervised method to automatically distinguish high-density voxel groups and low-density voxel groups, wherein the alarms corresponding to the high-density voxel groups are identified as true positive alarms, and the alarms corresponding to the low-density voxel groups are identified as false positive alarms.

[0067] Specifically, for each voxel group obtained, extract the density feature, and use the K-Means unsupervised clustering method to detect the voxel groups with high-density features. The alarms corresponding to the points represented by the voxel groups with high density are determined as true positives, and the alarms corresponding to the points represented by the voxel groups with low density are determined as false positives. Finally, the system filters out the false positive alarms and reports the true positive alarms. The steps of an embodiment of the density learning module can include:

[0068] S41, extract four density features for each voxel group. One of the features is the sum of points represented by each voxel in the voxel group. The remaining three features are the number of points in a high-dimensional sphere with a radius of 2*r, 4*r, and 8*r, respectively, and the center of the sphere being the center of the space covered by the voxel group.

[0069] S42, cluster all the obtained density feature vectors of the voxel groups using K-Means, where the parameter K of the algorithm is set to 1, and the density feature vectors with a Euclidean distance from the cluster center exceeding C are identified as abnormally high values, and the corresponding voxel groups are considered to be in a high-density region. Conversely, the voxel groups with a Euclidean distance not exceeding C are considered to be in a low-density region.

[0070] S43, for the voxel clusters in the high-density region, the alerts corresponding to the points represented by all the voxels in the cluster are considered to be true positive alerts, and the remaining alerts are false positive alerts. Finally, the false positive alerts are filtered out, and the true positive alerts are output.

[0071] In summary, the present application can effectively reduce the number of false positive alerts, thereby reducing the overhead of manual identification of false positive alerts by a malicious traffic identification system. Experiments show that the present application can reduce the number of false positive alerts by 95% for 11 of the most advanced intelligent malicious traffic identification schemes. By reducing the number of false positive alerts, the AUC accuracy index can be improved by 14%, and other seven accuracy indexes can be significantly improved. In addition, compared with the traditional retraining false positive cleaning method, more false positives can be filtered out. In addition, the method of the present application can process more than 200,000 alerts per second, with an average delay of 0.77 seconds.

[0072] The malicious traffic detection system false positive cleaning method based on point cloud analysis according to the embodiments of the present application can automatically classify the alerts generated by various malicious traffic identification systems into true positive alerts and false positive alerts, and then filter out the false positive alerts. And it can significantly reduce the manual overhead of distinguishing true positive alerts from false positive alerts, and has good real-time performance, accuracy, and robustness.

[0073] To achieve the above embodiments, as Figure 3 shown, the present embodiment also provides a malicious traffic detection system false positive cleaning device 10 based on point cloud analysis, which comprises a point cloud acquisition module 100, a voxel construction module 200, a voxel aggregation module 300, and a density learning module 400.

[0074] The point cloud acquisition module 100 is used to obtain a point cloud based on a traffic feature vector associated with an alarm signal generated by a malicious traffic detection system;

[0075] a voxel construction module 200, configured to cover the point cloud with a first plurality of voxels, and determine a second plurality of voxels based on a result of the point cloud covering;

[0076] a voxel aggregation module 300, configured to perform an aggregation operation on the second plurality of voxels to construct a voxel group;

[0077] a density learning module 400, configured to respectively identify alarm signals corresponding to a high-density voxel group and a low-density voxel group obtained based on voxel group clustering to obtain a true positive alarm identification result and a false positive alarm identification result.

[0078] Further, the point cloud acquisition module 100 is further configured to:

[0079] acquire a plurality of alarm signals of the malicious traffic detection system within a preset time;

[0080] associate each alarm signal of the plurality of alarm signals with a traffic feature vector of a preset length, and combine all the traffic feature vectors to obtain a first matrix;

[0081] perform logarithmic transformation on each element of the first matrix to obtain a second matrix, and perform maximum and minimum normalization on the second matrix to obtain a third matrix representing a point cloud.

[0082] Further, the voxel construction module 200 is further configured to:

[0083] obtain a first plurality of voxels based on a cube of a preset edge length in a multi-dimensional feature space;

[0084] cover points in the point cloud represented by the third matrix with the first plurality of voxels, and remove voxels in the first plurality of voxels that cover less than a preset number of points in the point cloud represented by the third matrix to obtain a second plurality of voxels.

[0085] Further, the voxel aggregation module 300 is further configured to:

[0086] acquire an index corresponding to the second plurality of voxels;

[0087] calculate a Manhattan distance between voxels represented by the index, and obtain a voxel adjacency relationship based on a comparison result of the Manhattan distance and a first distance threshold;

[0088] based on the voxel adjacency relationship, represent voxels that are adjacent to each other in the second plurality of voxels as a voxel group.

[0089] Further, the density learning module 400 is further configured to:

[0090] obtain a density feature vector based on a plurality of density features of the extracted voxel group;

[0091] The preset clustering algorithm is used to cluster all the density feature vectors, so as to identify the voxel group corresponding to the density feature vector with a distance greater than the second distance threshold from the clustering center as a voxel group in a high-density region, and vice versa, as a voxel group in a low-density region.

[0092] The alarm signals corresponding to the points represented by all the voxels in the voxel groups in the high-density region and the low-density region are identified respectively to obtain true positive alarm identification results and false positive alarm identification results.

[0093] The malicious traffic detection system false positive cleaning device based on point cloud analysis according to the embodiments of the present application can automatically classify the alarms generated by various malicious traffic identification systems into true positive alarms and false positive alarms, and further filter out the false positive alarms. Moreover, the manual cost of manually distinguishing the true positive alarms from the false positive alarms can be significantly reduced, and the device has good real-time performance, accuracy, and robustness.

[0094] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0095] In addition, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

Claims

1. A false positive washing method for a malicious traffic detection system based on point cloud analysis, characterized in that, The method comprises the following steps: A point cloud is obtained based on the traffic feature vectors associated with the alarm signals generated by the malicious traffic detection system; The point cloud is covered by a first plurality of voxels, and a second plurality of voxels is determined based on the point cloud coverage result; A voxel group is constructed by performing a clustering operation on the second plurality of voxels; High-density voxel groups and low-density voxel groups are clustered based on the voxel group, and alarm signals corresponding to the high-density voxel groups and the low-density voxel groups are identified to obtain true positive alarm identification results and false positive alarm identification results, respectively; The point cloud obtained based on the traffic feature vectors associated with the alarm signals generated by the malicious traffic detection system comprises: A plurality of alarm signals generated by the malicious traffic detection system within a preset time are obtained; Each of the plurality of alarm signals is associated with a traffic feature vector of a preset length, and all the traffic feature vectors are combined to obtain a first matrix; Each element of the first matrix is subjected to logarithmic transformation to obtain a second matrix, and the second matrix is subjected to maximum-minimum normalization processing to obtain a point cloud represented by a third matrix.

2. The method of claim 1, wherein, The point cloud is covered by a first plurality of voxels, and a second plurality of voxels is determined based on the point cloud coverage result; A first plurality of voxels is obtained based on a cube of a preset edge length in a multi-dimensional feature space; The first plurality of voxels is used to cover the points in the point cloud represented by the third matrix, and voxels in the first plurality of voxels that cover less than a preset number of points in the point cloud represented by the third matrix are removed to obtain a second plurality of voxels.

3. The method of claim 2, wherein, The voxel group is constructed by performing a clustering operation on the second plurality of voxels; Indexes corresponding to the second plurality of voxels are obtained; The Manhattan distance between voxels represented by the indexes is calculated, and voxel adjacency relationships are obtained based on a comparison result of the Manhattan distance and a first distance threshold; Based on the voxel adjacency relationships, voxels that are adjacent to each other in the second plurality of voxels are represented as a voxel group.

4. The method of claim 3, wherein, The high-density voxel groups and the low-density voxel groups are clustered based on the voxel group, and alarm signals corresponding to the high-density voxel groups and the low-density voxel groups are identified to obtain true positive alarm identification results and false positive alarm identification results, respectively. Density feature vectors are obtained based on a plurality of density features of the voxel group; All the density feature vectors are clustered using a preset clustering algorithm, so that a voxel group corresponding to a density feature vector whose distance from the clustering center is greater than a second distance threshold is identified as a voxel group in a high-density region, and vice versa, as a voxel group in a low-density region; Alarm signals corresponding to the points represented by all voxels in the voxel groups in the high-density region and the low-density region are identified to obtain true positive alarm identification results and false positive alarm identification results, respectively.

5. A false positive washing device for a malicious traffic detection system based on point cloud analysis, characterized in that, The method comprises the following steps: A point cloud is obtained based on the traffic feature vectors associated with the alarm signals generated by the malicious traffic detection system; A first plurality of voxels is used to cover the point cloud, and a second plurality of voxels is determined based on the point cloud coverage result; A voxel group is constructed by performing a clustering operation on the second plurality of voxels; The density learning module is configured to identify alarm signals corresponding to high-density voxel groups and low-density voxel groups respectively obtained based on the voxel group clustering to obtain true positive alarm identification results and false positive alarm identification results. The point cloud obtaining module is further configured to: Obtain a plurality of alarm signals of the malicious traffic detection system within a preset time; Associate each alarm signal of the plurality of alarm signals with a traffic feature vector of a preset length, and combine all the traffic feature vectors to obtain a first matrix; Perform logarithmic transformation on each element of the first matrix to obtain a second matrix, and perform maximum minimum normalization processing on the second matrix to obtain a third matrix representing a point cloud.

6. The apparatus of claim 5, wherein, The voxel constructing module is further configured to: Obtain a first plurality of voxels based on a cube of a preset edge length in a multi-dimensional feature space; Cover points in the point cloud represented by the third matrix with the first plurality of voxels, and remove voxels in the first plurality of voxels that cover less than a preset number of points in the point cloud represented by the third matrix to obtain a second plurality of voxels.

7. The apparatus of claim 6, wherein, The voxel clustering module is further configured to: Obtain indexes corresponding to the second plurality of voxels; Calculate Manhattan distances between voxels represented by the indexes, and obtain voxel adjacency relationships based on a comparison result of the Manhattan distances and a first distance threshold; Represent voxels that are adjacent to each other in the second plurality of voxels as voxel groups based on the voxel adjacency relationships.

8. The apparatus of claim 7, wherein, The density learning module is further configured to: Obtain density feature vectors based on a plurality of density features of the extracted voxel groups; Cluster all the density feature vectors using a preset clustering algorithm to identify voxel groups corresponding to density feature vectors whose distances from a clustering center are greater than a second distance threshold as voxel groups in a high-density region, and otherwise, as voxel groups in a low-density region; Identify alarm signals corresponding to points represented by all voxels in the voxel groups in the high-density region and the low-density region respectively to obtain true positive alarm identification results and false positive alarm identification results.

Images