A data center traffic anomaly monitoring method and system

Abstract

The present invention relates to the field of data flow monitoring technology, and specifically to a data center flow anomaly monitoring method and system. The present invention first obtains all flow data and corresponding type labels of a data center, and uses the flow data to build a random forest model. In the attribute splitting process of each non-leaf node of each decision tree of the random forest model, combined with the influence of the abnormal type imbalance of the sample flow data on the attribute splitting result, the preferred degree of each candidate attribute of each non-leaf node is analyzed and evaluated, and then the optimal splitting attribute of the non-leaf node is screened out; based on the optimal splitting attribute, the corresponding non-leaf node is recursively split to obtain a constructed random forest model; and then based on the constructed random forest model, the data center flow is monitored abnormally, which improves the construction effect of the decision tree, and then improves the accuracy of the random forest model in monitoring the abnormality of the data center flow.

Description

Technical Field

[0001] The present invention relates to the technical field of data flow monitoring, and in particular to a method and system for monitoring data center flow anomaly. Background Art

[0002] Data center traffic refers to the amount of data transmitted inside and outside the data center, and is a key indicator for measuring the operating efficiency and performance of the data center. It not only supports a variety of services and applications such as online business, parallel applications, and high-performance computing, but is also directly related to the security and integrity of data. By monitoring data center traffic, attacks on the data center can be discovered in a timely manner, and relevant processing measures can be taken to reduce losses and enhance the network's defense capabilities. Therefore, it is particularly important to effectively manage and monitor data center traffic.

[0003] In the prior art, the random forest algorithm is usually used to infer the abnormal type of data center traffic. However, the data center traffic in the training sample set may have an imbalance in sample types, and the samples used for random forest model training are randomly selected, which may cause the imbalance in sample types to be further amplified. As a result, during the construction process of the decision tree, excessive attention may be paid to categories with a relatively large proportion of samples in order to pursue the accuracy of the overall model in type judgment, while ignoring the type judgment results of data center traffic of a small number of samples, thereby affecting the decision tree construction effect, and further resulting in low accuracy in abnormal monitoring of data center traffic. Summary of the invention

[0004] In order to solve the technical problem of low accuracy of abnormal monitoring of data center traffic in the prior art, the purpose of the present invention is to provide a method and system for abnormal monitoring of data center traffic. The technical solution adopted is as follows:

[0005] The present invention provides a data center traffic anomaly monitoring method, the method comprising:

[0006] Acquire all traffic data of the data center, wherein the traffic data includes traffic attribute parameters under each attribute; acquire a type label of each traffic data, wherein the type label includes at least several abnormal labels;

[0007] A random forest model is constructed using the traffic data. In the attribute splitting process of each non-leaf node of each decision tree of the random forest model, the optimal splitting attribute of the non-leaf node is screened out from all the candidate attributes according to the preference degree of each candidate attribute of the non-leaf node; recursive splitting of the corresponding non-leaf node is performed based on the optimal splitting attribute until a preset splitting termination condition is met, thereby obtaining a constructed random forest model;

[0008] Anomaly monitoring of data center traffic based on the constructed random forest model;

[0009] The method for obtaining the preference level of each candidate attribute of a non-leaf node includes:

[0010] In the splitting result of each candidate attribute of each non-leaf node, according to the distribution of abnormal labels of the flow data in each splitting node, and the deviation of the distribution of abnormal labels of the flow data in the corresponding non-leaf node, the splitting node is divided into a valid splitting node and an invalid splitting node; according to the difference between the valid splitting node and the invalid splitting node, the first splitting effect coefficient of each candidate attribute of each non-leaf node is obtained;

[0011] In the splitting result of each candidate attribute of each non-leaf node, according to the distribution ratio of the flow data of each abnormal label in each invalid splitting node, the deviation of the distribution ratio of the flow data of the same abnormal label in the corresponding non-leaf node, combined with the difference in the distribution ratio of the flow data of each abnormal label in different invalid splitting nodes, obtain the second splitting effect coefficient of each candidate attribute of each non-leaf node on the flow data of each abnormal label;

[0012] According to the first splitting effect coefficient and the second splitting effect coefficient, the preference level of each candidate attribute corresponding to the non-leaf node in each decision tree is obtained.

[0013] Furthermore, the method for obtaining the valid split node and the invalid split node includes:

[0014] In each non-leaf node, a first distribution vector is constructed according to the data proportion of the traffic data under each abnormal label in all the traffic data; in each split node under each to-be-selected attribute of each non-leaf node, a second distribution vector is constructed according to the data proportion of the traffic data under each abnormal label in all the traffic data;

[0015] Obtaining a distribution similarity coefficient between each split node and a corresponding non-cotyledon node according to a vector difference between the first distribution vector and the second distribution vector; the vector difference is negatively correlated with the distribution similarity coefficient;

[0016] According to the distribution similarity coefficient, all split nodes under each candidate attribute of each non-leaf node are divided into two categories; according to the concentrated characteristics of the distribution similarity coefficients of all split nodes and non-cotyledon nodes in each category of split nodes, the poor splitting parameters of each category of split nodes are obtained, and the category of split nodes with the largest poor splitting parameters are regarded as invalid splitting nodes, and the other category of splitting nodes are regarded as valid splitting nodes.

[0017] Furthermore, the method of classifying all split nodes under each to-be-selected attribute of each non-leaf node into two categories according to the distribution similarity coefficient includes:

[0018] The difference between the different distribution similarity coefficients is used as the metric distance between the corresponding different split nodes, and all split nodes are clustered based on the preset K value and K-means algorithm to obtain two types of split nodes; the preset K value is 2.

[0019] Furthermore, the method for obtaining the first splitting effect coefficient includes:

[0020] Acquire a first effect parameter according to a difference in the number of the valid split nodes and the invalid split nodes;

[0021] Obtaining a second effect parameter according to a parameter difference between the poor splitting parameters of the splitting nodes of the category corresponding to the valid splitting node and the invalid splitting node; the parameter difference is positively correlated with the second effect parameter;

[0022] The first effect parameter and the second effect parameter are integrated to obtain a first splitting effect coefficient of a corresponding attribute to be selected of a corresponding non-leaf node.

[0023] Furthermore, the method for obtaining the second splitting effect coefficient includes:

[0024] Take any abnormal label as the target label; in each non-leaf node, take the data ratio of the traffic data under the target label in all the traffic data as the initial data ratio; in each invalid split node under each to-be-selected attribute of each non-leaf node, take the data ratio of the traffic data under the target label in all the traffic data as the split data ratio;

[0025] Among all invalid split nodes under each to-be-selected attribute of each non-leaf node, any invalid split node is used as a target node, and any invalid split node other than the target node is used as a reference node; according to the difference in the proportion of the split data between the target node and the reference node, a first split difference between the target node and the reference node is obtained;

[0026] The absolute value of the difference between the split data ratio of the target node and the initial data ratio is used as the target split difference; the absolute value of the difference between the split data ratio of the reference node and the initial data ratio is used as the reference split difference; the sum of the target split difference and the reference split difference is used as the second split difference;

[0027] The first split difference and the second split difference between different invalid split nodes are integrated to obtain the split difference parameters between different invalid split nodes; the split difference parameters between all different invalid split nodes are integrated to obtain the second split effect coefficient of each candidate attribute of each non-leaf node on the traffic data under the target label.

[0028] Furthermore, the method for obtaining the degree of preference includes:

[0029] Among the second splitting effect coefficients of each candidate attribute of each non-leaf node for the traffic data of each abnormal label, the largest second splitting effect coefficient is screened out as the representative splitting effect coefficient; the product of the first splitting effect coefficient and the representative splitting effect coefficient is used as the priority of the corresponding candidate attribute of the corresponding non-leaf node in the corresponding decision tree.

[0030] Furthermore, the method for obtaining the optimal splitting attribute includes:

[0031] Among all candidate attributes of non-leaf nodes, the candidate attribute corresponding to the maximum preference degree is used as the optimal splitting attribute of the non-leaf node.

[0032] Furthermore, all candidate attributes of each non-leaf node do not include the optimal splitting attributes of all upper-level nodes of the non-leaf node in the decision path to which the non-leaf node belongs.

[0033] Furthermore, the method for performing abnormal monitoring on data center traffic based on the constructed random forest model includes:

[0034] The current traffic data of the data center is input into the constructed random forest model, and the type label of the current traffic data is output; if the type label of the current traffic data is any abnormal label, an abnormal warning is issued.

[0035] The present invention also proposes a data center traffic anomaly monitoring system, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the data center traffic anomaly monitoring method are implemented.

[0036] The present invention has the following beneficial effects:

[0037] The present invention first obtains all traffic data and corresponding type labels of a data center, and uses the traffic data to build a random forest model. In the attribute splitting process of each non-leaf node of each decision tree of the random forest model, the influence of the abnormal type imbalance of sample traffic data on the attribute splitting result is combined to analyze and evaluate the preference degree of each candidate attribute in the splitting process of each non-leaf node, and then screen out the optimal splitting attribute of the non-leaf node; recursively split the corresponding non-leaf nodes based on the optimal splitting attribute until the preset splitting termination condition is met, and obtain the constructed random forest model; and then perform abnormal monitoring of data center traffic based on the constructed random forest model, thereby improving the construction effect of the decision tree, and then improving the accuracy of the random forest model in abnormal monitoring of data center traffic.

[0038] Among them, in the splitting result of each candidate attribute of each non-leaf node, according to the distribution of abnormal labels of traffic data in each splitting node, relative to the deviation of the distribution of abnormal labels of traffic data in the corresponding non-leaf node, the splitting node is divided into a valid splitting node and an invalid splitting node; according to the difference between the valid splitting node and the invalid splitting node, the first splitting effect coefficient of each candidate attribute of each non-leaf node is obtained; the first splitting effect coefficient takes into account that traditional standards such as information gain do not consider the particularity of abnormal labels, which may cause the splitting direction with potential abnormal characteristics to be ignored when the decision tree splits, so it is based on the analysis and evaluation of the distribution of abnormal labels, so as to improve the screening of subsequent candidate attributes Selection accuracy; then, in the splitting results of each candidate attribute of each non-leaf node, according to the distribution ratio of the flow data of each abnormal label in each invalid splitting node, the deviation of the distribution ratio of the flow data of the same abnormal label in the corresponding non-leaf node, combined with the difference in the distribution ratio of the flow data of each abnormal label in different invalid splitting nodes, obtain the second splitting effect coefficient of each candidate attribute of each non-leaf node on the flow data of each abnormal label; the second splitting effect coefficient indirectly reflects the interference effect of sample imbalance on the splitting effect, thereby adjusting the first splitting effect coefficient, and accurately obtaining the preference degree of each candidate attribute corresponding to the non-leaf node in each decision tree. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 A flow chart of a data center traffic anomaly monitoring method provided by an embodiment of the present invention;

[0041] Figure 2 A flow chart of a method for obtaining the preference level of each candidate attribute of a non-leaf node provided by an embodiment of the present invention;

[0042] Figure 3 A flow chart of a method for obtaining a valid split node and an invalid split node provided by an embodiment of the present invention;

[0043] Figure 4 A flow chart of a method for obtaining a second splitting effect coefficient provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of a data center traffic anomaly monitoring method and system proposed by the present invention, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0045] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0046] The specific scheme of a data center traffic anomaly monitoring method and system provided by the present invention is described in detail below with reference to the accompanying drawings.

[0047] See also Figure 1 , which shows a flow chart of a data center traffic anomaly monitoring method provided by an embodiment of the present invention, specifically comprising:

[0048] Step S1, obtaining all traffic data of the data center, the traffic data including traffic attribute parameters under each attribute; obtaining the type label of each traffic data, the type label including at least several types of abnormal labels.

[0049] In one embodiment of the present invention, firstly, a network traffic monitoring tool such as NetFlow or sFlow is used to capture the traffic data of a data center in real time every minute. The traffic data includes traffic attribute parameters under each attribute, wherein the attributes at least include CPU utilization, memory utilization, traffic bandwidth, inbound traffic, outbound traffic, port number, source IP address, HTTP request, etc., so that the traffic attribute parameters under the corresponding attributes can be obtained; then all the traffic attribute parameters are used as vector elements to construct a multidimensional vector, wherein each dimension corresponds to an attribute, and the multidimensional vector is used as traffic data; then all the traffic data of the data center in a preset historical period, such as a historical year, is obtained, and all the traffic data is used as sample data to provide a data basis for the subsequent construction of a random forest model;

[0050] Then, each traffic data is assigned a type label through manual annotation, where the type labels are divided into normal labels and several types of abnormal labels. When the traffic data shows normal access or interaction with the data center, it is assigned a normal label; when the traffic data shows that the data center is under attack, it is assigned an abnormal label. Among them, the types of abnormal labels include at least SQL Injection, DoS attack, FTP-BruteForce, etc.

[0051] It should be noted that, in other embodiments of the present invention, the implementer may also use other network traffic monitoring tools to obtain traffic data, may also set other types or quantities of attributes, and may also set other types or quantities of abnormal labels by themselves; among them, using network traffic monitoring tools to capture traffic data, analyze data attack types and manually label them are all existing technologies and will not be repeated here.

[0052] Step S2, using traffic data to build a random forest model. In the attribute splitting process of each non-leaf node of each decision tree of the random forest model, the optimal splitting attribute of the non-leaf node is screened out from all the candidate attributes according to the preference degree of each candidate attribute of the non-leaf node; based on the optimal splitting attribute, the corresponding non-leaf nodes are recursively split until the preset splitting termination condition is met, and the constructed random forest model is obtained.

[0053] After obtaining all the traffic data as samples, a random forest model can be constructed based on the samples; considering that when constructing the decision tree of the random forest model, it may be affected by the uneven distribution of sample types, resulting in poor decision tree construction effect, the embodiment of the present invention will, in the attribute splitting process of each non-leaf node of each decision tree, screen out the optimal splitting attribute of the non-leaf node from all the candidate attributes according to the preference degree of each candidate attribute of the non-leaf node, thereby facilitating the subsequent improvement of the decision tree construction effect.

[0054] To facilitate understanding of the solution, the construction process of each decision tree in the random forest provided by an embodiment of the present invention is briefly described here: randomly sample with replacement from the sample data to obtain training data for each decision tree; use the data set corresponding to the training data as the root node data set of the decision tree, screen out the optimal splitting attribute from all attributes of the sample data to split the root node data set, and obtain all split nodes of the root node; then repeat the splitting process of the root node, further split each split node, and iterate continuously until the preset split termination condition is met, and stop, and use all split nodes obtained from the last split as leaf nodes, and all nodes participating in the split including the root node as non-leaf nodes; the split path corresponding to "root node-non-leaf node-non-leaf node-...-non-leaf node-leaf node" is a decision path.

[0055] Among them, in one embodiment of the present invention, the preset split termination condition is that a suitable split feature cannot be found for splitting. In other embodiments, the implementer may also set the preset split termination condition to the depth of the decision tree reaching a preset value, the total amount of traffic data in the split node being less than a preset threshold, etc. The construction and application of the random forest model are existing technologies and will not be elaborated herein.

[0056] Since the optimal splitting attributes of all the parent nodes of each non-leaf node on its decision path have been used for sample partitioning, if the optimal splitting attributes of the parent nodes are still used for splitting, it may cause overfitting of the decision tree and fall into a local optimal solution, thereby affecting the balance of the decision tree structure, resulting in poor model generalization ability and construction effect; based on this, in a preferred embodiment of the present invention, all the candidate attributes of each non-leaf node do not include the optimal splitting attributes of all the parent nodes of the non-leaf node in the decision path to which the non-leaf node belongs.

[0057] The method for obtaining the preference level of each candidate attribute of a non-leaf node includes:

[0058] See also Figure 2 , which shows a flow chart of a method for obtaining the preference level of each candidate attribute of a non-leaf node provided by an embodiment of the present invention, specifically comprising:

[0059] Step S201, in the splitting results of each candidate attribute of each non-leaf node, according to the distribution of abnormal labels of traffic data in each splitting node and the deviation of the distribution of abnormal labels of traffic data in the corresponding non-leaf node, the splitting node is divided into a valid splitting node and an invalid splitting node; according to the difference between the valid splitting node and the invalid splitting node, the first splitting effect coefficient of each candidate attribute of each non-leaf node is obtained.

[0060] It should be noted that the splitting results and corresponding splitting nodes of each candidate attribute of each non-leaf node are not the splitting nodes in the already constructed decision tree, but the intermediate analysis process information in the process of screening the optimal splitting attributes during the attribute splitting process of each non-leaf node of each decision tree.

[0061] Considering that in the splitting process of each non-leaf node, an optimal attribute is usually selected from all the corresponding features of the attribute for splitting; considering that in the screening process of the optimal attribute, traditional standards such as information gain or Gini index are usually used, however, traditional splitting standards focus on the overall distribution of sample labels to maximize the purity of traffic data in the splitting node as much as possible, and for tasks such as abnormal monitoring of traffic data in data centers, traditional standards such as information gain do not consider the particularity of abnormal labels, which may cause the decision tree to ignore the splitting direction with potential abnormal features when splitting. Therefore, special attention should be paid to the distribution of abnormal labels, so as to improve the accuracy of subsequent random forest models in monitoring data center traffic anomalies;

[0062] Based on this, the embodiment of the present invention first divides the splitting nodes into valid splitting nodes and invalid splitting nodes in the splitting results of each candidate attribute of each non-leaf node according to the distribution of abnormal labels of the traffic data in each splitting node and the deviation of the distribution of abnormal labels of the traffic data in the corresponding non-leaf node; and then obtains the first splitting effect coefficient of each candidate attribute of each non-leaf node according to the difference between the valid splitting nodes and the invalid splitting nodes.

[0063] In an ideal splitting situation, it is hoped that the abnormal label distribution of the traffic data in the split node after the split can be relatively different from the overall abnormal label distribution in its parent node, that is, the corresponding non-leaf node. That is, this splitting can effectively divide the abnormal data, so that the decision rule corresponding to this splitting process can accurately distinguish the abnormal labels;

[0064] Therefore, an effective split node is a split node that has a large distribution deviation from the abnormal type distribution of traffic data in non-leaf nodes, and it can effectively distinguish some abnormal labels; on the contrary, an invalid split node is a split node that is relatively similar to the abnormal type distribution of traffic data in non-leaf nodes, and its ability to distinguish abnormal labels is weak.

[0065] Preferably, in one embodiment of the present invention, the method for obtaining valid split nodes and invalid split nodes includes:

[0066] See also Figure 3 , which shows a flow chart of a method for obtaining a valid split node and an invalid split node provided by an embodiment of the present invention, specifically comprising:

[0067] Step S211, in each non-leaf node, construct a first distribution vector according to the data proportion of the traffic data under each abnormal label in all the traffic data; in each split node under each selected attribute of each non-leaf node, construct a second distribution vector according to the data proportion of the traffic data under each abnormal label in all the traffic data.

[0068] As an example, in each non-leaf node, the proportion of traffic data under each abnormal label in all traffic data is counted, and the data proportion is used as a vector element to construct a multidimensional vector to obtain a first distribution vector; similarly, in each split node under each candidate attribute of each non-leaf node, a second distribution vector is constructed to prepare for the subsequent evaluation of distribution deviation.

[0069] It should be noted that the number or type of abnormal labels in each non-leaf node and the corresponding split node may not be consistent. For example, if the abnormal label in the non-leaf node is ABCDEF, and the abnormal label in the split node is ABD, then when constructing the second distribution vector, the vector dimension needs to correspond one-to-one with the vector dimension in the non-leaf node for subsequent comparative analysis; that is, when there is no corresponding abnormal label in the split node, the data proportion under the corresponding dimension in the second distribution vector is set to 0.

[0070] Step S212, obtaining a distribution similarity coefficient between each split node and the corresponding non-cotyledon node according to the vector difference between the first distribution vector and the second distribution vector; the vector difference is negatively correlated with the distribution similarity coefficient.

[0071] As an example, taking any non-cotyledon node and any of its split nodes as an example, the calculation formula for the distribution similarity coefficient is:

[0072] ; Where s is the distribution similarity coefficient between the split node and the corresponding non-cotyledon node; v is the second distribution vector of the split node; is the first distribution vector of non-leaf nodes; is the symbol of Euclidean norm; is the vector difference between the first distribution vector and the second distribution vector; is the standard normalization function.

[0073] In this example, the Euclidean norm is used to measure the vector difference. The larger the Euclidean norm, the greater the vector difference. The function form of maps the vector differences in a negative correlation, adjusts the logic and normalizes them so that the smaller the vector difference, the larger the distribution similarity coefficient.

[0074] In other examples, the Pearson correlation coefficient or cosine similarity can also be used to evaluate the similarity between vectors. The greater the similarity, the greater the distribution similarity coefficient; or the difference measurement method such as Manhattan distance can be used to directly measure the vector difference, and then negative correlation mapping can be performed to obtain the distribution similarity coefficient; implementers can also use other negative correlation mapping methods, such as taking the inverse or using the vector difference as the x in the exponential function exp(-x) with the natural constant e as the base, which are all existing technologies and will not be elaborated here.

[0075] Step S213, according to the distribution similarity coefficient, divide all split nodes under each candidate attribute of each non-leaf node into two categories; according to the concentrated characteristics of the distribution similarity coefficients of all split nodes and non-cotyledon nodes in each category of split nodes, obtain the poor splitting parameters of each category of split nodes, and take the category of split nodes with the largest poor splitting parameters as invalid splitting nodes, and take the other category of split nodes as valid splitting nodes.

[0076] After obtaining the distribution similarity coefficient between each split node and the corresponding non-leaf node, all split nodes can be divided into two categories, namely, valid split nodes and invalid split nodes.

[0077] Preferably, in one embodiment of the present invention, considering that clustering can cluster similar data into one cluster, the difference between different distribution similarity coefficients is used as the metric distance between corresponding different split nodes, and all split nodes are clustered based on the preset K value and K-means algorithm to obtain two types of split nodes; the preset K value is 2.

[0078] As an example, the absolute value of the difference in distribution similarity coefficients between different split nodes and non-leaf nodes is used as the metric distance between the corresponding different split nodes, and then the split nodes are classified based on the distance clustering algorithm K-means algorithm; then in each cluster cluster, the mean of the distribution similarity coefficients of all split nodes with non-leaf nodes is used as the poor splitting parameter of each type of split node; the larger the poor splitting parameter, the worse the effect of the splitting process on distinguishing abnormal traffic data, and the corresponding category of split nodes is an invalid splitting node, and the other category is a valid splitting node.

[0079] It should be noted that the K-means algorithm is an existing technology well known to those skilled in the art. In other embodiments, the implementer may also adopt other clustering algorithms, and may also classify the split nodes based on the threshold segmentation method, such as using the median as the preset threshold, and treating all split nodes greater than or equal to the preset threshold as invalid split nodes, and all split nodes less than the preset threshold as valid split nodes.

[0080] In the attribute splitting process of each non-leaf node of each decision tree, if the difference in the number of valid split nodes and invalid split nodes is greater, and the number of valid split nodes is greater, it means that the candidate attribute can effectively distinguish the traffic data corresponding to different types of abnormal labels; and if there are more valid split nodes, if there is a large difference between the splitting poor parameters corresponding to all valid split nodes and all invalid split nodes between the valid split node category and the invalid split node category, and the smaller the splitting poor parameters corresponding to all valid split nodes, it can also be said that the splitting effect of the candidate attribute is better, and it can effectively distinguish the traffic data corresponding to different types of abnormal labels;

[0081] Based on this, in a preferred embodiment of the present invention, a method for obtaining the first splitting effect coefficient includes:

[0082] Obtaining a first effect parameter according to a difference in the number of valid split nodes and the number of invalid split nodes;

[0083] Obtaining a second effect parameter according to a parameter difference between the poor splitting parameters of the splitting nodes of the corresponding categories of the effective splitting nodes and the invalid splitting nodes; the parameter difference is positively correlated with the second effect parameter;

[0084] The first effect parameter and the second effect parameter are integrated to obtain a first splitting effect coefficient of a corresponding attribute to be selected of a corresponding non-leaf node.

[0085] As an example, taking the splitting process of any non-leaf node under any candidate attribute as an example, the calculation formula of the first splitting effect coefficient is:

[0086] ; Wherein, y is the first splitting effect coefficient of the corresponding attribute to be selected of the non-leaf node; M1 is the category code of the invalid splitting node; M2 is the category code of the valid splitting node; is the total number of valid split nodes; is the total number of invalid split nodes; is the distribution similarity coefficient between each split node and the corresponding non-cotyledon node; is the first effect parameter; is the mean of the distribution similarity coefficients of all invalid split nodes, and is also the poor splitting parameter of the corresponding category split nodes of the invalid split nodes; It is the mean of the distribution similarity coefficients of all valid split nodes, and also the poor splitting parameter of the corresponding category splitting node of the valid split node; is the second effect parameter.

[0087] In this example, the first effect parameter is evaluated in the form of a ratio, so that the greater the difference in the number of valid split nodes and invalid split nodes, and the more valid split nodes, the larger the first effect parameter; the second effect parameter is evaluated in the form of a difference, so that there is a large difference between the poor splitting parameters corresponding to all valid splitting nodes and all invalid splitting nodes, and the smaller the poor splitting parameters corresponding to all valid splitting nodes, the larger the second effect parameter; and then the first effect parameter and the second effect parameter are combined by multiplication and fusion to obtain the first splitting effect coefficient of the corresponding candidate attribute of the corresponding non-leaf node.

[0088] In other examples, implementers can also use the difference form such as To determine the first effect coefficient, the first effect parameter and the second effect parameter may also be combined by other basic mathematical operations such as addition or weighted summation or related mapping means, which will not be described in detail herein.

[0089] Step S202, in the splitting results of each candidate attribute of each non-leaf node, according to the distribution ratio of the traffic data of each abnormal label in each invalid splitting node, the deviation of the distribution ratio of the traffic data of the same abnormal label in the corresponding non-leaf node, combined with the difference in the distribution ratio of the traffic data of each abnormal label in different invalid splitting nodes, obtain the second splitting effect coefficient of each candidate attribute of each non-leaf node on the traffic data of each abnormal label.

[0090] Considering that under normal circumstances, the traffic data with normal labels will be far more than the traffic data with abnormal labels, and the traffic data with different abnormal labels may also have a certain imbalance. In the process of randomly sampling to obtain the training data for each decision tree, the imbalance of sample types may be further amplified, which will cause the decision tree to be biased towards abnormal traffic types with significantly more numbers, thereby ignoring a small number of abnormal traffic types, which will lead to poor construction effect of the decision tree.

[0091] Considering that in each splitting node, the distribution proportion of traffic data with different abnormal labels in all traffic data is different, the distribution change of traffic data corresponding to a large number of abnormal labels usually masks the distribution change of traffic data corresponding to a small number of abnormal labels, which may cause the distribution change of traffic data corresponding to a small number of abnormal labels to be easily ignored when evaluating the distribution similarity coefficient, thereby causing interference to the evaluation of the splitting effect of the candidate attribute, making the obtained splitting effect coefficient may not be accurate enough;

[0092] Therefore, in order to reduce the impact of unbalanced sample types on the construction of the decision tree, the embodiment of the present invention will further obtain the second splitting effect coefficient of each candidate attribute of each non-leaf node on the traffic data of each abnormal label in each invalid splitting node according to the distribution ratio of the traffic data of each abnormal label in each invalid splitting node, combined with the difference in the distribution ratio of the traffic data of each abnormal label in different invalid splitting nodes in the splitting result of each candidate attribute of each non-leaf node; the second splitting effect coefficient reflects the interference effect of sample imbalance on the splitting, which is convenient for the subsequent combination with the first splitting effect coefficient to accurately evaluate the preference degree of each candidate attribute.

[0093] Preferably, in one embodiment of the present invention, the method for obtaining the second splitting effect coefficient includes:

[0094] See also Figure 4 , which shows a flow chart of a method for obtaining a second splitting effect coefficient provided by an embodiment of the present invention, specifically comprising:

[0095] Step S221, take any abnormal label as the target label; in each non-leaf node, take the data ratio of the traffic data under the target label in all the traffic data as the initial data ratio; in each invalid split node under each to-be-selected attribute of each non-leaf node, take the data ratio of the traffic data under the target label in all the traffic data as the split data ratio.

[0096] As an example, take any abnormal label as the target label for analysis and description. In each non-leaf node, the data ratio of the traffic data under the target label in all the traffic data is counted to obtain the initial data ratio. Similarly, in each invalid split node under each candidate attribute of each non-leaf node, the data ratio of the traffic data under the target label in all the traffic data is counted to obtain the split data ratio. This provides data preparation for subsequent analysis.

[0097] Step S222, among all invalid split nodes under each to-be-selected attribute of each non-leaf node, take any invalid split node as the target node, and take any invalid split node other than the target node as the reference node; based on the difference in the proportion of split data between the target node and the reference node, obtain the first split difference between the target node and the reference node.

[0098] Considering that among all the invalid split nodes under each candidate attribute of each non-leaf node, under the target label, if there is an obvious difference in the proportion of split data between different invalid split nodes, it means that the candidate attribute has a strong ability to distinguish the traffic data under the target label, so the first split difference between different invalid split nodes can be obtained; the first split difference reflects the ability of the candidate attribute to distinguish the traffic data under the target label, and the larger the first split difference, the better the splitting effect of the candidate attribute.

[0099] As an example, in order to facilitate the subsequent comparative analysis of the split differences between different invalid split nodes, among all the invalid split nodes under each candidate attribute of each non-leaf node, any invalid split node will be taken as the target node, and any invalid split node other than the target node will be taken as the reference node; then the absolute value of the difference in the proportion of split data between the target node and the reference node will be linearly normalized, and the normalized result will be taken as the first split difference; the implementer may also adopt other normalization methods, which will not be repeated here.

[0100] Step S223, take the absolute value of the difference between the split data ratio of the target node and the initial data ratio as the target split difference; take the absolute value of the difference between the split data ratio of the reference node and the initial data ratio as the reference split difference; take the sum of the target split difference and the reference split difference as the second split difference.

[0101] Considering that among all the invalid split nodes under each candidate attribute of each non-leaf node, under the target label, if there is an obvious difference between the split data ratio of the invalid split node and the initial data ratio, it means that the candidate attribute has a strong ability to distinguish the traffic data under the target label, and then the second split difference between different invalid split nodes can be obtained; the second split difference also indirectly reflects the ability of the candidate attribute to distinguish the traffic data under the target label. The larger the second split difference, the better the splitting effect of the candidate attribute.

[0102] In the method for obtaining the second split difference, the larger the target split difference and the reference split difference are, the more obvious the difference is between the split data ratio and the initial data ratio of the target node and the split node under the target label, further indicating that the selected attribute has a strong ability to distinguish the traffic data under the target label; the two are merged in the form of summation to obtain the second split difference; in other embodiments, the implementer may also use basic mathematical operations such as weighted summation or multiplication or related mapping methods to merge the two, which will not be repeated here.

[0103] Step S224, merge the first split difference and the second split difference between different invalid split nodes to obtain the split difference parameters between different invalid split nodes; combine the split difference parameters between all different invalid split nodes to obtain the second split effect coefficient of each candidate attribute of each non-leaf node on the traffic data under the target label.

[0104] As an example, the calculation formula for the second split effect coefficient is:

[0105] ;in, is the second splitting effect coefficient of the jth candidate attribute of the non-leaf node on the traffic data under the target label; k is the serial number of the target node; l is the serial number of the reference node; M is the total number of invalid splitting nodes among all splitting nodes under the jth candidate attribute of the non-leaf node; o is the serial number of the non-leaf node; i is the serial number of the target label; j is the serial number of the candidate attribute; It is the proportion of traffic data under the target label in non-leaf nodes in all traffic data, which is also the proportion of initial data; It is the proportion of traffic data under the target label in all traffic data in the kth target node under the jth candidate attribute of the non-leaf node, and also the proportion of split data of the kth target node under the target label; It is the proportion of traffic data under the target label in all traffic data in the lth reference node under the jth candidate attribute of the non-leaf node, and also the proportion of split data of the lth reference node under the target label; is the absolute value symbol; Splitting the difference for the goal; for the reference split difference; is the second split difference between the kth target node and the lth reference node; is the first split difference between the kth target node and the lth reference node under the jth candidate attribute of the non-leaf node; is the split difference parameter between the kth target node and the lth reference node under the jth candidate attribute of the non-leaf node.

[0106] In this example, the first split difference and the second split difference between different invalid split nodes, i.e., the target node and the reference node, are multiplied and combined to obtain the split difference parameters between the corresponding different invalid split nodes; then, the split difference parameters between all different invalid split nodes are accumulated to obtain the second split effect coefficient of each candidate attribute of each non-leaf node on the flow data under the target label; the larger the split difference parameter, the stronger the ability of the candidate attribute to distinguish the flow data under the target label, the higher the splitting effect, and the larger the second split effect coefficient.

[0107] In other examples, implementers may also use basic mathematical operations such as addition or weighted summation or related mapping means to merge the first split difference and the second split difference, which will not be described in detail here.

[0108] By changing the target label, the second splitting effect coefficient of each candidate attribute of each non-leaf node on the traffic data of each abnormal label can be obtained.

[0109] Step S203, obtaining the preference level of each candidate attribute corresponding to the non-leaf node in each decision tree according to the first splitting effect coefficient and the second splitting effect coefficient.

[0110] Considering that both the first splitting effect coefficient and the second splitting effect coefficient reflect the splitting effect of the corresponding candidate attribute, the better the splitting effect, the more accurate the ability to split the traffic data corresponding to the abnormal label under the premise of sample imbalance, thereby improving the construction effect of the decision data; therefore, the first splitting effect coefficient and the second splitting effect coefficient can be fused to obtain the preference degree of each candidate attribute corresponding to the non-leaf node in each decision tree.

[0111] Preferably, in one embodiment of the present invention, considering that the larger the second splitting effect coefficient is, the greater the degree of influence of sample imbalance in the splitting process is, which leads to the existence of invalid splitting nodes, but it may be caused by a small proportion of samples, and the actual splitting ability of the attribute may be stronger; therefore, a maximum second splitting effect coefficient can be selected as a representative splitting effect coefficient, and then combined with the first splitting effect coefficient, the flow data differentiation ability of each candidate attribute is evaluated to obtain its preferred degree; therefore, the method for obtaining the preferred degree includes:

[0112] Among the second splitting effect coefficients of each candidate attribute of each non-leaf node for the traffic data of each abnormal label, the largest second splitting effect coefficient is screened out as the representative splitting effect coefficient; the product of the first splitting effect coefficient and the representative splitting effect coefficient is used as the priority of the corresponding candidate attribute of the corresponding non-leaf node in the corresponding decision tree.

[0113] After obtaining the preference level of each candidate attribute of each non-leaf node, the optimal splitting attribute of the non-leaf node can be screened out from all the candidate attributes; in a preferred embodiment of the present invention, the method for obtaining the optimal splitting attribute includes: among all the candidate attributes of the non-leaf node, the candidate attribute corresponding to the maximum preference level is used as the optimal splitting attribute of the non-leaf node.

[0114] At this point, in the attribute splitting process of each non-leaf node of each decision tree of the random forest model, the optimal splitting attribute of each non-leaf node is obtained; then, the corresponding non-leaf nodes can be recursively split based on the optimal splitting attribute to obtain the constructed random forest model.

[0115] It should be noted that the construction of the random forest model is already an existing technology and will not be described in detail here.

[0116] Step S3, performing abnormal monitoring of data center traffic based on the constructed random forest model.

[0117] Preferably, in one embodiment of the present invention, the method for monitoring data center traffic anomalies based on the constructed random forest model includes:

[0118] The traffic data of the data center is collected in real time, and the new traffic data collected at the current moment is used as the current traffic data, which is input into the constructed random forest model, and the type label of the current traffic data is output; if the type label of the current traffic data is any abnormal label, an abnormal warning is issued; the relevant network security management personnel can take relevant defense measures in time to reduce losses and enhance the network's defense capabilities.

[0119] The present invention also proposes a data center traffic anomaly monitoring system, the system includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the above-mentioned data center traffic anomaly monitoring method when executing the computer program.

[0120] In summary, the present invention first obtains all traffic data and corresponding type labels of the data center, and uses the traffic data to build a random forest model. In the attribute splitting process of each non-leaf node of each decision tree of the random forest model, combined with the influence of the abnormal type imbalance of the sample traffic data on the attribute splitting result, the preferred degree of each candidate attribute in the splitting process of each non-leaf node is analyzed and evaluated, and then the optimal splitting attribute of the non-leaf node is screened out; based on the optimal splitting attribute, the corresponding non-leaf node is recursively split until the preset split termination condition is met, and the constructed random forest model is obtained; and then the data center traffic is monitored for abnormalities based on the constructed random forest model, which improves the construction effect of the decision tree, and then improves the accuracy of the random forest model in monitoring abnormalities of the data center traffic.

[0121] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0122] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A data center traffic anomaly monitoring method, characterized in that: The method comprises: Acquire all traffic data of the data center, wherein the traffic data includes traffic attribute parameters under each attribute; acquire a type label of each traffic data, wherein the type label includes at least several abnormal labels; A random forest model is constructed using the traffic data. In the attribute splitting process of each non-leaf node of each decision tree of the random forest model, the optimal splitting attribute of the non-leaf node is screened out from all the candidate attributes according to the preference degree of each candidate attribute of the non-leaf node; recursive splitting of the corresponding non-leaf node is performed based on the optimal splitting attribute until a preset splitting termination condition is met, thereby obtaining a constructed random forest model; Anomaly monitoring of data center traffic based on the constructed random forest model; The method for obtaining the preference level of each candidate attribute of a non-leaf node includes: In the splitting result of each candidate attribute of each non-leaf node, according to the distribution of abnormal labels of the flow data in each splitting node, and the deviation of the distribution of abnormal labels of the flow data in the corresponding non-leaf node, the splitting node is divided into a valid splitting node and an invalid splitting node; according to the difference between the valid splitting node and the invalid splitting node, the first splitting effect coefficient of each candidate attribute of each non-leaf node is obtained; In the splitting result of each candidate attribute of each non-leaf node, according to the distribution ratio of the flow data of each abnormal label in each invalid splitting node, the deviation of the distribution ratio of the flow data of the same abnormal label in the corresponding non-leaf node, combined with the difference in the distribution ratio of the flow data of each abnormal label in different invalid splitting nodes, obtain the second splitting effect coefficient of each candidate attribute of each non-leaf node on the flow data of each abnormal label; According to the first splitting effect coefficient and the second splitting effect coefficient, the preference level of each candidate attribute corresponding to the non-leaf node in each decision tree is obtained.

2. A data center traffic anomaly monitoring method according to claim 1, characterized in that: The method for obtaining the valid split node and the invalid split node includes: In each non-leaf node, a first distribution vector is constructed according to the data proportion of the traffic data under each abnormal label in all the traffic data; in each split node under each to-be-selected attribute of each non-leaf node, a second distribution vector is constructed according to the data proportion of the traffic data under each abnormal label in all the traffic data; Obtaining a distribution similarity coefficient between each split node and a corresponding non-cotyledon node according to a vector difference between the first distribution vector and the second distribution vector; the vector difference is negatively correlated with the distribution similarity coefficient; According to the distribution similarity coefficient, all split nodes under each candidate attribute of each non-leaf node are divided into two categories; according to the concentrated characteristics of the distribution similarity coefficients of all split nodes and non-cotyledon nodes in each category of split nodes, the poor splitting parameters of each category of split nodes are obtained, and the category of split nodes with the largest poor splitting parameters are regarded as invalid splitting nodes, and the other category of splitting nodes are regarded as valid splitting nodes.

3. A data center traffic anomaly monitoring method according to claim 2, characterized in that: The method of classifying all split nodes under each candidate attribute of each non-leaf node into two categories according to the distribution similarity coefficient includes: The difference between the different distribution similarity coefficients is used as the metric distance between the corresponding different split nodes, and all split nodes are clustered based on the preset K value and K-means algorithm to obtain two types of split nodes; the preset K value is 2.

4. A data center traffic anomaly monitoring method according to claim 2, characterized in that: The method for obtaining the first splitting effect coefficient includes: Acquire a first effect parameter according to a difference in the number of the valid split nodes and the invalid split nodes; According to the parameter difference between the effective splitting node and the invalid splitting node and the poor splitting parameters of the corresponding category splitting nodes, a second effect parameter is obtained; the parameter difference is positively correlated with the second effect parameter; The first effect parameter and the second effect parameter are integrated to obtain a first splitting effect coefficient of a corresponding attribute to be selected of a corresponding non-leaf node.

5. A data center traffic anomaly monitoring method according to claim 1, characterized in that: The method for obtaining the second splitting effect coefficient includes: Take any abnormal label as the target label; in each non-leaf node, take the data ratio of the traffic data under the target label in all the traffic data as the initial data ratio; in each invalid split node under each to-be-selected attribute of each non-leaf node, take the data ratio of the traffic data under the target label in all the traffic data as the split data ratio; Among all invalid split nodes under each to-be-selected attribute of each non-leaf node, any invalid split node is used as a target node, and any invalid split node other than the target node is used as a reference node; according to the difference in the proportion of the split data between the target node and the reference node, a first split difference between the target node and the reference node is obtained; The absolute value of the difference between the split data ratio of the target node and the initial data ratio is used as the target split difference; the absolute value of the difference between the split data ratio of the reference node and the initial data ratio is used as the reference split difference; the sum of the target split difference and the reference split difference is used as the second split difference; The first split difference and the second split difference between different invalid split nodes are integrated to obtain the split difference parameters between different invalid split nodes; the split difference parameters between all different invalid split nodes are integrated to obtain the second split effect coefficient of each candidate attribute of each non-leaf node on the traffic data under the target label.

6. A data center traffic anomaly monitoring method according to claim 1, characterized in that: The method for obtaining the preference degree includes: Among the second splitting effect coefficients of each candidate attribute of each non-leaf node for the traffic data of each abnormal label, the largest second splitting effect coefficient is screened out as the representative splitting effect coefficient; the product of the first splitting effect coefficient and the representative splitting effect coefficient is used as the priority of the corresponding candidate attribute of the corresponding non-leaf node in the corresponding decision tree.

7. A data center traffic anomaly monitoring method according to claim 1, characterized in that: The method for obtaining the optimal splitting attribute includes: Among all candidate attributes of non-leaf nodes, the candidate attribute corresponding to the maximum preference degree is used as the optimal splitting attribute of the non-leaf node.

8. A data center traffic anomaly monitoring method according to claim 1, characterized in that: All candidate attributes of each non-leaf node do not include the optimal splitting attributes of all parent nodes of the non-leaf node in the decision path to which the non-leaf node belongs.

9. A data center traffic anomaly monitoring method according to claim 1, characterized in that: The method for monitoring data center traffic anomalies based on the constructed random forest model includes: The current traffic data of the data center is input into the constructed random forest model, and the type label of the current traffic data is output; if the type label of the current traffic data is any abnormal label, an abnormal warning is issued.

10. A data center traffic anomaly monitoring system, characterized in that: The system includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of a data center traffic anomaly monitoring method as described in any one of claims 1 to 9 are implemented.

Images