Network traffic anomaly detection method based on Bayesian optimization

Abstract

The present invention discloses a network traffic anomaly detection method based on Bayesian optimization, and relates to the technical field of data anomaly detection. According to whether the original traffic data is the same, the historical network abnormal traffic data passing through the same transmission line is divided into the same transmission data set and the difference transmission data set, and then the abnormal data overlap evaluation values ​​before and after the transmission are respectively detected, thereby reflecting the overlap of various abnormal traffic data, and then based on this, the line with abnormal traffic data is disabled or traced, so as to ensure the accuracy of the data in the transmission process, solve the technical problem of large repeatability of network abnormal traffic detection when the existing network anomaly detection method is used, in addition, according to the size of the abnormal data overlap evaluation value, it can be seen that the probability of occurrence of various abnormal traffic data on different transmission lines, thereby controlling the detection system to give priority to the detection of high-incidence abnormal traffic data, and improve the detection efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data anomaly detection, and in particular to a network traffic anomaly detection method based on Bayesian optimization. Background Art

[0002] At present, a Chinese patent with an existing patent application number of CN202310906687.4 discloses a new type of network traffic combination prediction model, including collecting original network traffic data and eliminating abnormal network traffic data; filling the missing network traffic data generated when eliminating abnormal network traffic data to obtain corrected network traffic data; performing sample normalization processing on the corrected network traffic data, and converting the old time series of the corrected network traffic data into a new time series; decomposing the new time series into PF components and residuals through LMD; optimizing the hyperparameters of the BiLSTM neural network prediction model; using the PF components and residuals as training data, and using the Bayesian optimization method to obtain the optimized hyperparameters of the BiLSTM neural network; using the Bayesian optimized BiLSTM neural network prediction model to predict the PF components and residuals, and respectively obtain the predicted values ​​of each component and residual; linearly superimposing all the predicted values ​​to obtain the final predicted value of the corrected network traffic data; although this method makes the network traffic prediction more accurate, in actual use, the data transmission path cannot be optimized.

[0003] However, during the implementation of the above technical solution, it was found that there were at least the following technical problems:

[0004] It is impossible to effectively control abnormal network traffic, and the detection of abnormal network traffic is highly repetitive: the existing network anomaly detection method mainly targets the data after transmission (the final acquired data), and then performs abnormal data analysis on the data, removes the abnormal traffic data in the acquired data, and then completes it. This method has serious lag. In addition, due to the single data transmission path and the inability to effectively avoid the path with abnormal data, abnormal data repeatedly appears in the acquired data. Therefore, when the abnormal data recognition and analysis algorithm is performing operations, it needs to spend a lot of time and energy on repetitive work, which seriously affects the data processing efficiency. For example, when abnormal data appears on a certain transmission path, all the original data passing through the transmission path will have a high probability of containing this abnormal data, resulting in the abnormal data recognition and analysis system needing to repeatedly delete the corresponding abnormal data. This method is not only easy to affect the root cause of abnormal data, but also cannot guarantee the accuracy of the original data. For this reason, we propose a network traffic anomaly detection method based on Bayesian optimization. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] In view of the shortcomings of the prior art, the present invention provides a network traffic anomaly detection method based on Bayesian optimization. By obtaining the data change situation before and after the original data transmission, the risks existing in the data transmission path are deduced to facilitate subsequent avoidance, thereby obtaining a data change estimate value that can intuitively evaluate the severity of the risk existing in the line. In addition, by combining the data change estimate value with the complete situation of data transmission, a risk level classification standard is obtained. The original data corresponding to the data change estimate value is divided into three gears of high abnormality, low abnormality, and non-abnormality by means of the classification standard. Then, the reliability of data transmission and the detection strength of the abnormality are determined according to the abnormal gear situation, thereby improving the pertinence of detection, providing a basis for solving the transmission efficiency of transmission, and solving the technical problems that the existing network anomaly detection method cannot effectively control the network abnormal traffic when in use, and the network abnormal traffic detection has high repetitiveness.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] A network traffic anomaly detection method based on Bayesian optimization, the detection method includes:

[0010] Set the standard threshold interval of abnormal network traffic characteristics, use NetFlow traffic analysis software to collect raw traffic data during network transmission, and then correct the abnormal traffic data in the raw traffic data according to the set standard threshold interval; based on the data changes of the raw traffic data before and after transmission, detect the degree of change of the raw traffic data when it is transmitted on different paths, where the network abnormal traffic characteristics include data flow, traffic transmission rate, traffic type and the integrity rate of the transmission traffic;

[0011] The transmission routes are classified based on the type corresponding to the network abnormal traffic feature with the largest change before and after the transmission of the original traffic data; the historical network abnormal traffic passing through the transmission line is retrieved from the database according to the classified type, and the overlap evaluation value between the historical network abnormal traffic is calculated, and then the historical network abnormal traffic is divided into heavy overlap data, moderate overlap data and light overlap data according to the positional relationship between the overlap evaluation value and the preset overlap threshold interval, and a disable instruction is issued for the heavy overlap data, and the same part of the original traffic data as the heavy overlap data is removed, and it is re-compared with the standard threshold interval of the network abnormal traffic feature; for the moderate overlap data, a tracing instruction is issued to obtain the node where the moderate overlap data appears; for the light overlap data, no response is given;

[0012] Retrieve the available transmission paths of the original traffic data, and generate corresponding credibility evaluation values ​​according to the coincidence evaluation values ​​between the historical network abnormal traffic on the corresponding transmission routes; sort the available transmission paths of the original traffic data according to the credibility evaluation values, discard the transmission paths below the preset credibility threshold, and record them as the credible transmission path set;

[0013] The trusted transmission path set is retrieved, and Bayesian optimization is used to combine the parameters of the LGBM model searched in the process of classifying abnormal network traffic, and the optimal path combination is output.

[0014] Preferably, the specific process of correcting abnormal flow data in the original flow data is as follows:

[0015] According to the types of each network abnormal traffic feature in the original traffic data, the corresponding standard threshold interval is set. When the network abnormal traffic feature is within the corresponding standard threshold interval, it is eliminated; when the network abnormal traffic feature is higher than the corresponding standard threshold interval, it is retransmitted; conversely, when the network abnormal traffic feature is lower than the corresponding standard threshold interval, it is not eliminated;

[0016] According to the removed abnormal data, the corresponding flow data segment is retrieved, and the flow data segment is added to the original flow data to obtain the completed original flow data.

[0017] Preferably, based on the inconsistency between the behavior pattern of abnormal network traffic and standard network traffic, the abnormal network traffic is divided into abnormal network traffic magnitude, abnormal network traffic distribution, abnormal network traffic transmission rate, abnormal network traffic content and abnormal network traffic source address;

[0018] The magnitude of abnormal network traffic is greater than the standard network traffic;

[0019] The network traffic distribution is abnormal, and it is distributed differently from the standard network traffic at different times, between different network protocols and between different communication ports;

[0020] The network traffic transmission rate is abnormal. The transmission rate of abnormal network traffic is usually higher than that of standard network traffic.

[0021] The network traffic content is abnormal, containing protocols, requests, responses or data content that are different from standard network traffic;

[0022] Abnormal network traffic source addresses, illegal or fake IP addresses, and Spoofing attacks.

[0023] Preferably, the process of analyzing the degree of data change before and after the transmission of the original traffic data is as follows:

[0024] Set the threshold of the integrity rate after data transmission;

[0025] Retrieve the transmission path of the original traffic data and record the normal data volume before the data passes through the transmission node, the normal data volume after the data passes through the transmission node, the number of errors, and the data change on the kth transmission route of the original traffic data, which are recorded as Wz, Wr, , ;

[0026] By analyzing the formula Calculate the integrity rate of the original traffic data after transmission ,when When the detection value is greater than or equal to the demarcation threshold, the data obtained from the detection is discarded and the transmission path is closed. Otherwise, no response is made.

[0027] Will Substitute into the analysis formula Calculate the estimated value of the data change after the original traffic data is transmitted , where Indicates the number of errors after the original traffic data is transmitted through the nth transmission route. Indicates the amount of change in the original traffic data after it is transmitted through the nth transmission route. Indicates the normal data volume of the original traffic data before it is transmitted through the nth transmission route. Indicates the maximum integrity rate of the original traffic data after passing through the transmission node. Indicates the minimum integrity rate of the original traffic data after passing through the transmission node. They represent the weights of the average error of the original traffic data, the limit difference of the completeness rate, and the change value before and after the transmission of the original traffic data;

[0028] Estimated value of data changes before and after the original traffic data is transmitted Compare with the standard range of transmission error change, and estimate the value based on the comparison and data change The corresponding original traffic data is classified.

[0029] Preferably, the estimated value of the data change before and after the original traffic data is transmitted The analysis process for comparison with the standard range of transmission error variation is as follows:

[0030] The estimated value of the data change before and after the calculated original traffic data transmission is , substitute Calculate the upper limit standard value of transmission error change And the transmission error change lower limit standard value , where and Constant correction coefficients representing the preset transmission error variation upper limit standard value and the transmission error variation lower limit standard value respectively;

[0031] According to the transmission error change upper limit standard value And the transmission error change lower limit standard value As a dividing line, when the data changes the estimated value ≤Lower limit standard value When the data changes, the estimated value The corresponding original flow data is recorded as high abnormal transmission data, the transmission line is replaced, and the original flow data is transmitted again; the lower limit standard value <Estimated data change <Upper limit standard value When the data changes, the estimated value The corresponding original traffic data is recorded as low abnormal transmission data, and the original traffic data is retransmitted; when the data change estimate ≥ Upper limit standard value , then the estimated value of the data change The corresponding original traffic data is recorded as non-abnormal transmission data and no response is given.

[0032] Preferably, the analysis process of the overlap evaluation value between abnormal network traffic is as follows:

[0033] According to whether the original traffic data is the same, the historical network abnormal traffic data passing through the same transmission line is divided into the same transmission data set and the different transmission data set;

[0034] Collect the flow transmission rate and the completeness rate of the same transmission data during the transmission process, which are recorded as Gv and Qx respectively. The completeness rate of the transmission flow Qx≠0, and randomly select two identical transmission data from the same transmission data set for combination, and select L times;

[0035] By analyzing the formula Calculate the overlap evaluation value between the same transmission data , where represents the overlap rate between the i-th group of identical transmission data, They represent the weights of the same transmission data transmission completeness ratio and traffic transmission rate, ,and ;

[0036] Collect the flow transmission rate and the completeness rate of the transmission flow of the differential transmission data during the transmission process, record them as Gt and Qt respectively, and randomly select two differential transmission data from the differential transmission data set for combination, and select D times;

[0037] By analyzing the formula Calculate the overlap estimate between differentially transmitted data , where represents the overlap rate between the mth group of differentially transmitted data, They represent the weights of the difference transmission data transmission completeness ratio and the flow transmission rate, ,and .

[0038] Preferably, and Substitute into the analysis formula Calculate the overlap evaluation value between historical network abnormal traffic , where They represent the overlap ratio, the overlap evaluation value of the same transmission data, and the overlap evaluation value of the different transmission data respectively;

[0039] Will overlap the evaluation value Compare with the preset overlap threshold interval, when the overlap evaluation value When the value is greater than the maximum value of the overlap threshold interval, the historical network abnormal traffic passing through the transmission line is recorded as severe overlap data and the disable instruction is executed; when the overlap evaluation value is When the value is less than the minimum value of the overlap threshold interval, the historical network abnormal traffic passing through the transmission line is recorded as slight overlap data, and no response is made; when the overlap assessment value is When it is within the overlap threshold range, the historical network abnormal traffic passing through the transmission line is recorded as moderate overlap data, and the tracing command is executed.

[0040] Preferably, when a traceability instruction is received, the traceability command is executed, and the specific analysis process is as follows:

[0041] Obtain the transmission path during the moderately overlapping data transmission process, which is recorded as a simulation path;

[0042] Input transmission simulation data into the obtained simulation path, and record the data at each node;

[0043] Close the node where moderate overlapping data first appears during the transmission simulation data transmission process, remove the node from the simulation path and replace it with a replaceable node, re-input the transmission simulation data into the simulation path until moderate overlapping data no longer appears, and mark the removed node as a moderately abnormal transmission node.

[0044] Preferably, the formula for generating the corresponding credibility evaluation value is as follows:

[0045]

[0046] In the formula, are the ratio coefficients of the historical network abnormal traffic overlap assessment value and the data change estimate value, and .

[0047] Preferably, the specific steps of using Bayesian optimization LGBM are as follows:

[0048] Determine the parameter space of the LGBM model to be optimized;

[0049] Combine the LGBM model with the parameters to be optimized, define an objective function as the optimization target, and select the average accuracy of 20 cross-validations as the objective function;

[0050] Select the Bayesian Optimization library and initialize it according to the defined parameter space and objective function;

[0051] In each iteration, the Bayesian optimizer samples according to the current parameter space and objective function to obtain a set of parameter values, uses this set of parameters to train the LGBM model, and evaluates the performance of the model on the training data. Based on the evaluation results, the Bayesian optimizer model is updated and the next set of parameters is selected for sampling. During the iteration process, the Bayesian optimizer continuously updates the model to find the optimal solution to the objective function.

[0052] When the predetermined number of iterations is reached, the Bayesian optimization process ends, and the optimal parameter combination is found from the parameter values ​​and the values ​​of the objective function recorded during the optimization process.

[0053] (III) Beneficial effects

[0054] 1. By obtaining the data changes before and after the original data transmission, the risks existing in the data transmission path are deduced to facilitate subsequent avoidance, thereby obtaining a data change estimate that can intuitively evaluate the severity of the risk existing in the line. In addition, by combining the data change estimate with the complete situation of data transmission, a risk level classification standard is obtained. The original data corresponding to the data change estimate is divided into three levels of high abnormality, low abnormality, and non-abnormality by this classification standard. Then, the reliability of data transmission and the intensity of abnormal detection are determined according to the abnormal gear situation, thereby improving the targeted detection and providing a basis for solving the transmission efficiency of the transmission.

[0055] 2. According to whether the original traffic data is the same, the historical network abnormal traffic data passing through the same transmission line is divided into the same transmission data set and the difference transmission data set. Then, the abnormal data overlap evaluation values ​​before and after the transmission are detected respectively, thereby reflecting the overlap of various abnormal traffic data. On this basis, the lines with abnormal traffic data are disabled or traced, so as to ensure the accuracy of the data during the transmission process, and at the same time reduce the deviation range before and after data transmission. In addition, according to the size of the abnormal data overlap evaluation value, the probability of occurrence of various abnormal traffic data on different transmission lines can be seen, so that the detection system can be controlled to give priority to the detection of high-incidence abnormal traffic data, thereby further improving the efficiency of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings.

[0057] Figure 1 This is an overall flow chart of the network traffic anomaly detection method based on Bayesian optimization of the present invention. DETAILED DESCRIPTION

[0058] The embodiment of the present application provides a network traffic anomaly detection method based on Bayesian optimization, which effectively solves the technical problems that the existing network anomaly detection method cannot effectively control network abnormal traffic when in use, and the network abnormal traffic detection has a large repeatability. According to whether the original traffic data is the same, the historical network abnormal traffic data passing through the same transmission line is divided into the same transmission data set and the difference transmission data set, and then the abnormal data overlap evaluation value before and after the transmission is detected respectively, thereby reflecting the overlap of various abnormal traffic data, and then based on this, the line with abnormal traffic data is disabled or traced, so as to ensure the accuracy of the data during the transmission process, and at the same time reduce the deviation range before and after the data transmission. In addition, according to the size of the abnormal data overlap evaluation value, it can be seen that the probability of occurrence of various abnormal traffic data on different transmission lines, thereby controlling the detection system to give priority to the detection of high-incidence abnormal traffic data, thereby further improving the efficiency of detection.

[0059] Due to the increasing amount of data transmission, the data that network anomaly detection methods need to process is also increasing. In order to improve the detection efficiency without changing the existing detection algorithm, a new detection method is proposed. By eliminating and simplifying the abnormal traffic data that has been repeatedly detected, the existing network detection needs can be met. Figure 1 This is a flowchart of the overall process of the network traffic anomaly detection method based on Bayesian optimization of the present invention. Figure 1 , the specific plan is as follows:

[0060] Raw data change detection

[0061] Set the standard threshold interval of abnormal network traffic characteristics, use NetFlow traffic analysis software to collect raw traffic data during network transmission, and then correct the abnormal traffic data in the raw traffic data according to the set standard threshold interval; based on the data changes of the raw traffic data before and after transmission, detect the degree of change of the raw traffic data when it is transmitted on different paths, where the network abnormal traffic characteristics include data flow, traffic transmission rate, traffic type, and the completeness rate of the transmission traffic. Among them, the correction of abnormal traffic data in the raw traffic data is divided into data elimination and data completion.

[0062] Abnormal network traffic can be distinguished from the following aspects:

[0063] (1) Traffic size: Attack traffic is usually larger than normal traffic because attackers usually send a large number of data packets to the target computer or try to access a large number of network resources;

[0064] (2) Traffic rate: Attack traffic is usually faster than normal traffic because the attacker will try to send as many requests as possible to the target computer in the shortest possible time. Normal traffic is usually stable and does not have instantaneous peaks. Therefore, the size and rate of attack traffic per unit time are usually higher than normal traffic;

[0065] (3) Traffic type: Attack traffic usually includes a large number of repeated requests or requests from different IP addresses, while normal traffic usually contains a mixture of multiple types of requests, such as web page requests, file transfers, and emails.

[0066] (4) Completeness rate: Compare the original data before and after transmission to ensure that the proportion of the same part is greater than 95%. Otherwise, the transmission is invalid and needs to be retransmitted.

[0067] Data elimination: Set the corresponding standard threshold interval according to the type of each abnormal network traffic feature in the original traffic data. When the abnormal network traffic feature is within the corresponding standard threshold interval, it will be eliminated; when the abnormal network traffic feature is higher than the corresponding standard threshold interval, it will be retransmitted; conversely, when the abnormal network traffic feature is lower than the corresponding standard threshold interval, it will not be eliminated.

[0068] Taking the traffic transmission rate on a certain transmission line as an example, the standard threshold interval of the traffic transmission rate is initially set to [GBmin, GBmax], where GBmin is the lower limit of the traffic transmission rate and GBmax is the lower limit of the traffic transmission rate. The traffic transmission rate on the transmission route is detected. When the value of the traffic transmission rate at a certain moment is greater than the upper limit GBmax (or less than the lower limit GBmin), it means that the data at this end is abnormal. The original data transmitted at this moment is recorded, and the obtained data is recorded as the abnormal traffic data corresponding to the traffic transmission rate. It is removed from the original data after transmission, and then the corresponding part of the data in the original data before transmission is re-filled.

[0069] Similarly, the data flow, flow type and completeness rate of the transmission flow during the data transmission process are set, so that all data that are not in the standard threshold range are eliminated, and all the obtained data are recorded as abnormal flow data.

[0070] Data completion: According to the eliminated abnormal data, the corresponding traffic data segment is retrieved, and the traffic data segment is added to the original traffic data to obtain the completed original traffic data.

[0071] Abnormal data overlap

[0072] The transmission routes are classified based on the types corresponding to the network abnormal traffic characteristics with the largest changes in the original traffic data before and after transmission; the historical network abnormal traffic passing through the transmission line is retrieved from the database according to the classified types, and the overlap evaluation value between the historical network abnormal traffic is calculated; then, according to the positional relationship between the overlap evaluation value and the preset overlap threshold interval, the historical network abnormal traffic is divided into heavy overlap data, moderate overlap data and light overlap data; a disable instruction is issued for the heavy overlap data, the same part of the original traffic data as the heavy overlap data is removed, and the data is re-compared with the standard threshold interval of the network abnormal traffic characteristics; a tracing instruction is issued for the moderate overlap data to obtain the node where the moderate overlap data appears; no response is given to the light overlap data.

[0073] Abnormal network traffic refers to network data traffic that does not conform to the behavior pattern of normal network traffic and may be malicious or offensive. Its characteristics are mainly reflected in the following aspects:

[0074] (1) Abnormal network traffic level: abnormal traffic is usually larger than normal traffic, which may be caused by attackers injecting large amounts of traffic or duplicating packets.

[0075] (2) Abnormal network traffic distribution: Abnormal traffic may have different distributions at different times, between different network protocols and between different communication ports, which is usually inconsistent with the distribution pattern of normal traffic;

[0076] (3) Abnormal network traffic transmission rate: The transmission rate of abnormal network traffic is usually much higher than that of normal traffic, because attackers may use a large number of data packets to attack in order to achieve the purpose of denial of service attack;

[0077] (4) Abnormal network traffic content: Abnormal network traffic may contain protocols, requests, responses, and data content that are different from normal traffic;

[0078] (5) Abnormal network traffic source address: The source address of abnormal traffic usually comes from illegal IP addresses, false IP addresses, spoofing attacks, etc. These source addresses may cause firewalls or other network security devices to be unable to identify and block attacks;

[0079] (6) Abnormal network traffic destination address: The destination address of abnormal network traffic is usually a vulnerable device or system in the network. Attackers may achieve their attack objectives by attacking these vulnerable targets.

[0080] The process of analyzing the degree of data change before and after the transmission of original traffic data is as follows:

[0081] Set the threshold of the integrity rate after data transmission;

[0082] Retrieve the transmission path of the original traffic data and record the normal data volume before the data passes through the transmission node, the normal data volume after the data passes through the transmission node, the number of errors, and the data change on the kth transmission route of the original traffic data, which are recorded as Wz, Wr, , ;

[0083] By analyzing the formula Calculate the integrity rate of the original traffic data after transmission ,when When the detection value is greater than or equal to the demarcation threshold, the data obtained from the detection is discarded and the transmission path is closed. Otherwise, no response is made.

[0084] Will Substitute into the analysis formula Calculate the estimated value of the data change after the original traffic data is transmitted , where Indicates the number of errors after the original traffic data is transmitted through the nth transmission route. Indicates the amount of change in the original traffic data after it is transmitted through the nth transmission route. Indicates the normal data volume of the original traffic data before it is transmitted through the nth transmission route. Indicates the maximum integrity rate of the original traffic data after passing through the transmission node. Indicates the minimum integrity rate of the original traffic data after passing through the transmission node. They represent the weights of the average error of the original traffic data, the limit difference of the completeness rate, and the change value before and after the transmission of the original traffic data;

[0085] The estimated value of the data change before and after the calculated original traffic data transmission is , substitute Calculate the upper limit standard value of transmission error change And the transmission error change lower limit standard value , where and Constant correction coefficients representing the preset transmission error variation upper limit standard value and the transmission error variation lower limit standard value respectively;

[0086] According to the transmission error change upper limit standard value And the transmission error change lower limit standard value As a dividing line, when the data changes the estimated value ≤Lower limit standard value When the data changes, the estimated value The corresponding original flow data is recorded as high abnormal transmission data, the transmission line is replaced, and the original flow data is transmitted again; the lower limit standard value <Estimated data change <Upper limit standard value When the data changes, the estimated value The corresponding original traffic data is recorded as low abnormal transmission data, and the original traffic data is retransmitted; when the data change estimate ≥ Upper limit standard value , then the estimated value of the data change The corresponding raw traffic data is recorded as non-abnormal transmission data and no response is made. By combining the estimated data change value with the completeness of data transmission, the risk level classification standard is obtained. The raw data corresponding to the estimated data change value is divided into three gears: high abnormality, low abnormality, and non-abnormal. Then, the reliability of data transmission and the strength of abnormal detection are determined according to the abnormal gear situation, thereby improving the pertinence of detection and providing a basis for solving the transmission efficiency of transmission.

[0087] Path screening

[0088] Retrieve the available transmission paths of the original traffic data, and generate the corresponding credibility evaluation value based on the overlap evaluation value between the historical network abnormal traffic on the corresponding transmission route. Sort the available transmission paths of the original traffic data according to the credibility evaluation value, discard the transmission paths below the preset credibility threshold, and record them as the credible transmission path set. The analysis process of the overlap evaluation value between network abnormal traffic is as follows:

[0089] According to whether the original traffic data is the same, the historical network abnormal traffic data passing through the same transmission line is divided into the same transmission data set and the different transmission data set;

[0090] Collect the flow transmission rate and the completeness rate of the same transmission data during the transmission process, which are recorded as Gv and Qx respectively. The completeness rate of the transmission flow Qx≠0, and randomly select two identical transmission data from the same transmission data set for combination, and select L times;

[0091] By analyzing the formula Calculate the overlap evaluation value between the same transmission data , where represents the overlap rate between the i-th group of identical transmission data, They represent the weights of the same transmission data transmission completeness ratio and traffic transmission rate, ,and ;

[0092] The flow transmission rate and the completeness rate of the transmission flow of the differential transmission data during the transmission process are collected, which are recorded as Gt and Qt respectively, and two differential transmission data are randomly selected from the differential transmission data set for combination, and the selection is performed D times;

[0093] By analyzing the formula Calculate the overlap estimate between differentially transmitted data , where represents the overlap rate between the mth group of differentially transmitted data, They represent the weights of the difference transmission data transmission completeness ratio and the flow transmission rate, ,and .

[0094] Will and Substitute into the analysis formula Calculate the overlap evaluation value between historical network abnormal traffic , where Respectively represent the overlap ratio, the overlap evaluation value of the same transmission data, and the overlap evaluation value of the different transmission data; Compare with the preset overlap threshold interval, when the overlap evaluation value When the value is greater than the maximum value of the overlap threshold interval, the historical network abnormal traffic passing through the transmission line is recorded as severe overlap data and the disable instruction is executed; when the overlap evaluation value is When the value is less than the minimum value of the overlap threshold interval, the historical network abnormal traffic passing through the transmission line is recorded as slight overlap data, and no response is made; when the overlap assessment value is When it is within the overlap threshold range, the historical network abnormal traffic passing through the transmission line is recorded as moderate overlap data, and the tracing command is executed. When the tracing command is received, the tracing command is executed. The specific analysis process is as follows:

[0095] Obtain the transmission path during the moderately overlapping data transmission process, which is recorded as a simulation path;

[0096] Input transmission simulation data into the obtained simulation path, and record the data at each node;

[0097] Close the node where moderate overlapping data first appears during the transmission simulation data transmission process, remove the node from the simulation path and replace it with a replaceable node, re-input the transmission simulation data into the simulation path until moderate overlapping data no longer appears, and mark the removed node as a moderately abnormal transmission node.

[0098] The available transmission paths of the original traffic data are retrieved, and the corresponding credibility evaluation values ​​are generated based on the overlap evaluation values ​​between the historical network abnormal traffic on the corresponding transmission routes. The formula for generating the corresponding credibility evaluation values ​​is as follows:

[0099]

[0100] In the formula, are the ratio coefficients of the historical network abnormal traffic overlap assessment value and the data change estimate value, and

[0101] Trusted transmission channel construction

[0102] Retrieve the trusted transmission path set, use Bayesian optimization to combine the parameters of the LGBM model in the process of classifying abnormal network traffic, and output the optimal path combination. The specific steps of using Bayesian optimization LGBM are as follows:

[0103] Determine the parameter space of the LGBM model to be optimized;

[0104] Combine the LGBM model with the parameters to be optimized, define an objective function as the optimization target, and select the average accuracy of 20 cross-validations as the objective function;

[0105] Select the Bayesian Optimization library and initialize it according to the defined parameter space and objective function;

[0106] In each iteration, the Bayesian optimizer samples according to the current parameter space and objective function to obtain a set of parameter values, uses this set of parameters to train the LGBM model, and evaluates the performance of the model on the training data. Based on the evaluation results, the Bayesian optimizer model is updated and the next set of parameters is selected for sampling. During the iteration process, the Bayesian optimizer continuously updates the model to find the optimal solution to the objective function.

[0107] When the predetermined number of iterations is reached, the Bayesian optimization process ends, and the optimal parameter combination is found from the parameter values ​​and the values ​​of the objective function recorded during the optimization process.

[0108] Finally, it should be noted that: Obviously, the above embodiments are only examples for clearly explaining the present invention, and are not intended to limit the implementation methods. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from this are still within the scope of protection of the present invention.

Claims

1. A network traffic anomaly detection method based on Bayesian optimization, characterized in that: The test method includes: Set the standard threshold interval of abnormal network traffic characteristics, use NetFlow traffic analysis software to collect raw traffic data during network transmission, and then correct the abnormal traffic data in the raw traffic data according to the set standard threshold interval; based on the data changes of the raw traffic data before and after transmission, detect the degree of change of the raw traffic data when it is transmitted on different paths. Among them, the network abnormal traffic characteristics include data flow, traffic transmission rate, traffic type and transmission traffic integrity rate. The specific process of correcting abnormal traffic data in the raw traffic data is as follows: According to the types of each network abnormal traffic feature in the original traffic data, the corresponding standard threshold interval is set. When the network abnormal traffic feature is within the corresponding standard threshold interval, it is eliminated; when the network abnormal traffic feature is higher than the corresponding standard threshold interval, it is retransmitted; conversely, when the network abnormal traffic feature is lower than the corresponding standard threshold interval, it is not eliminated; According to the removed abnormal data, the corresponding flow data segment is retrieved, and the flow data segment is added to the original flow data to obtain the completed original flow data; The process of analyzing the degree of data change before and after the transmission of original traffic data is as follows: Set the threshold of the integrity rate after data transmission; Retrieve the transmission path of the original traffic data and record the normal data volume before the data passes through the transmission node, the normal data volume after the data passes through the transmission node, the number of errors, and the data change on the kth transmission route of the original traffic data, which are recorded as Wz, Wr, , ; By analyzing the formula Calculate the integrity rate of the original traffic data after transmission ,when When the detection value is greater than or equal to the demarcation threshold, the data obtained from the detection is discarded and the transmission path is closed. Otherwise, no response is made. Will Substitute into the analysis formula Calculate the estimated value of the data change after the original traffic data is transmitted , where Indicates the number of errors after the original traffic data is transmitted through the nth transmission route. Indicates the amount of change in the original traffic data after it is transmitted through the nth transmission route. Indicates the normal data volume of the original traffic data before it is transmitted through the nth transmission route. Indicates the maximum integrity rate of the original traffic data after passing through the transmission node. Indicates the minimum integrity rate of the original traffic data after passing through the transmission node. They represent the weights of the average error of the original traffic data, the limit difference of the completeness rate, and the change value before and after the transmission of the original traffic data; Estimated value of data changes before and after the original traffic data is transmitted Compare with the standard range of transmission error change, and estimate the value based on the comparison and data change The corresponding original traffic data is classified; According to the classification type, the historical network abnormal traffic passing through the transmission line is retrieved from the database, and the overlap evaluation value between the historical network abnormal traffic is calculated. Then, according to the positional relationship between the overlap evaluation value and the preset overlap threshold interval, the historical network abnormal traffic is divided into severe overlap data, moderate overlap data and mild overlap data. For severe overlap data, a disable instruction is issued, and the same part of the original traffic data as the severe overlap data is removed, and the data is re-compared with the standard threshold interval of the network abnormal traffic characteristics; for moderate overlap data, a tracing instruction is issued to obtain the node where the moderate overlap data appears; for mild overlap data, no response is given. The analysis process of the overlap evaluation value between network abnormal traffic is as follows: According to whether the original traffic data is the same, the historical network abnormal traffic data passing through the same transmission line is divided into the same transmission data set and the different transmission data set; Collect the flow transmission rate and the completeness rate of the same transmission data during the transmission process, which are recorded as Gv and Qx respectively. The completeness rate of the transmission flow Qx≠0, and randomly select two identical transmission data from the same transmission data set for combination, and select L times; By analyzing the formula Calculate the overlap evaluation value between the same transmission data , where represents the overlap rate between the i-th group of identical transmission data, They represent the weights of the same transmission data transmission completeness ratio and traffic transmission rate, ,and ; The flow transmission rate and the completeness rate of the transmission flow of the differential transmission data during the transmission process are collected, which are recorded as Gt and Qt respectively, and two differential transmission data are randomly selected from the differential transmission data set for combination, and the selection is performed D times; By analyzing the formula Calculate the overlap estimate between differentially transmitted data , where represents the overlap rate between the mth group of differentially transmitted data, They represent the weights of the difference transmission data transmission completeness ratio and the flow transmission rate, ,and ; Will and Substitute into the analysis formula Calculate the overlap evaluation value between historical network abnormal traffic , where They represent the overlap ratio, the overlap evaluation value of the same transmission data, and the overlap evaluation value of the different transmission data respectively; Retrieve the available transmission paths of the original traffic data, and generate the corresponding credibility evaluation value based on the overlap evaluation value between the historical network abnormal traffic on the corresponding transmission route; sort the available transmission paths of the original traffic data according to the credibility evaluation value, discard the transmission paths below the preset credibility threshold, and record the undiscarded parts as the credible transmission path set. The formula for generating the corresponding credibility evaluation value is as follows: , In the formula, are the ratio coefficients of the historical network abnormal traffic overlap assessment value and the data change estimate value, and ; The trusted transmission path set is retrieved, and Bayesian optimization is used to combine the parameters of the LGBM model searched in the process of classifying abnormal network traffic, and the optimal path combination is output.

2. The network traffic anomaly detection method based on Bayesian optimization according to claim 1, characterized in that: Based on the fact that the behavior pattern of abnormal network traffic does not match that of standard network traffic, abnormal network traffic is classified into abnormal network traffic magnitude, abnormal network traffic distribution, abnormal network traffic transmission rate, abnormal network traffic content, and abnormal network traffic source address; The magnitude of abnormal network traffic is greater than the standard network traffic; The network traffic distribution is abnormal, and it is distributed differently from the standard network traffic at different times, between different network protocols and between different communication ports; The network traffic transmission rate is abnormal. The transmission rate of abnormal network traffic is usually higher than that of standard network traffic. The network traffic content is abnormal, containing protocols, requests, responses or data content that are different from standard network traffic; Abnormal network traffic source addresses, illegal or fake IP addresses, and Spoofing attacks.

3. The network traffic anomaly detection method based on Bayesian optimization according to claim 1, characterized in that: Estimated value of data change before and after the transmission of original traffic data The analysis process for comparison with the standard range of transmission error variation is as follows: The estimated value of the data change before and after the calculated original traffic data transmission is , substitute Calculate the upper limit standard value of transmission error change And the transmission error change lower limit standard value , where and Constant correction coefficients representing the preset transmission error variation upper limit standard value and the transmission error variation lower limit standard value respectively; According to the transmission error change upper limit standard value And the transmission error change lower limit standard value As a dividing line, when the data changes the estimated value ≤Lower limit standard value When the data changes, the estimated value The corresponding original flow data is recorded as high abnormal transmission data, the transmission line is replaced, and the original flow data is transmitted again; the lower limit standard value <Estimated data change <Upper limit standard value When the data changes, the estimated value The corresponding original traffic data is recorded as low abnormal transmission data, and the original traffic data is retransmitted; when the data change estimate ≥ Upper limit standard value , then the estimated value of the data change The corresponding original traffic data is recorded as non-abnormal transmission data and no response is given.

4. The network traffic anomaly detection method based on Bayesian optimization according to claim 1, characterized in that: Will overlap the evaluation value Compare with the preset overlap threshold interval, when the overlap evaluation value When the value is greater than the maximum value of the overlap threshold interval, the historical network abnormal traffic passing through the transmission line is recorded as severe overlap data and the disable instruction is executed; when the overlap evaluation value is When the value is less than the minimum value of the overlap threshold interval, the historical network abnormal traffic passing through the transmission line is recorded as slight overlap data, and no response is made; when the overlap assessment value is When it is within the overlap threshold range, the historical network abnormal traffic passing through the transmission line is recorded as moderate overlap data, and the tracing command is executed.

5. The network traffic anomaly detection method based on Bayesian optimization as claimed in claim 4, characterized in that: When receiving the traceability instruction, execute the traceability command. The specific analysis process is as follows: Obtain the transmission path during the moderately overlapping data transmission process, which is recorded as a simulation path; Input transmission simulation data into the obtained simulation path, and record the data at each node; Close the node where moderate overlapping data first appears during the transmission simulation data transmission process, remove the node from the simulation path and replace it with a replaceable node, re-input the transmission simulation data into the simulation path until moderate overlapping data no longer appears, and mark the removed node as a moderately abnormal transmission node.

6. The network traffic anomaly detection method based on Bayesian optimization according to claim 1, characterized in that: The specific steps of using Bayesian optimization LGBM are as follows: Determine the parameter space of the LGBM model to be optimized; Combine the LGBM model with the parameters to be optimized, define an objective function as the optimization target, and select the average accuracy of 20 cross-validations as the objective function; Select the Bayesian optimization library and initialize it according to the defined parameter space and objective function; In each iteration, the Bayesian optimizer samples according to the current parameter space and objective function to obtain a set of parameter values, uses this set of parameters to train the LGBM model, and evaluates the performance of the model on the training data. Based on the evaluation results, the Bayesian optimizer model is updated and the next set of parameters is selected for sampling. During the iteration process, the Bayesian optimizer continuously updates the model to find the optimal solution of the objective function; When the predetermined number of iterations is reached, the Bayesian optimization process ends, and the optimal parameter combination is found from the parameter values ​​and the values ​​of the objective function recorded during the optimization process.

Images