A large-scale encrypted traffic frame clustering analysis method based on a big data architecture

Abstract

The application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, relates to the technical field of big data, and comprises target encrypted data data partitioning and storage obtained by preprocessing original encrypted traffic data; target clustering results obtained by frame clustering analysis on the target encrypted data are visualized to obtain clustering visualized results to determine whether there is data traffic abnormal behavior, and when there is, an abnormal analysis report is generated; the abnormal analysis report is transmitted to a security management personnel to take corresponding measures for defense. Through frame clustering analysis on the target clustering results obtained by processing and partitioning storage of encrypted traffic data based on the big data architecture, traffic data abnormalities are visualized and identified, and when there are abnormalities, an abnormal analysis report is generated and transmitted to a security management personnel to take corresponding measures for defense, so that the network threat identification capability is effectively enhanced, and the overall protection level of network security is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a large-scale encrypted traffic frame clustering analysis method based on a big data architecture. BACKGROUND

[0002] In recent years, with the development of the Internet and the Internet of Things, the amount of data has increased explosively, the popularity of HTTPS and the use of various encryption protocols have caused most of the traffic in the network to become encrypted. However, the introduction of encrypted traffic has significantly improved the security and privacy of network communication, but also increased the complexity of network security defense, and posed a challenge to traditional traffic monitoring and anomaly analysis. Therefore, how to efficiently and securely process and analyze massive encrypted data to support network security monitoring and reduce threat detection difficulty has become one of the current research focuses.

[0003] Therefore, the present application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture. SUMMARY

[0004] The present application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, which can visualize the target clustering result obtained by frame clustering analysis on the encrypted traffic data processed and stored by the big data architecture, identify traffic data anomalies, and generate an abnormal analysis report when there is an anomaly to transmit to a security management personnel for taking corresponding measures for defense, thereby effectively enhancing the network threat identification capability and further improving the overall protection level of network security.

[0005] The present application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, which includes:

[0006] Step 1: data partitioning and storage of target encrypted data obtained by obtaining and preprocessing original encrypted traffic data;

[0007] Step 2: obtaining a target clustering result by frame clustering analysis on the target encrypted data, and visualizing the target clustering result to obtain a clustering visualization result;

[0008] Step 3: determining whether there is traffic data anomaly based on the clustering visualization result, and generating an abnormal analysis report when there is an anomaly;

[0009] Step 4: transmitting the abnormal analysis report to a security management personnel, which is conducive to taking corresponding measures for defense in a timely manner.

[0010] Preferably, the data partitioning and storage of target encrypted data obtained by obtaining and preprocessing original encrypted traffic data includes:

[0011] Real-time capture of original encrypted traffic data by using streaming data transmission technology, and transmission to a target big data platform;

[0012] The target big data platform uses a preset data cleaning tool to preprocess the received original encrypted traffic data to obtain first encrypted data;

[0013] Based on the current business requirements, determine the corresponding matching target data partition strategy;

[0014] According to the target data partition strategy, the first encrypted data is divided into target encrypted data and stored in a preset storage system.

[0015] Preferably, the target encrypted data is subjected to frame clustering analysis to obtain a target clustering result, and the target clustering result is visualized to obtain a clustering visualization result, including:

[0016] The target encrypted data of each partition is segmented by frame to obtain first frame data;

[0017] Extract the feature information of the first frame data to obtain key frame data features;

[0018] According to the current partition business requirements, match the corresponding first clustering algorithm list;

[0019] If there is only one first clustering algorithm in the first clustering algorithm list, then use the current first clustering algorithm to perform clustering analysis on the first frame data based on the key frame data features to obtain the target clustering result;

[0020] If there are multiple first clustering algorithms in the first clustering algorithm list, then use each first clustering algorithm to perform a preset number of clustering analyses on the first frame data based on the key frame data features, and collect the clustering labels of the frame data points to obtain a pending clustering result;

[0021] Based on the pending clustering result, perform data clustering analysis and effect evaluation to generate a target clustering result;

[0022] Use a preset interactive tool to visualize the target clustering result to obtain a clustering visualization result.

[0023] Preferably, the key frame data features include source IP address, target IP address, port, protocol type, timestamp, TLS / SSL handshake information, and traffic features.

[0024] Preferably, based on the pending clustering result, perform clustering analysis and effect evaluation to generate a target clustering result, including:

[0025] According to the pending clustering result, obtain the frequency of each frame data point being assigned to each cluster;

[0026] assigning frame data points to the cluster with the highest frequency to generate a first clustering result;

[0027] evaluating the first clustering result to obtain a first evaluation coefficient;

[0028] if the first evaluation coefficient is greater than a set evaluation threshold, the corresponding first clustering result is marked as a first-level clustering result;

[0029] otherwise, the corresponding first clustering result is marked as a second-level clustering result;

[0030] obtaining and determining a comprehensive clustering evaluation coefficient based on the proportion of the first-level clustering result and the second-level clustering result;

[0031] when the comprehensive clustering evaluation coefficient is greater than a set comprehensive evaluation threshold, generating a target clustering result based on the current first clustering result and outputting the target clustering result;

[0032] when the comprehensive clustering evaluation coefficient is not greater than the set comprehensive evaluation threshold, performing preset times of re-clustering analysis on the current first frame data using the first clustering algorithm to generate a new clustering result;

[0033] performing weighted average on all new clustering results to create a first comprehensive attribution matrix;

[0034] obtaining a first attribution value of each frame data point in each cluster according to the first comprehensive attribution matrix;

[0035] performing deviation analysis on the first attribution value to obtain a first deviation coefficient;

[0036] correcting the first attribution value using the first deviation coefficient to obtain a target attribution value of each frame data point in different clusters;

[0037] regarding the cluster with the highest target attribution value as the attribution cluster of the current frame data point;

[0038] collecting the cluster attribution decisions of all frame data points to generate a target clustering result.

[0039] Preferably, the calculation formula of the first deviation coefficient is as follows:

[0040] In the formula, P represents the first deviation coefficient; represents the average value of the first attribution value of the current frame data point in each cluster; represents the first attribution value of the current frame data point in the current cluster; represents the initial weight value of the i-th selectable clustering algorithm forming the current cluster; represents the attribution influence weight; a cluster data distribution evaluation coefficient of the current cluster; a cluster data distribution influence weight; a total value of cluster effect evaluation values of optional clustering algorithms forming the current cluster; a cluster effect evaluation value of an i-th optional clustering algorithm forming the current cluster; a cluster effect evaluation threshold value; a j-th cluster characteristic of the current cluster; an adjustment influence coefficient of the j-th cluster characteristic on the cluster effect evaluation value of the i-th optional clustering algorithm forming the current cluster.

[0041] Preferably, whether there is abnormal traffic data is judged based on the clustering visualization result, and when there is, an abnormality analysis report is generated, including:

[0042] According to the clustering visualization result, whether there is frame data point not belonging to any cluster is analyzed, and if there is, the corresponding frame data point is marked as possible abnormal data;

[0043] The first distance of each frame data point in each cluster in the target clustering result to the center of the cluster to which the frame data point belongs is extracted;

[0044] When the first distance is greater than a set distance, the corresponding frame data point is marked as possible abnormal data;

[0045] The possible abnormal data is subjected to abnormality detection, and when the detection result is abnormal, the current possible abnormal data is determined as abnormal data;

[0046] Based on all the abnormal data, an abnormality analysis report is generated.

[0047] Preferably, the possible abnormal data is subjected to abnormality detection, and when the detection result is abnormal, the current possible abnormal data is determined as abnormal data, including:

[0048] A preset amount of historical encrypted traffic data is extracted from the traffic database;

[0049] A corresponding historical traffic clustering result of the historical encrypted traffic data is extracted, and each cluster in the historical traffic clustering result is respectively added with a normal and abnormal label;

[0050] The historical traffic clustering result is added as a feature to the historical encrypted traffic data, and a training data set is constructed;

[0051] The machine learning model is trained using the training data set, and an abnormality detection model is generated;

[0052] The possible abnormal data is input into the abnormality detection model, and an abnormality detection result is obtained;

[0053] According to the abnormality detection result, if the detection result of the current possible abnormal data is abnormal, the abnormal data is determined.

[0054] Compared with the prior art, the application has the following beneficial effects:

[0055] The target clustering result obtained by frame clustering analysis on the encrypted traffic data processed and partitioned based on the big data architecture is visualized, traffic data anomalies are identified, and when there are anomalies, an abnormality analysis report is generated and transmitted to the security management personnel to take corresponding measures for defense, thereby effectively enhancing the network threat identification capability and further improving the overall protection level of network security.

[0056] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent to those skilled in the art from the description, or can be learned by practice of the present application. The objects and other advantages of the present application will be realized and achieved by means of the structures particularly pointed out in the written description and the appended drawings.

[0057] The technical solutions of the present application will be further described in detail below with the help of the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0058] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0059] Figure 1 A flowchart of a large-scale encrypted traffic frame clustering analysis method based on a big data architecture in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The preferred embodiments of the present application will be described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and do not limit the present application.

[0061] The embodiment of the present application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, as shown in Figure 1 The method comprises the following steps:

[0062] Step 1: data partitioning and storage are performed on the target encrypted data obtained by acquiring and pre-processing the original encrypted traffic data;

[0063] Step 2: frame clustering analysis is performed on the target encrypted data to obtain a target clustering result, and the target clustering result is visualized to obtain a clustering visualization result;

[0064] Step 3: judging whether there is abnormal traffic data by visualizing the clustering result, and generating an abnormal analysis report when there is;

[0065] Step 4: transmitting the abnormal analysis report to the security management personnel, which is conducive to taking corresponding measures for defense in time.

[0066] In this embodiment, the encrypted traffic data refers to data packets protected and transmitted by various encryption algorithms and technologies; the target encrypted data is data obtained by preprocessing the original encrypted traffic data using a preset data cleaning tool, and then divided and stored according to a target data partitioning strategy, wherein the preset data cleaning tool is set in advance; the preprocessing includes data duplicate value processing, missing value completion, denoising processing and the like.

[0067] In this embodiment, the frame clustering analysis refers to clustering the encrypted traffic data according to its frame structure or characteristics, which usually involves extracting and analyzing the frame header, frame length, timestamp and other information of the network encrypted traffic data packet; the target clustering result is the final result obtained by the process of dividing the data according to the frame clustering analysis of the target encrypted data.

[0068] In this embodiment, the clustering visualization result refers to a way of using visualization technology to display the distribution of clusters and the relationship between data points in the target clustering result, such as scatter plot, heat map, tree chart and the like; the abnormal analysis report is a report generated based on the abnormal encrypted traffic data, which is used to lay a data support for effectively analyzing the clustering result and identifying abnormal behavior patterns in the data, and then taking corresponding measures for processing or optimization.

[0069] The beneficial effects of the above technical solution are: by visualizing the target clustering result obtained by frame clustering analysis of the encrypted traffic data processed and partitioned stored based on the big data architecture, identifying the abnormal traffic data, and generating an abnormal analysis report when there is an abnormality and transmitting it to the security management personnel to take corresponding measures for defense, the network threat recognition capability is effectively enhanced, and the overall protection level of network security is improved.

[0070] The embodiment of the application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, which obtains and preprocesses target encrypted data obtained from original encrypted traffic data, and performs data partitioning and storage, comprising:

[0071] Real-time capture of original encrypted traffic data is realized by using stream data transmission technology, and the original encrypted traffic data is transmitted to a target big data platform;

[0072] The target big data platform preprocesses the received original encrypted traffic data using a preset data cleaning tool to obtain first encrypted data;

[0073] Determine a corresponding matching target data partition strategy based on current business requirements;

[0074] According to the target data partition strategy, the first encrypted data is divided into target encrypted data and stored in a preset storage system.

[0075] In this embodiment, the encrypted traffic data refers to data packets protected and transmitted using various encryption algorithms and technologies; the first encrypted data is obtained by preprocessing the original encrypted traffic data using a preset data cleaning tool, wherein the preset data cleaning tool is set in advance; the preprocessing includes data duplicate value processing, missing value completion, and denoising processing, etc.; the big data platform is used for processing, storing, and analyzing large-scale data.

[0076] In this embodiment, the business requirements include real-time performance, data volume, access mode, and security compliance requirements; the target data partition strategy is a data partition strategy selected based on the business requirements, wherein the data partition strategy includes time sequence partition, protocol type partition, and traffic feature partition, etc.; the target encrypted data is encrypted data obtained by dividing the first encrypted traffic data according to the target data partition strategy; the preset storage system is used for storing massive network encrypted traffic data, such as Hadoop HDFS and Spark.

[0077] The above technical solution has the beneficial effects that: by capturing, processing the original encrypted traffic data based on the big data structure, and dividing and partitioning the encrypted traffic data using a reasonable data partition storage strategy, the efficiency and response speed of large-scale encrypted traffic analysis can be improved, and the compliance and security of data processing can be ensured, thereby meeting diversified business requirements.

[0078] The embodiment of the present application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, which performs frame clustering analysis on target encrypted data to obtain target clustering results, and visualizes the target clustering results to obtain clustering visualization results, including:

[0079] Segmenting the target encrypted data of each partition by frame to obtain first frame data;

[0080] Extracting feature information of the first frame data to obtain key frame data features;

[0081] Matching a corresponding first clustering algorithm list according to the current partition business requirements;

[0082] If there is only one first clustering algorithm in the first clustering algorithm list, then the first frame data is clustered and analyzed based on the key frame data features using the current first clustering algorithm to obtain target clustering results;

[0083] If there are multiple first clustering algorithms in the first clustering algorithm list, each first clustering algorithm is used to perform clustering analysis on the first frame data based on the key frame data features for a preset number of times, and the clustering labels of the frame data points are collected to obtain a to-be-determined clustering result;

[0084] Based on the to-be-determined clustering result, data clustering analysis and effect evaluation are performed to generate a target clustering result;

[0085] The target clustering result is visualized by using a pre-set interactive tool to obtain a clustering visualization result.

[0086] In this embodiment, the first frame data is frame data obtained by dividing the target encrypted data by frame; the frame clustering analysis refers to clustering encrypted traffic data according to its frame structure or characteristics, which usually involves extracting and analyzing the frame header, frame length, timestamp and other information of the network encrypted traffic packet; the key frame data features include source IP address, target IP address, port, protocol type, timestamp, TLS / SSL handshake information and traffic characteristics.

[0087] In this embodiment, the first clustering algorithm list refers to a list established by clustering algorithms matched with business requirements, wherein the clustering algorithms include K-means algorithm, hierarchical clustering, density-based clustering algorithm, spectral clustering and the like; the target clustering result refers to the final result obtained in the process of dividing data by frame clustering analysis of the target encrypted data.

[0088] In this embodiment, the preset number is determined in advance; the clustering label includes feature description, time label, sequence label, normal and abnormal label and the like; the to-be-determined clustering result is obtained by performing clustering analysis on the corresponding first frame data of multiple first clustering algorithms in the first clustering algorithm list for a preset number of times; the pre-set interactive tool is set in advance, such as MATLAB clustering toolbox, QlikView and the like.

[0089] The above technical solution has the beneficial effects that: the target clustering result obtained by frame clustering analysis of the target encrypted data can provide data support for understanding the potential target behavior, feature or category in the encrypted data; the visualization of the target clustering result can provide an intuitive way for quickly understanding the patterns and abnormalities in complex data sets, improving the efficiency and depth of data analysis.

[0090] The embodiment of the application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, which performs clustering analysis and effect evaluation based on the to-be-determined clustering result to generate a target clustering result, comprising:

[0091] According to the to-be-determined clustering result, the frequency of each frame data point being assigned to each cluster is obtained;

[0092] assigning frame data points to the cluster with the highest frequency to generate a first clustering result;

[0093] evaluating the first clustering result to obtain a first evaluation coefficient;

[0094] if the first evaluation coefficient is greater than a set evaluation threshold, the first clustering result is marked as a first-level clustering result;

[0095] otherwise, the first clustering result is marked as a second-level clustering result;

[0096] obtaining and determining a comprehensive clustering evaluation coefficient based on the proportion of the first-level clustering result and the second-level clustering result;

[0097] when the comprehensive clustering evaluation coefficient is greater than a set comprehensive evaluation threshold, generating a target clustering result based on the current first clustering result and outputting the target clustering result;

[0098] when the comprehensive clustering evaluation coefficient is not greater than the set comprehensive evaluation threshold, performing preset times of re-clustering analysis on the current first frame data using the first clustering algorithm to generate a new clustering result;

[0099] performing weighted averaging on all new clustering results to create a first comprehensive attribution matrix;

[0100] obtaining a first attribution value of each frame data point in each cluster according to the first comprehensive attribution matrix;

[0101] performing deviation analysis on the first attribution value to obtain a first deviation coefficient;

[0102] correcting the first attribution value using the first deviation coefficient to obtain a target attribution value of each frame data point in different clusters;

[0103] regarding the cluster with the highest target attribution value as the attribution cluster of the current frame data point;

[0104] collecting the cluster attribution decisions of all frame data points to generate a target clustering result.

[0105] In this embodiment, the first evaluation coefficient is used to evaluate the clustering effect of the first clustering result, the set evaluation threshold is set in advance, the comprehensive clustering evaluation coefficient is used to represent the clustering effect of the first frame data, and the set comprehensive evaluation threshold is set in advance.

[0106] In this embodiment, for example, there are first evaluation coefficients l1, l2, and l3 of first clustering results a1, a2, and a3, wherein l1 and l2 are greater than the set evaluation threshold, and l3 is equal to the set evaluation threshold. At this time, the first clustering results a1 and a2 are marked as first-level clustering results, and the first clustering result a3 is marked as a second-level clustering result.

[0107] In this embodiment, the preset number of times is set in advance; and the new clustering result is obtained by re-clustering analysis of the current first frame data by using the first clustering algorithm for the preset number of times when the comprehensive clustering evaluation coefficient is not greater than the set comprehensive evaluation threshold.

[0108] In this embodiment, the first comprehensive attribution matrix is a matrix constructed by combining and weightedly averaging the weight allocated to each first clustering algorithm and the new clustering result generated by re-clustering analysis of the first frame data by using each first clustering algorithm for the preset number of times; the first attribution value refers to the highest value of the frame data point in each cluster in the first comprehensive attribution matrix; the first deviation coefficient is used for correcting the first attribution value; the target attribution value is a value obtained by correcting the first attribution value by using the first deviation coefficient; the attribution cluster refers to the cluster with the highest target attribution value of the frame data point; and the target clustering result is obtained by collecting the cluster attribution decisions of all frame data points, wherein the cluster attribution decision refers to the process of deciding the attribution cluster of each frame data point.

[0109] The above technical solution has the beneficial effects that: the new clustering result obtained by re-clustering analysis of the current first frame data by using different clustering algorithms for the preset number of times is combined and analyzed, and the clustering effect evaluation is performed, so that the target clustering result with the best clustering effect is generated, and the comprehensiveness and accuracy of the clustering analysis are enhanced.

[0110] The clustering result obtained by the clustering analysis is combined and analyzed, and the clustering effect evaluation is performed, so that the target clustering result with the best clustering effect is generated, and the comprehensiveness and accuracy of the clustering analysis are enhanced.

[0111] The embodiment of the present application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, and a calculation formula of the first deviation coefficient is as follows:

[0112] In the formula, P represents the first deviation coefficient; represents the average value of the first attribution value of the current frame data point in each cluster; represents the first attribution value of the current frame data point in the current cluster; represents the initial weight value of the i-th selectable clustering algorithm for forming the current cluster; represents the attribution influence weight; represents the cluster data distribution evaluation coefficient of the current cluster; represents the cluster data distribution influence weight; represents the total value of the clustering effect evaluation values of the selectable clustering algorithms for forming the current cluster; represents the clustering effect evaluation value of the i-th selectable clustering algorithm for forming the current cluster; represents the set clustering effect evaluation threshold; represents the j-th cluster characteristic of the current cluster; represents the adjustment influence coefficient of the j-th cluster characteristic on the clustering effect evaluation value of the i-th selectable clustering algorithm for forming the current cluster.

[0113] The beneficial effects of the above technical solutions are: by correcting the first attribution value by calculating the first deviation coefficient, the most accurate attribution cluster can be determined for each frame data point to generate the target clustering result with the best clustering effect, and the accurate reflection of the real distribution and internal structure of the data is realized, that is, the quality and efficiency of data analysis are improved.

[0114] The embodiment of the application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, which judges whether there is abnormal traffic data based on the clustering visualization result, and generates an abnormal analysis report when there is, comprising:

[0115] According to the clustering visualization result, it is analyzed whether there is frame data point not attributed to any cluster, and if there is, the corresponding frame data point is marked as possible abnormal data;

[0116] Extracting the first distance of each frame data point in each cluster in the target clustering result to the cluster center;

[0117] When the first distance is greater than the set distance, the corresponding frame data point is marked as possible abnormal data;

[0118] Abnormal detection is performed on the possible abnormal data, and when the detection result is abnormal, the current possible abnormal data is determined as abnormal data;

[0119] Based on all the abnormal data, an abnormal analysis report is generated.

[0120] In this embodiment, the clustering visualization result refers to a way of using visualization technology to display the distribution of clusters in the target clustering result and the relationship between data points, such as scatter plot, heat map, tree chart, etc.; the possible abnormal data refers to the frame data point not attributed to any cluster and the frame data point with the first distance to the cluster center greater than the set distance; the first distance refers to the distance from the frame data point to the cluster center; and the set distance is set in advance.

[0121] In this embodiment, the abnormal analysis report is a report generated based on abnormal encrypted traffic data, which is used to lay a data foundation for effectively analyzing the clustering result and identifying abnormal behavior patterns in the data, and then take corresponding measures for processing or optimization.

[0122] The beneficial effects of the above technical solutions are: by judging whether there is abnormal data traffic behavior based on the clustering visualization result, and generating an abnormal analysis report when there is, the network threat recognition capability can be effectively enhanced, and the overall protection level of network security can be improved.

[0123] The embodiment of the present application provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, performs abnormality detection on the possible abnormal data, and judges the current possible abnormal data as abnormal data when the detection result is abnormal, comprising:

[0124] extracting a preset amount of historical encrypted traffic data from a traffic database;

[0125] extracting a corresponding historical traffic clustering result of the historical encrypted traffic data, and adding normal and abnormal labels to each cluster in the historical traffic clustering result respectively;

[0126] adding the historical traffic clustering result as a feature to the historical encrypted traffic data to construct a training data set;

[0127] training a machine learning model by using the training data set to generate an abnormality detection model;

[0128] inputting the possible abnormal data into the abnormality detection model to obtain an abnormality detection result;

[0129] if the detection result of the current possible abnormal data is abnormal according to the abnormality detection result, judging that the current possible abnormal data is abnormal data.

[0130] In the embodiment, the traffic database is used for storing historical encrypted traffic data and a corresponding historical traffic clustering result; the preset amount is set in advance; the historical traffic clustering result refers to the final result of data division of historical encrypted traffic data clustering analysis; the training data set is a data set formed by adding the historical traffic clustering result as a feature to the historical encrypted traffic data; the abnormality detection model is a data anomaly classification model generated by training a machine learning model by using the training data set, and is used for judging whether data is abnormal or normal; and the abnormality detection result includes two detection results of data normal and data abnormal.

[0131] The above technical solution has the beneficial effects that: the abnormality detection model established by introducing machine learning is used for performing abnormality judgment on possible abnormal data, the accuracy and effectiveness of data analysis can be improved, and then the accuracy of network threat recognition ability and the optimization of abnormal defense decision can be improved.

[0132] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A method for frame clustering analysis of large-scale encrypted traffic based on a big data architecture, characterized in that, The method comprises the following steps: Step 1: data partitioning and storage of target encrypted data obtained after obtaining and preprocessing original encrypted traffic data; Step 2: target clustering result obtained by frame clustering analysis of the target encrypted data, and the target clustering result is visualized to obtain a clustering visualization result; Step 3: judging whether there is traffic data anomaly based on the clustering visualization result, and generating an anomaly analysis report when there is; Step 4: transmitting the anomaly analysis report to the security management personnel, which is conducive to timely taking corresponding measures for defense; The target clustering result obtained by frame clustering analysis of the target encrypted data, and the target clustering result is visualized to obtain a clustering visualization result, comprises: Segmenting the target encrypted data of each partition by frame to obtain first frame data; Extracting feature information of the first frame data to obtain key frame data features; Matching a corresponding first clustering algorithm list according to the current partition business requirement; If there is only one kind of first clustering algorithm in the first clustering algorithm list, then using the current first clustering algorithm to perform clustering analysis on the first frame data based on the key frame data features to obtain the target clustering result; If there are multiple first clustering algorithms in the first clustering algorithm list, then using each first clustering algorithm to perform clustering analysis on the first frame data based on the key frame data features for a preset number of times, and collecting clustering labels of frame data points to obtain a pending clustering result; Performing data clustering analysis and effect evaluation based on the pending clustering result to generate a target clustering result; Visualizing the target clustering result using a pre-set interactive tool to obtain a clustering visualization result. 2.The method of claim 1, wherein, The target encrypted data obtained after obtaining and preprocessing original encrypted traffic data is partitioned and stored, comprising: Real-time capturing original encrypted traffic data using stream data transmission technology, and transmitting to a target big data platform; The target big data platform uses a pre-set data cleaning tool to preprocess the received original encrypted traffic data to obtain first encrypted data; Based on the current business requirement, a corresponding matching target data partition strategy is determined; According to the target data partition strategy, the first encrypted data is divided into target encrypted data and partitioned and stored in a pre-set storage system. 3.The method of claim 1, wherein, The key frame data features include source IP address, target IP address, port, protocol type, timestamp, TLS / SSL handshake information and traffic features. 4.The method of claim 1, wherein, The target clustering result obtained by frame clustering analysis of the target encrypted data, and the target clustering result is visualized to obtain a clustering visualization result, comprises: According to the pending clustering result, the frequency of each frame data point being assigned to each cluster is obtained; Assigning the frame data point to the cluster with the highest frequency to generate a first clustering result; Performing clustering evaluation on the first clustering result to obtain a first evaluation coefficient; If the first evaluation coefficient is greater than a set evaluation threshold, then the corresponding first clustering result is marked as a first-level clustering result; Otherwise, the corresponding first clustering result is marked as a second-level clustering result; Obtaining and determining a comprehensive clustering evaluation coefficient based on the proportion of the first-level clustering result and the second-level clustering result; When the comprehensive clustering evaluation coefficient is greater than a set comprehensive evaluation threshold, a target clustering result is generated and output based on the current first clustering result; When the comprehensive clustering evaluation coefficient is not greater than the set comprehensive evaluation threshold, a new clustering result is generated by performing preset number of re-clustering analysis on the current first frame data using a first clustering algorithm; After weighted average of all new clustering results, a first comprehensive attribution matrix is created; According to the first comprehensive attribution matrix, a first attribution value of each frame data point in each cluster is obtained; A first deviation coefficient is obtained by performing deviation analysis on the first attribution value; The first attribution value is corrected using the first deviation coefficient to obtain a target attribution value of each frame data point in different clusters; The cluster with the highest target attribution value is regarded as the attribution cluster of the current frame data point; The cluster attribution decisions of all frame data points are collected to generate a target clustering result.

5. The method of claim 4, wherein the method is based on a big data architecture. The calculation formula of the first deviation coefficient is as follows: ; wherein P represents a first deviation coefficient; represents an average value of the first membership value of the current frame data point in each cluster; represents the first membership value of the current frame data point in the current cluster; represents an initial weight value of the i-th alternative clustering algorithm for forming the current cluster; represents a membership influence weight; represents a cluster data distribution evaluation coefficient of the current cluster; represents a cluster data distribution influence weight; represents a total value of the clustering effect evaluation values of the alternative clustering algorithms for forming the current cluster; represents a clustering effect evaluation value of the i-th alternative clustering algorithm for forming the current cluster; represents a clustering effect evaluation threshold value; represents the j-th cluster characteristic of the current cluster; represents an adjustment influence coefficient of the j-th cluster characteristic on the clustering effect evaluation value of the i-th alternative clustering algorithm for forming the current cluster. 6.The method of claim 1, wherein, Based on the clustering visualization result, it is determined whether there is abnormal flow data, and when there is, an abnormal analysis report is generated, including: According to the clustering visualization result, it is analyzed whether there is frame data point not attributed to any cluster, and if there is, the corresponding frame data point is marked as possible abnormal data; The first distance of each frame data point in each cluster in the target clustering result to the cluster center is extracted; When the first distance is greater than a set distance, the corresponding frame data point is marked as possible abnormal data; The possible abnormal data is detected for abnormality, and when the detection result is abnormal, the current possible abnormal data is determined as abnormal data; Based on all the abnormal data, an abnormal analysis report is generated.

7. The method of claim 6, wherein the method is based on a big data architecture, and the method comprises: The possible abnormal data is detected for abnormality, and when the detection result is abnormal, the current possible abnormal data is determined as abnormal data, including: ​ A preset amount of historical encrypted flow data is extracted from the flow database; The corresponding historical flow clustering result of the historical encrypted flow data is extracted, and each cluster in the historical flow clustering result is added with normal and abnormal labels respectively; The historical flow clustering result is added as a feature to the historical encrypted flow data to construct a training data set; The machine learning model is trained using the training data set to generate an abnormality detection model; The possible abnormal data is input into the abnormality detection model to obtain an abnormality detection result; According to the abnormality detection result, if the detection result of the current possible abnormal data is abnormal, it is determined as abnormal data.

Images