Large-scale encrypted traffic frame-by-frame clustering analysis method based on big data architecture

Abstract

The invention provides a large-scale encrypted traffic frame-by-frame clustering analysis method based on a big data architecture, and relates to the technical field of big data, and the method comprises the steps: carrying out the data partitioning and storage of target encrypted data obtained through the preprocessing of original encrypted traffic data; based on a clustering visualization result obtained by visualizing a target clustering result obtained by carrying out frame clustering analysis on the target encrypted data, judging whether a data traffic abnormal behavior exists or not, and when the data traffic abnormal behavior exists, generating an abnormal analysis report; and an abnormal analysis report is transmitted to safety management personnel so as to take corresponding measures in time for defense. The method comprises the following steps: visualizing a target clustering result obtained by carrying out frame clustering analysis after processing and partition storage on encrypted traffic data based on a big data architecture, identifying traffic data exception, generating an exception analysis report when the exception exists, and transmitting the exception analysis report to safety management personnel to take corresponding measures for defense. The network threat identification capability is effectively enhanced, and the overall protection level of network security is further improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture. Background Art

[0002] In recent years, with the development of the Internet and the Internet of Things, data volumes have exploded. The widespread adoption of HTTPS and various encryption protocols has resulted in the majority of network traffic becoming encrypted. While the introduction of encrypted traffic significantly improves the security and privacy of network communications, it also increases the complexity of network security defenses, posing challenges to traditional traffic monitoring and anomaly analysis. Therefore, efficient and secure processing and analysis of massive amounts of encrypted data to support network security monitoring and reduce threat detection has become a key research priority.

[0003] Therefore, the present invention provides a frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture. Summary of the Invention

[0004] The present invention provides a large-scale encrypted traffic frame clustering analysis method based on a big data architecture, which is used to visualize the target clustering results obtained by frame clustering analysis after processing and partitioning the encrypted traffic data based on the big data architecture, identify traffic data anomalies, and generate anomaly analysis reports when anomalies exist, and transmit them to security management personnel so that corresponding measures can be taken for defense, thereby effectively enhancing the ability to identify network threats and thus improving the overall protection level of network security.

[0005] The present invention provides a frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture, comprising: Step 1: Obtain and pre-process the original encrypted traffic data to obtain the target encrypted data, partition and store it; Step 2: performing frame-by-frame clustering analysis on the target encrypted data to obtain a target clustering result, and visualizing the target clustering result to obtain a cluster visualization result; Step 3: Based on the cluster visualization results, determine whether there is any traffic data anomaly, and generate an anomaly analysis report if there is any; Step 4: Transmitting the abnormality analysis report to the security management personnel is conducive to taking corresponding measures for defense in a timely manner.

[0006] Preferably, obtaining and pre-processing the original encrypted traffic data to obtain target encrypted data, partitioning and storing the data, includes: Use streaming data transmission technology to capture original encrypted traffic data in real time and transmit it to the target big data platform; The target big data platform pre-processes the received original encrypted traffic data using a preset data cleaning tool to obtain first encrypted data; Determine the corresponding target data partitioning strategy based on current business needs; According to the target data partitioning strategy, the first encrypted data is divided into target encrypted data, and the target encrypted data is stored in partitions in a pre-set storage system.

[0007] Preferably, performing frame-by-frame clustering analysis on the target encrypted data to obtain a target clustering result, and visualizing the target clustering result to obtain a cluster visualization result, including: The target encrypted data of each partition is divided into frames to obtain the first frame data; Extracting feature information of the first frame data to obtain key frame data features; Match the corresponding first clustering algorithm list according to the business needs of the current partition; If there is only one first clustering algorithm in the first clustering algorithm list, cluster analysis is performed on the first frame data based on the key frame data features using the current first clustering algorithm to obtain a target clustering result; If there are multiple first clustering algorithms in the first clustering algorithm list, each first clustering algorithm is used to perform a preset number of clustering analyses on the first frame data based on the key frame data features, and cluster labels of the frame data points are collected to obtain a pending clustering result; Performing data clustering analysis and effect evaluation based on the pending clustering results to generate target clustering results; The target clustering result is visualized using a preset interactive tool to obtain a clustering visualization result.

[0008] Preferably, the key frame data features include source IP address, destination IP address, port, protocol type, timestamp, TLS / SSL handshake information and traffic characteristics.

[0009] Preferably, cluster analysis and effect evaluation are performed based on the pending clustering results to generate target clustering results, including: According to the pending clustering result, obtaining the frequency of each frame data point being assigned to each cluster; Assign the frame data points to the cluster with the highest frequency to generate the first clustering result; Performing cluster evaluation on the first clustering result to obtain a first evaluation coefficient; If the first evaluation coefficient is greater than the set evaluation threshold, the corresponding first clustering result is marked as a first-level clustering result; Otherwise, the corresponding first clustering result will be marked as the second-level clustering result; Obtaining and determining a comprehensive clustering evaluation coefficient based on the proportions of the first-level clustering results and the second-level clustering results; When the comprehensive clustering evaluation coefficient is greater than the set comprehensive evaluation threshold, a target clustering result is generated based on the current first clustering result and output; When the comprehensive clustering evaluation coefficient is not greater than the set comprehensive evaluation threshold, performing a preset number of re-clustering analyses on the current first frame data using the first clustering algorithm to generate a new clustering result; After weighted averaging all new clustering results, the first comprehensive attribution matrix is ​​created; Obtaining a first attribution value of each frame data point in each cluster according to the first comprehensive attribution matrix; Performing a deviation analysis on the first attributed value to obtain a first deviation coefficient; Correcting the first attribution value using the first deviation coefficient to obtain a target attribution value for each frame data point in a different cluster; The cluster with the highest target belonging value is regarded as the belonging cluster of the current frame data point; The cluster attribution decisions of all frame data points are collected to generate the target clustering results.

[0010] Preferably, the calculation formula of the first deviation coefficient is as follows: ; Where P represents the first deviation coefficient; It is expressed as the average of the first attribute value of the current frame data point in each cluster; It represents the first attribute value of the current frame data point in the current cluster; It represents the initial weight value of the i-th optional clustering algorithm that forms the current cluster; Expressed as attributed impact weight; It is represented as the cluster data distribution evaluation coefficient of the current cluster; Expressed as the cluster data distribution influence weight; It is expressed as the total value of the clustering effect evaluation value of the optional clustering algorithm that forms the current cluster; is the clustering effect evaluation value of the i-th optional clustering algorithm that forms the current cluster; It represents the setting of clustering effect evaluation threshold; Represented as the jth cluster feature of the current cluster; It is expressed as the adjustment influence coefficient of the j-th cluster characteristic on the clustering effect evaluation value of the i-th optional clustering algorithm that forms the current cluster.

[0011] Preferably, judging whether there is traffic data anomaly based on the cluster visualization result and generating an anomaly analysis report when there is an anomaly includes: According to the cluster visualization results, analyzing whether there are frame data points that are not classified into any cluster, and if so, marking the corresponding frame data points as possible abnormal data; Extracting a first distance between each frame data point in each cluster in the target clustering result and the center of the cluster to which it belongs; When the first distance is greater than a set distance, the corresponding frame data point is marked as possible abnormal data; Performing anomaly detection on the possible abnormal data, and when the detection result is abnormal, determining the current possible abnormal data as abnormal data; Based on all the abnormal data, an abnormality analysis report is generated.

[0012] Preferably, performing anomaly detection on the possible abnormal data, and when the detection result is abnormal, determining the current possible abnormal data as abnormal data, includes: extracting a preset amount of historical encrypted traffic data from a traffic database; Extract the corresponding historical traffic clustering results of the historical encrypted traffic data, and add normal and abnormal labels to each cluster in the historical traffic clustering results; The historical traffic clustering results are added as features to the historical encrypted traffic data to construct a training dataset. Using the training data set to train a machine learning model to generate an anomaly detection model; Inputting the possible abnormal data into an anomaly detection model to obtain an anomaly detection result; According to the abnormality detection result, if the detection result of the current possible abnormal data is abnormal, it is determined to be abnormal data.

[0013] Compared with the prior art, the present invention has the following advantages: By visualizing the target clustering results obtained by frame clustering analysis after processing encrypted traffic data based on a big data architecture and partitioning and storing it, traffic data anomalies can be identified. When anomalies occur, an anomaly analysis report is generated and transmitted to security management personnel so that corresponding measures can be taken for defense. This effectively enhances the ability to identify network threats, thereby improving the overall protection level of network security.

[0014] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0015] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a flowchart of a frame-based clustering analysis method for large-scale encrypted traffic based on a big data architecture in an embodiment of the present invention. DETAILED DESCRIPTION The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0017] The embodiment of the present invention provides a large-scale encrypted traffic frame clustering analysis method based on big data architecture, such as Figure 1 Shown, including: Step 1: Obtain and pre-process the original encrypted traffic data to obtain the target encrypted data, partition and store it; Step 2: performing frame-by-frame clustering analysis on the target encrypted data to obtain a target clustering result, and visualizing the target clustering result to obtain a cluster visualization result; Step 3: Based on the cluster visualization results, determine whether there is any traffic data anomaly, and generate an anomaly analysis report if there is any; Step 4: Transmitting the abnormality analysis report to the security management personnel is conducive to taking corresponding measures for defense in a timely manner.

[0018] In this embodiment, encrypted traffic data refers to data packets that are protected and transmitted using various encryption algorithms and technologies; target encrypted data is data obtained by pre-processing the original encrypted traffic data using a preset data cleaning tool, and the data is divided and stored in partitions according to a target data partitioning strategy, wherein the preset data cleaning tool is set in advance; pre-processing includes data duplicate value processing, missing value completion, and denoising processing, etc.

[0019] In this embodiment, frame clustering analysis refers to clustering the encrypted traffic data according to its frame structure or characteristics, which usually involves extracting and analyzing the frame header, frame length, timestamp and other information of the network encrypted traffic data packet; the target clustering result is the final result obtained by clustering the target encrypted data by frame and dividing the data.

[0020] In this embodiment, cluster visualization results refer to the use of visualization technology to display the distribution of clusters in the target clustering results and the relationship between data points, such as scatter plots, heat maps, tree diagrams, etc.; the anomaly analysis report is a report generated based on abnormal encrypted traffic data, which is used to lay data support for effectively analyzing clustering results and identifying abnormal behavior patterns in the data, and then taking corresponding measures for processing or optimization.

[0021] The beneficial effects of the above technical solution are: by visualizing the target clustering results obtained by frame clustering analysis after processing and partitioning encrypted traffic data based on a big data architecture, identifying traffic data anomalies, and generating anomaly analysis reports when anomalies exist and transmitting them to security management personnel to take corresponding measures for defense, the ability to identify network threats is effectively enhanced, thereby improving the overall protection level of network security.

[0022] The embodiment of the present invention provides a frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture, which obtains and pre-processes the original encrypted traffic data to obtain target encrypted data, and performs data partitioning and storage, including: Use streaming data transmission technology to capture original encrypted traffic data in real time and transmit it to the target big data platform; The target big data platform pre-processes the received original encrypted traffic data using a preset data cleaning tool to obtain first encrypted data; Determine the corresponding target data partitioning strategy based on current business needs; According to the target data partitioning strategy, the first encrypted data is divided into target encrypted data, and the target encrypted data is stored in partitions in a pre-set storage system.

[0023] In this embodiment, encrypted traffic data refers to data packets that are protected and transmitted using various encryption algorithms and technologies; the first encrypted data is obtained by pre-processing the original encrypted traffic data using a preset data cleaning tool, wherein the preset data cleaning tool is set in advance; the pre-processing includes data duplicate value processing, missing value filling, and denoising processing, etc.; the big data platform is used to process, store, and analyze large-scale data.

[0024] In this embodiment, business requirements include real-time performance, data volume, access mode, and security compliance requirements; the target data partitioning strategy is a data partitioning strategy selected based on business requirements, wherein the data partitioning strategy includes time series partitioning, protocol type partitioning, traffic feature partitioning, etc.; the target encrypted data is the encrypted data obtained by dividing the first encrypted traffic data according to the target data partitioning strategy; the pre-set storage system is used to store massive network encrypted traffic data, such as Hadoop HDFS and Spark.

[0025] The beneficial effects of the above technical solution are: by capturing and processing the original encrypted traffic data based on the big data structure and adopting a reasonable data partitioning storage strategy to divide and partition the encrypted traffic data, it can not only improve the efficiency and response speed of large-scale encrypted traffic analysis, but also ensure the compliance and security of data processing to meet diverse business needs.

[0026] The embodiment of the present invention provides a frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture. The method performs frame-by-frame clustering analysis on target encrypted data to obtain target clustering results, and visualizes the target clustering results to obtain cluster visualization results, including: The target encrypted data of each partition is divided into frames to obtain the first frame data; Extracting feature information of the first frame data to obtain key frame data features; Match the corresponding first clustering algorithm list according to the business needs of the current partition; If there is only one first clustering algorithm in the first clustering algorithm list, cluster analysis is performed on the first frame data based on the key frame data features using the current first clustering algorithm to obtain a target clustering result; If there are multiple first clustering algorithms in the first clustering algorithm list, each first clustering algorithm is used to perform a preset number of clustering analyses on the first frame data based on the key frame data features, and cluster labels of the frame data points are collected to obtain a pending clustering result; Performing data clustering analysis and effect evaluation based on the pending clustering results to generate target clustering results; The target clustering result is visualized using a preset interactive tool to obtain a clustering visualization result.

[0027] In this embodiment, the first frame data is frame data obtained by segmenting the target encrypted data by frame; frame clustering analysis refers to clustering the encrypted traffic data according to its frame structure or characteristics, which usually involves the extraction and analysis of frame header, frame length, timestamp and other information of the network encrypted traffic data packet; key frame data features include source IP address, target IP address, port, protocol type, timestamp, TLS / SSL handshake information and traffic characteristics.

[0028] In this embodiment, the first clustering algorithm list refers to a list established by clustering algorithms that match business requirements, where the clustering algorithms include K-means algorithm, hierarchical clustering, density-based clustering algorithm, spectral clustering, etc.; the target clustering result refers to the final result obtained by clustering and analyzing the target encrypted data by frame and dividing the data.

[0029] In this embodiment, the preset number of times is determined in advance; the clustering labels include feature descriptions, time labels, sequence labels, normal and abnormal labels, etc.; the pending clustering results are obtained by performing a preset number of clustering analyses on the first frame data corresponding to multiple first clustering algorithms in the first clustering algorithm list; the preset interactive tools are set in advance, such as MATLAB clustering toolbox, QlikView, etc.

[0030] The beneficial effects of the above technical solution are: by performing frame-by-frame clustering analysis on the target encrypted data, the target clustering results can provide data support for understanding the potential target behaviors, characteristics or categories in the encrypted data; by visualizing the target clustering results, it can provide an intuitive way to quickly understand the patterns and anomalies in complex data sets, thereby improving the efficiency and depth of data analysis.

[0031] The embodiment of the present invention provides a frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture, which performs clustering analysis and effect evaluation based on the pending clustering results to generate target clustering results, including: According to the pending clustering result, obtaining the frequency of each frame data point being assigned to each cluster; Assign the frame data points to the cluster with the highest frequency to generate the first clustering result; Performing cluster evaluation on the first clustering result to obtain a first evaluation coefficient; If the first evaluation coefficient is greater than the set evaluation threshold, the corresponding first clustering result is marked as a first-level clustering result; Otherwise, the corresponding first clustering result will be marked as the second-level clustering result; Obtaining and determining a comprehensive clustering evaluation coefficient based on the proportions of the first-level clustering results and the second-level clustering results; When the comprehensive clustering evaluation coefficient is greater than the set comprehensive evaluation threshold, a target clustering result is generated based on the current first clustering result and output; When the comprehensive clustering evaluation coefficient is not greater than the set comprehensive evaluation threshold, performing a preset number of re-clustering analyses on the current first frame data using the first clustering algorithm to generate a new clustering result; After weighted averaging all new clustering results, the first comprehensive attribution matrix is ​​created; Obtaining a first attribution value of each frame data point in each cluster according to the first comprehensive attribution matrix; Performing a deviation analysis on the first attributed value to obtain a first deviation coefficient; Correcting the first attribution value using the first deviation coefficient to obtain a target attribution value for each frame data point in a different cluster; The cluster with the highest target belonging value is regarded as the belonging cluster of the current frame data point; The cluster attribution decisions of all frame data points are collected to generate the target clustering results.

[0032] In this embodiment, the first evaluation coefficient is used to evaluate the clustering effect of the first clustering result; the evaluation threshold is set in advance; the comprehensive clustering evaluation coefficient is used to characterize the clustering effect of the first frame data; the comprehensive evaluation threshold is set in advance.

[0033] In this embodiment, for example, there are first evaluation coefficients l1, l2, and l3 of the first clustering results a1, a2, and a3, where l1 and l2 are greater than the set evaluation threshold, and l3 is equal to the set evaluation threshold. At this time, the first clustering results a1 and a2 are calibrated as first-level clustering results, and the first clustering result a3 is calibrated as a second-level clustering result.

[0034] In this embodiment, the preset number of times is set in advance; the new clustering result is the clustering result obtained after re-clustering analysis of the current first frame data for a preset number of times using the first clustering algorithm when the comprehensive clustering evaluation coefficient is not greater than the set comprehensive evaluation threshold.

[0035] In this embodiment, the first comprehensive attribution matrix is ​​a matrix constructed by combining the weights assigned to each first clustering algorithm and the new clustering results generated by re-clustering analysis of the first frame data for a preset number of times using each first clustering algorithm, and performing weighted averaging; the first attribution value refers to the highest value of the frame data point in each cluster in the first comprehensive attribution matrix; the first deviation coefficient is used to correct the first attribution value; the target attribution value is the value obtained by correcting the first attribution value by calculating the first deviation coefficient; the attribution cluster refers to the cluster with the highest target attribution value of the frame data point; the target clustering result is obtained by collecting cluster attribution decisions for all frame data points, wherein the cluster attribution decision refers to the process of determining the cluster to which each frame data point belongs.

[0036] The beneficial effect of the above technical solution is: by using different clustering algorithms to cluster the current first frame data The clustering results obtained by cluster analysis are combined and analyzed and the clustering effect is evaluated to generate the target clustering result with the best clustering effect, which can enhance the comprehensiveness and accuracy of cluster analysis.

[0037] An embodiment of the present invention provides a frame-based clustering analysis method for large-scale encrypted traffic based on a big data architecture. The calculation formula for the first deviation coefficient is as follows: ; Where P represents the first deviation coefficient; It is expressed as the average of the first attribute value of the current frame data point in each cluster; It represents the first attribute value of the current frame data point in the current cluster; It represents the initial weight value of the i-th optional clustering algorithm that forms the current cluster; Expressed as attributed impact weight; It is represented as the cluster data distribution evaluation coefficient of the current cluster; Expressed as the cluster data distribution influence weight; It is expressed as the total value of the clustering effect evaluation value of the optional clustering algorithm that forms the current cluster; is the clustering effect evaluation value of the i-th optional clustering algorithm that forms the current cluster; It represents the setting of clustering effect evaluation threshold; Represented as the jth cluster feature of the current cluster; It is expressed as the adjustment influence coefficient of the j-th cluster characteristic on the clustering effect evaluation value of the i-th optional clustering algorithm that forms the current cluster.

[0038] The beneficial effect of the above technical solution is: by calculating the first deviation coefficient to correct the first attribution value, the most accurate attribution cluster can be determined for each frame data point to generate the target clustering result with the best clustering effect, thereby achieving an accurate reflection of the true distribution and internal structure of the data, that is, improving the quality and efficiency of data analysis.

[0039] The present invention provides a frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture. The method determines whether traffic data anomalies exist based on the clustering visualization results and generates an anomaly analysis report when anomalies exist, including: According to the cluster visualization results, analyzing whether there are frame data points that are not classified into any cluster, and if so, marking the corresponding frame data points as possible abnormal data; Extracting a first distance between each frame data point in each cluster in the target clustering result and the center of the cluster to which it belongs; When the first distance is greater than a set distance, the corresponding frame data point is marked as possible abnormal data; Performing anomaly detection on the possible abnormal data, and when the detection result is abnormal, determining the current possible abnormal data as abnormal data; Based on all the abnormal data, an abnormality analysis report is generated.

[0040] In this embodiment, cluster visualization results refer to a method of using visualization technology to display the distribution of clusters and the relationship between data points in the target clustering results, such as a scatter plot, a heat map, a tree diagram, etc.; possible abnormal data refers to frame data points that are not classified into any cluster and frame data points whose first distance from the cluster to the center of the cluster is greater than the set distance; the first distance refers to the distance from the frame data point to the center of the cluster; the set distance is set in advance.

[0041] In this embodiment, the anomaly analysis report is a report generated based on abnormal encrypted traffic data, which is used to provide data support for effectively analyzing clustering results and identifying abnormal behavior patterns in the data, and then taking corresponding measures for processing or optimization.

[0042] The beneficial effect of the above technical solution is: by judging whether there is abnormal data traffic behavior based on cluster visualization results, and generating an abnormal analysis report when it exists, it can be beneficial to effectively enhance the ability to identify network threats, thereby improving the overall protection level of network security.

[0043] An embodiment of the present invention provides a frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture, which performs anomaly detection on the possible abnormal data and, when the detection result is abnormal, determines the current possible abnormal data as abnormal data, including: extracting a preset amount of historical encrypted traffic data from a traffic database; Extract the corresponding historical traffic clustering results of the historical encrypted traffic data, and add normal and abnormal labels to each cluster in the historical traffic clustering results; The historical traffic clustering results are added as features to the historical encrypted traffic data to construct a training dataset. Using the training data set to train a machine learning model to generate an anomaly detection model; Inputting the possible abnormal data into an anomaly detection model to obtain an anomaly detection result; According to the abnormality detection result, if the detection result of the current possible abnormal data is abnormal, it is determined to be abnormal data.

[0044] In this embodiment, the traffic database is used to store historical encrypted traffic data and corresponding historical traffic clustering results; the preset quantity is set in advance; the historical traffic clustering result refers to the final result of data division of the clustering analysis of the historical encrypted traffic data; the training data set is a data set formed by adding the historical traffic clustering results as features to the historical encrypted traffic data; the anomaly detection model is a data anomaly classification model generated by training the machine learning model using the training data set, which is used to determine whether the data is abnormal or normal; the anomaly detection results include two detection results: normal data and abnormal data.

[0045] The beneficial effect of the above technical solution is: by introducing machine learning to establish an anomaly detection model to make anomaly judgments on possible abnormal data, the accuracy and effectiveness of data analysis can be improved, which in turn is conducive to improving the accuracy of network threat identification capabilities and optimizing abnormal defense decisions.

[0046] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A frame-by-frame clustering analysis method for large-scale encrypted traffic based on a big data architecture, characterized in that: include: Step 1: Obtain and pre-process the original encrypted traffic data to obtain the target encrypted data, partition and store it; Step 2: performing frame-by-frame clustering analysis on the target encrypted data to obtain a target clustering result, and visualizing the target clustering result to obtain a cluster visualization result; Step 3: Based on the cluster visualization results, determine whether there is any traffic data anomaly, and generate an anomaly analysis report if there is any; Step 4: Transmitting the abnormality analysis report to the security management personnel is conducive to taking corresponding measures for defense in a timely manner.

2. A frame-by-frame clustering analysis method for large-scale encrypted traffic based on big data architecture according to claim 1, characterized in that: The target encrypted data obtained after obtaining and preprocessing the original encrypted traffic data is partitioned and stored, including: Use streaming data transmission technology to capture original encrypted traffic data in real time and transmit it to the target big data platform; The target big data platform pre-processes the received original encrypted traffic data using a preset data cleaning tool to obtain first encrypted data; Determine the corresponding target data partitioning strategy based on current business needs; According to the target data partitioning strategy, the first encrypted data is divided into target encrypted data, and the target encrypted data is stored in partitions in a pre-set storage system.

3. The method for clustering and analyzing large-scale encrypted traffic by frame based on big data architecture according to claim 1 is characterized in that: Performing frame-by-frame clustering analysis on the target encrypted data to obtain a target clustering result, and visualizing the target clustering result to obtain a cluster visualization result, including: The target encrypted data of each partition is divided into frames to obtain the first frame data; Extracting feature information of the first frame data to obtain key frame data features; Match the corresponding first clustering algorithm list according to the business needs of the current partition; If there is only one first clustering algorithm in the first clustering algorithm list, cluster analysis is performed on the first frame data based on the key frame data features using the current first clustering algorithm to obtain a target clustering result; If there are multiple first clustering algorithms in the first clustering algorithm list, each first clustering algorithm is used to perform a preset number of clustering analyses on the first frame data based on the key frame data features, and cluster labels of the frame data points are collected to obtain a pending clustering result; Performing data clustering analysis and effect evaluation based on the pending clustering results to generate target clustering results; The target clustering result is visualized using a preset interactive tool to obtain a clustering visualization result.

4. The method for clustering and analyzing large-scale encrypted traffic by frame based on a big data architecture according to claim 3 is characterized in that: Key frame data features include source IP address, destination IP address, port, protocol type, timestamp, TLS / SSL handshake information, and traffic characteristics.

5. The method for clustering and analyzing large-scale encrypted traffic by frame based on big data architecture according to claim 3 is characterized in that: Perform cluster analysis and effect evaluation based on the pending clustering results to generate target clustering results, including: According to the pending clustering result, obtaining the frequency of each frame data point being assigned to each cluster; Assign the frame data points to the cluster with the highest frequency to generate the first clustering result; Performing cluster evaluation on the first clustering result to obtain a first evaluation coefficient; If the first evaluation coefficient is greater than the set evaluation threshold, the corresponding first clustering result is marked as a first-level clustering result; Otherwise, the corresponding first clustering result will be marked as the second-level clustering result; Obtaining and determining a comprehensive clustering evaluation coefficient based on the proportions of the first-level clustering results and the second-level clustering results; When the comprehensive clustering evaluation coefficient is greater than the set comprehensive evaluation threshold, a target clustering result is generated based on the current first clustering result and output; When the comprehensive clustering evaluation coefficient is not greater than the set comprehensive evaluation threshold, performing a preset number of re-clustering analyses on the current first frame data using the first clustering algorithm to generate a new clustering result; After weighted averaging all new clustering results, the first comprehensive attribution matrix is ​​created; Obtaining a first attribution value of each frame data point in each cluster according to the first comprehensive attribution matrix; Performing a deviation analysis on the first attributed value to obtain a first deviation coefficient; Correcting the first attribution value using the first deviation coefficient to obtain a target attribution value for each frame data point in a different cluster; The cluster with the highest target belonging value is regarded as the belonging cluster of the current frame data point; The cluster attribution decisions of all frame data points are collected to generate the target clustering results.

6. The method for clustering and analyzing large-scale encrypted traffic by frame based on big data architecture according to claim 5 is characterized in that: The calculation formula of the first deviation coefficient is as follows: ; Where P represents the first deviation coefficient; It is expressed as the average of the first attribute value of the current frame data point in each cluster; It represents the first attribute value of the current frame data point in the current cluster; It represents the initial weight value of the i-th optional clustering algorithm that forms the current cluster; Expressed as attributed impact weight; It is represented as the cluster data distribution evaluation coefficient of the current cluster; Expressed as the cluster data distribution influence weight; It is expressed as the total value of the clustering effect evaluation value of the optional clustering algorithm that forms the current cluster; is the clustering effect evaluation value of the i-th optional clustering algorithm that forms the current cluster; It represents the threshold value for clustering effect evaluation; Represented as the jth cluster feature of the current cluster; It is expressed as the adjustment influence coefficient of the j-th cluster characteristic on the clustering effect evaluation value of the i-th optional clustering algorithm that forms the current cluster.

7. The method for clustering and analyzing large-scale encrypted traffic by frame based on a big data architecture according to claim 1 is characterized in that: Based on the cluster visualization results, it is determined whether there is any traffic data anomaly, and if so, an anomaly analysis report is generated, including: According to the cluster visualization results, analyzing whether there are frame data points that are not classified into any cluster, and if so, marking the corresponding frame data points as possible abnormal data; Extracting a first distance between each frame data point in each cluster in the target clustering result and the center of the cluster to which it belongs; When the first distance is greater than a set distance, the corresponding frame data point is marked as possible abnormal data; Performing anomaly detection on the possible abnormal data, and when the detection result is abnormal, determining the current possible abnormal data as abnormal data; Based on all the abnormal data, an abnormality analysis report is generated.

8. The method for clustering and analyzing large-scale encrypted traffic by frame based on big data architecture according to claim 7 is characterized in that: Performing anomaly detection on the possible abnormal data, and when the detection result is abnormal, determining the current possible abnormal data as abnormal data, including: extracting a preset amount of historical encrypted traffic data from a traffic database; Extract the corresponding historical traffic clustering results of the historical encrypted traffic data, and add normal and abnormal labels to each cluster in the historical traffic clustering results; The historical traffic clustering results are added as features to the historical encrypted traffic data to construct a training dataset. Using the training data set to train a machine learning model to generate an anomaly detection model; Inputting the possible abnormal data into an anomaly detection model to obtain an anomaly detection result; According to the abnormality detection result, if the detection result of the current possible abnormal data is abnormal, it is determined to be abnormal data.

Images