Network flow space mapping representation method, device and storage medium based on clustering compression

Abstract

The present invention provides a network flow space mapping characterization method, device and storage medium based on clustering compression. The method comprises: a training sample acquisition step, which is used to acquire training samples of different network space behavior categories to form a sample set; clustering the training samples of each behavior category in the training sample set; inputting subcategory sample data in the sample space into a multi-layer mapping network model created based on the principle of maximizing the distance between classes and minimizing the distance within classes, calculating and updating the model parameters including the expansion factor and the compression factor layer by layer starting from the first layer of the model, and generating a filter for calculating the weight of the compression factor and weighting the result in the test phase based on the updated model parameters; a test sample acquisition step; and a test step, inputting the obtained sample set into the multi-layer mapping network model trained in the training phase, and outputting a space mapping matrix.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a network flow space mapping characterization method, device and storage medium based on clustering compression. Background Art

[0002] Network characterization is the use of structured data to characterize different network space behaviors, so that more in-depth analysis or detection can be performed based on the characterization results. Anomaly-based detection technology is a commonly used detection technology in network defense. It extracts different data features from the original network flow to characterize network behavior and perform subsequent intrusion detection tasks. Common network flow data feature analysis tools include Wireshark and CICFlowMeter. With the complexity of the network environment, network attacks tend to be hidden. The data features (original features) of the original network flow have limited capabilities in network characterization tasks. Extracting original features from network flows can no longer well characterize these network behaviors. In order to improve the expressiveness of original features for network behavior, some existing research methods are to use machine learning or deep learning to automatically mine data internal features. In machine learning, data feature information can be mined according to the characteristics of different algorithms, such as using GBDT+LR to estimate CTR click-through rate; in deep learning, deep features of data can be continuously extracted through operations such as convolution and pooling. The separability of these deep features is relatively high and can well achieve data characterization.

[0003] However, most existing machine learning algorithms directly apply mined information to tasks such as classification or regression, and are unable to present data features in a homogeneous manner. Deep learning, as an end-to-end "black box" model, cannot explain the relationship between deep features and raw data and is not interpretable. These problems limit the application of machine learning and deep learning in data representation tasks in actual business scenarios. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a network flow space mapping characterization method and device based on clustering compression, constructs a multi-layer mapping model through the principles of data compression and classification, and maps the complex degenerate original features into different independent subspaces according to different network space behavior categories, so as to improve the expressiveness of network flow data on network space behavior.

[0005] In one aspect of the present invention, a network flow space mapping characterization method based on clustering compression is provided, the method comprising the following steps:

[0006] The method includes a training phase and a testing phase;

[0007] The training phase consists of the following steps:

[0008] A training sample acquisition step is used to acquire training samples of different cyberspace behavior categories to form a sample set, wherein the training samples of different cyberspace behavior categories are numerical matrices of different behavior categories generated from features of a predetermined number of dimensions extracted from captured network flow data of different cyberspace behavior categories;

[0009] Clustering step: clustering the training samples of each behavior category in the training sample set using a clustering algorithm, clustering the samples of each behavior category into a predetermined number of clusters, and generating subcategory sample data in the sample space based on each cluster of different behavior categories;

[0010] The multi-layer mapping network model construction step is as follows: the sub-category sample data in the sample space is input into the multi-layer mapping network model created based on the principle of maximizing the inter-class distance and minimizing the intra-class distance, and the model parameters including the expansion factor and the compression factor are calculated and updated layer by layer starting from the first layer of the model, and a filter is generated based on the updated model parameters for calculating the compression factor weight and weighting the result in the test phase, wherein the expansion factor is used to increase the inter-class distance of the overall sample space, and the compression factor is used to reduce the intra-class distance of each category of sample space;

[0011] The testing phase includes the following steps:

[0012] A test sample acquisition step for obtaining a predetermined number of test samples in the form of a numerical matrix in a sample space;

[0013] In the testing step, the obtained sample set is input into the multi-layer mapping network model trained in the training phase, and the spatial mapping matrix is ​​output.

[0014] In some embodiments of the present invention, the clustering algorithm is a K-means clustering algorithm;

[0015] The input of the K-means clustering algorithm is the numerical matrix of a single behavior category and the number of clusters of the current category. The clustering of the numerical matrix of each behavior category obtained by using the clustering algorithm includes:

[0016] Randomly select a predetermined number of clusters from the cluster sample set formed by the numerical matrix of each behavior category as the mean vector of each cluster in the current category;

[0017] According to the distance between each sample of the current behavior category and each mean vector, the samples are divided into the cluster where the nearest mean vector is located;

[0018] Recalculate the mean vector of the corresponding cluster based on the currently divided cluster;

[0019] Clustering is iterated based on the recalculated mean vector until the maximum number of iterations is reached or the mean vector no longer changes.

[0020] In some embodiments of the present invention, the method further comprises: outputting each subspace matrix data in the output flow spectrum space matrix to a downstream network flow characterization task.

[0021] In some embodiments of the present invention, the expansion factor is expressed as:

[0022] The compression factor is expressed as:

[0023] Among them, the matrix is the input data of the lth layer The expansion factor, is the input data of the lth layer The compression factor of the j-th class sample in , ∈ is the error rate, represents the input matrix of the lth layer in the multi-layer mapping network model, Representation Matrix The adjoint matrix, I is the identity matrix, n is the number of features of the sample data, and m represents the sample space The total number of samples in tr j ) is the matrix Π j The trace of j is a set of diagonal matrices used to screen sample data of the jth category.

[0024] In some embodiments of the present invention, the filter is constructed as follows:

[0025]

[0026] in, is a filter, which indicates the probability that the output data of the lth layer is filtered into the jth category, and λ is a constant.

[0027] In some embodiments of the present invention, the output data of each layer in the trained multi-layer mapping network model satisfies the following formula:

[0028]

[0029] Among them, η is the learning rate, γ j It is calculated during the training phase based on the following formula:

[0030]

[0031] Among them, tr j ) is the matrix Π j The trace of represents the number of samples of category j.

[0032] In another aspect of the present invention, before generating the numerical matrix corresponding to the behavior category, the method further comprises:

[0033] In the case where the captured network flow data of different cyberspace behavior categories are original traffic packet data captured from a network environment, reducing the feature dimension by principal component analysis or PCA dimensionality reduction;

[0034] In the case where the captured network flow data of different cyberspace behavior categories is a feature set obtained by feature engineering, some features are deleted according to the feature importance in the classification task to reduce the feature dimension.

[0035] In some embodiments of the present invention, during the testing phase, the input data of the multi-layer mapping network model is an n×m matrix, the expansion factor and compression factor are an m×m matrix, and the output data is an n×n matrix, where n is the number of features of the sample data and m is the total number of samples in the sample space Z.

[0036] Another aspect of the present invention provides a network flow space mapping characterization device based on clustering compression, the device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0037] Another aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above-mentioned method when executed by a processor.

[0038] The network flow space mapping characterization method and device based on clustering compression provided by the present invention can realize the task of improving the network flow characterization capability by starting from the two principles of data compression and classification, and mapping the originally complex and de-distributed traffic data to a homogeneous space with stronger characterization capability according to different network behavior categories through spatial mapping. The model adopted by the present invention is constructed forwardly by designing optimization targets, which makes the flow spectrum construction process highly interpretable.

[0039] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0041] Figure 1The figure is a flow chart of a network flow space mapping characterization method based on clustering compression in one embodiment of the present invention.

[0042] Figure 2 The figure is a general architecture diagram of a network flow space mapping model based on clustering compression in one embodiment of the present invention.

[0043] Figure 3 Schematic diagram of the structure of a multi-layer mapping model in one embodiment of the present invention.

[0044] Figure 4 Schematic diagram of the comparison of the thermal map of the original flow characteristic (a) and the flow spectrum characteristic (b) in another embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0046] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0047] It should be emphasized that the terms “include / comprises / has” when used herein refer to the presence of features, elements, steps or components, but do not exclude the presence or addition of one or more other features, elements, steps or components.

[0048] In order to solve the problems in the prior art that it is difficult for machine learning algorithms to present data features in an isomorphic way when performing network characterization, and the relationship between deep features obtained by deep learning and original data is not interpretable. The present invention constructs a multi-layer mapping model that can present isomorphic space mapping through the principles of data compression and classification. The present invention maps the complex degenerate original features into different independent subspaces according to different network space behavior categories, thereby improving the expressiveness of network flow data on network space behavior. In the present invention, this isomorphic space obtained by mapping the original feature space is called "flow spectrum space", which is highly interpretable. At the same time, the present invention introduces a clustering method in the front of spatial mapping, extracts the distribution features inside the original data in an unsupervised manner, and constructs a subsequent spatial mapping model in combination with the data distribution characteristics, which has a better effect than directly mapping the space according to the behavior label.

[0049] In an embodiment of the present invention, a spatial mapping method is used to enhance the representation capability of network flow data for different network space behaviors. Spatial mapping refers to the "corresponding" relationship between elements of two spaces. Data analysis in the original data space is often very difficult due to the complexity and diversity of the data structure. If a certain correspondence can be found to map the data to other spaces, the difficulty of the data analysis task may be greatly reduced. By performing isomorphic mapping on the network flow data, the originally complexly distributed features can be mapped to relatively independent subspaces according to different network behavior categories. However, how to achieve isomorphic mapping is a difficult problem, and it is also a problem that the present invention aims to solve. In the present invention, network space behavior refers to a set of operations formed by users in a network environment according to different needs. In an embodiment of the present invention, different attack types are used as different network space behavior categories, but the present invention is not limited to this.

[0050] In the embodiment of the present invention, the processing flow of the multi-layer mapping network model can be divided into two stages: training and testing. According to the structure of the multi-layer mapping network model, it can be divided into four parts: data processing part, clustering part, multi-layer mapping network model construction (model training) and testing part, wherein the data processing part is a part that may be involved in both the training stage and the testing stage, and the clustering part is a part that is executed for training the model. There is no need to perform the clustering step in the testing stage. Therefore, in the embodiment of the present invention, the clustering part and the model training part belong to the training stage, and the testing part belongs to the testing stage. In addition, the output data of the testing stage can be further sent to the downstream task part. The training stage and the testing stage will be described separately below.

[0051] Figure 1 FIG. 1 is a flow chart of a network flow space mapping characterization method based on clustering compression proposed in one embodiment of the present invention. Figure 1 As shown, the method includes a training phase and a testing phase. The training phase includes steps S110-S130, and the testing phase includes steps S140-S150.

[0052] Step S110, a training sample acquisition step, is used to acquire training samples of different cyberspace behavior categories to form a sample set, wherein the training samples of different cyberspace behavior categories are numerical matrices of different behavior categories generated by extracting features of a predetermined number of dimensions from captured network flow data of different cyberspace behavior categories.

[0053] The input data of the multi-layer mapping network model is a numerical matrix composed of features. The Ethernet data packets can be captured by the network flow capture tool and the captured data flow can be split into sample data according to the five-tuple (protocol, source IP address, destination IP address, source port and destination port) by the feature extraction tool (such as CICFlowMeter) and the data features can be extracted. Different categories of data can be distinguished according to the captured network behavior. In one embodiment of the present invention, six different categories of network space behaviors are used: Reconnaissance, Fuzzers, Exploits, Generic, DoS and Normal, of which the first five are abnormal network behaviors, and the sample data uses sample data from the benchmark data set UNSW-NB15.

[0054] After extracting features from the captured data stream, to obtain a numerical matrix, it is preferred to perform additional data processing operations on the extracted features. As an example, data processing operations may include: removing unique attributes (e.g., removing unique features of attributes), processing missing values ​​(removing features with missing values), encoding attributes, data standardization and regularization, etc. Among them, the encoding part adopts a sequential encoding method, that is, encoding in natural number order by counting the types of non-numerical features. This encoding does not increase the data dimension and is suitable for model training; the processing of default values ​​can be completed by mean interpolation, that is, the mean value or median of all sample data under this type of feature is counted to interpolate the missing values. Since these data processing operations are common operations performed on the extracted features, they will not be repeated here. This data processing step is an optional step and can also be omitted if the extracted features meet the requirements.

[0055] Step S120, clustering step: clustering the training samples of each behavior category in the training sample set using a clustering algorithm, clustering the samples of each behavior category into a predetermined number of clusters, and generating subcategory sample data in the sample space based on each cluster of different behavior categories.

[0056] In one embodiment of the present invention, a prototype-based K-means clustering algorithm is used to mine the intrinsic distribution structure of data, that is, clustering is performed on the numerical matrix obtained after data processing. A clustering algorithm is used for each different category, and the number of clusters is set to k. Assuming that the network traffic has N behavior categories when captured, the final sample subcategories are kN in total. Because there are certain differences in data distribution between subcategories, these data are regarded as network flow data of kN different behavior categories as samples for subsequent model training to achieve the effect of data enhancement.

[0057] The K-means algorithm targets the input sample set D = {x 1 ,x 2,…,x N The algorithm aims to find a set of "prototypes" to characterize the clustering structure of the data. The input of the algorithm is the unlabeled sample set D and the number of clusters k to be divided. The output is the clusters divided according to the criterion of minimizing the square error. The calculation formula for the squared error in clustering is as follows:

[0058]

[0059] Among them, μ i Represents the i-th cluster C i The mean vector of x is the sample data in the cluster. Minimizing the above formula requires finding all possible cluster divisions in the data set, which is often an NP-hard problem. Therefore, in the embodiment of the present invention, the K-means algorithm actually used adopts a greedy strategy to achieve an approximate solution through iterative optimization. First, k samples are randomly selected from the sample set D as clusters C i The mean vector μ i ={μ 1 ,μ 2 ,…,μ k}, and then according to the distance from the sample data to each mean vector, such as the Euclidean distance The samples are divided into the clusters with the nearest mean vector, and the corresponding cluster mean vectors are recalculated from the divided clusters. The iteration is continued until the maximum number of iterations is reached or the mean vector no longer changes.

[0060] After clustering the sample data, subcategory sample data may be generated based on each cluster obtained after clustering, and the subcategory sample data may be used as samples in the sample space to train the multi-layer mapping network model.

[0061] Step S130, a multi-layer mapping network model construction step.

[0062] In this step, the subcategory sample data in the sample space is input into a multi-layer mapping network model created based on the principle of maximizing the inter-class distance and minimizing the intra-class distance. Starting from the first layer of the model, the model parameters including the expansion factor and the compression factor are calculated and updated layer by layer, and a filter is generated based on the updated model parameters for calculating the compression factor weight and weighting the result in the test phase, wherein the expansion factor is used to increase the inter-class distance of the overall sample space, and the compression factor is used to reduce the intra-class distance of each category of sample space.

[0063] Generally speaking, whether different categories can be effectively classified depends on the separability between feature components. For network flow data, due to the lack of expressiveness of its features, the separability of its feature distribution is often poor. In the embodiment of the present invention, by mapping complex structured data to a low-rank separable homogeneous space, the intra-class spacing of sample spaces of each category is reduced to improve the expressiveness of the features. This process of mapping a complex degenerate distributed data to a low-rank and highly expressive homogeneous space is the data compression process.

[0064] In the embodiment of the present invention, the process of building a multi-layer mapping network model is divided into three parts: first, the network flow data set of kN different behavior categories obtained after clustering is regarded as a sample set composed of kN different categories, and a multi-layer mapping network is constructed according to the principle of "maximizing the distance between classes and minimizing the distance within classes", and the parameters of each network layer of the model are initialized. and ( is the number of network layers, j is the category label). Secondly, according to the model input data from the first layer Start to calculate and update the parameters of each network layer of the model layer by layer. After completing the construction of each layer of the network, pass the training data through the current network to obtain the input of the next layer of the network. Finally, after completing the construction of all layers of the network, store the model parameters for subsequent testing and downstream tasks. This part belongs to the testing phase and will be described later. Here we first describe the initialization and parameter update of the model parameters.

[0065] The present invention uses the rate-distortion method to measure the size of the sample feature space. In the rate-distortion expression, n represents the number of features of the sample data, and m represents the total number of samples in the sample space Z. Under the set error rate, the average number of bits required to encode each sample data in the sample set Z is expressed as follows:

[0066]

[0067] Among them, R(Z,∈) represents the average number of bits required to encode each sample data in the sample set Z under the error rate ∈, and det() represents the square matrix The determinant of (Determinant); I is the unit matrix; Z * Denotes the adjoint matrix of the matrix Z.

[0068] With the help of the rate-distortion measurement method, the optimization objective of the mapping model can be constructed according to the principle of "maximizing the distance between classes and minimizing the distance within classes" as follows:

[0069]

[0070] Among them, ΔR represents the optimization target of the model. According to the model construction principle, the larger the optimization target, the better the behavior representation ability; Π j is a set of diagonal matrices used to filter sample data of a certain category (category label is j). Only when the sample data of category j is located at the position where 1 is located, the other positions are 0; tr j ) is the matrix Π j The trace represents the number of samples of category j. The mapping target consists of two parts. The first half of the formula R is used to measure the distance between classes (obtained from the rate-distortion expression), while the latter R c It is used to measure the intra-class distance. It calculates the rate distortion of different categories according to the rate distortion formula, and then weights them according to the proportion of different sample categories in the sample set.

[0071] Taking partial derivatives of input data, we get the following equation:

[0072]

[0073] in, Combined with the design principle of optimizing the target, the present invention will As input data The "expansion factor" is an n×n dimensional matrix, that is, by left-multiplying an n×n dimensional matrix To increase the inter-class spacing of the overall sample space, Indicates the number of network layers; As input data The "compression factor" of the j-th sample in is an n×n-dimensional matrix, that is, it is multiplied by an n×n matrix with dimension And multiply it by Π for filtering categories j To reduce the intra-class distance between samples of different categories. and Then, the layers of the multi-layer mapping model can be constructed.

[0074] In the embodiment of the present invention, the parameters of the initially constructed multi-layer mapping network model are and The parameters are initialized and the parameters can be updated by training the multi-layer mapping network model using training samples. And can be combined with Π j Calculate the value of each category The purpose of building a multi-layer mapping model can be achieved by continuously stacking mapping structures.

[0075] Figure 2 4 is an overall architecture diagram of a network flow space mapping model based on clustering compression in one embodiment of the present invention. Figure 3 It is a schematic diagram of the structure of the multi-layer mapping model. Figure 2 and Figure 3 As shown in the figure, after the traffic data is processed to obtain a numerical matrix composed of features, in the training phase, the obtained N categories of data matrix samples are clustered to obtain kN sub-category data samples. The kN sub-category data samples are used as a sample set composed of kN different categories and input into the initially constructed network flow space mapping model. Based on the clustering labels, the model parameters are calculated and updated layer by layer starting from the first layer of the model. and The output of each layer is used as the input of the next layer. The output of the last layer is an n×n dimensional homogeneous space (flow spectrum space) matrix that embodies the principle of "maximizing the distance between classes and minimizing the distance within classes", where n is the characteristic dimension of the sample data. More specifically, the output data of each layer (such as the lth layer) can be expressed as:

[0076]

[0077] Wherein, η is a learning rate, and its value may be, for example, between 100 and 200, but the present invention is not limited thereto; γ j It is calculated based on the following formula during the training phase:

[0078] During the testing phase, the input data of the multi-layer mapping network model is an n×m matrix, the expansion factor and compression factor are m×m matrices, and the output data is an n×n matrix, where n is the number of features of the sample data and m is the total number of samples in the sample space Z.

[0079] In the embodiment of the present invention, the model parameters and After the update is completed, the trained multi-layer mapping network model is obtained, and then the testing phase begins.

[0080] Step S140, a test sample acquisition step, is used to obtain a predetermined number of test samples in the form of a numerical matrix in the sample space.

[0081] This step is basically the same as step S110 and will not be described again here.

[0082] In the embodiment of the present invention, during the testing phase, the test samples are no longer subjected to clustering operations, but the test samples in the test sample set (sample space) are directly used as inputs of the multi-layer mapping network model.

[0083] Step S150, a testing step, inputs the obtained sample set into the multi-layer mapping network model trained in the training phase, and outputs a spatial mapping matrix.

[0084] When the model is used in the inference phase (test phase), the weight parameters of all categories are Multiply it with the sample data, and the result represents the compression factor In addition, in the test phase, since the categories are not clustered by the clustering step, the diagonal matrix Π used to screen the categories in the above formula (5) is j The value of is unknown, so the present invention uses the softmax function to calculate the compression factor weight parameter And weight the calculation results to construct a similar j Filters

[0085]

[0086] Among them, λ is a constant, which is set to 500 in the experiment, but the present invention is not limited to this, and can also be set to a larger or smaller number. This filter is used to represent the probability that the output data of the lth layer is filtered into the jth category. Combined with this filter, the part used to measure the intra-class distance in the inference stage of the model is transformed accordingly:

[0087]

[0088] Among them, therefore, the final output data for each layer (such as the lth layer) can be expressed as:

[0089]

[0090] Therefore, the present invention constructs a multi-layer mapping model in the network anomaly detection task to map the complex and degenerate original features into a homogeneous space (called flow spectrum space), in which different network behaviors are divided into several independent subspaces, so it has a higher representation ability. In addition, unlike the "black box" model of deep learning, the model used in the present invention is constructed by designing an optimization target forward, which makes the flow spectrum construction process highly interpretable. That is, the present invention improves the representation ability of the original network flow data for different network behaviors.

[0091] Based on the homogeneous flow spectrum obtained by the present invention, the present invention can better extract the intrinsic distribution of data and apply it to network flow characterization tasks. A data enhancement solution based on unsupervised learning is adopted, and unsupervised technology is introduced in the front of the model to mine the distribution characteristics between similar data. Different clustering results of similar network behaviors are regarded as different behavior categories to complete the subsequent flow spectrum construction task, which further improves the flow spectrum's ability to characterize network behaviors.

[0092] After the model is built, the test data is directly fed into the model to complete the mapping of the homogeneous space and obtain the mapping of the original data to the flow spectrum. After each layer of network mapping, the data performs better on the optimization target, that is, in the new homogeneous space, compared with the complex degenerate original features, the distribution differences between sample data according to different categories and the same category are more expressive. Based on these characteristics of the flow spectrum, subsequent downstream tasks can be completed, such as spectral line visualization analysis, flow spectrum separability research, etc.

[0093] The present invention also has certain robustness and portability. The flow spectrum theory constructed by data compression and classification principles can show considerable effectiveness in different scenarios. The model is applied to multiple different network flow data sets (CICIDS2017 and CICIDS2018) and the effect is analyzed from two perspectives: from the perspective of separability, the distance between classes and within classes is quantified by distance measurement indicators to determine whether the mapped flow spectrum data has a higher representation ability than the original traffic data; from the perspective of visual analysis, the similarity between and within classes of the sample set is visualized by means of heat maps, and the changes in separability between different network behaviors are analyzed from an intuitive perspective. The results show that the flow spectrum model has certain classification performance and has very good robustness and portability.

[0094] The effect diagram of quantitative analysis and visual analysis of measurement indicators is as follows:

[0095] Table quantification analysis of measurement indicators.

[0096]

[0097] The index μ in the table is a comprehensive metric μ that represents the data discrimination before and after mapping. For rate distortion, the larger the index, the farther the sample distance, so For cosine similarity, the larger the index, the closer the sample distance, so

[0098] from Figure 4 As can be seen in (a), the original traffic features only have a clear distinction between the data of the Generic (A4) and DoS (A5) categories, and it is difficult to find other data blocks along the way. Figure 4 In the heat map of the flow spectrum feature shown in (b), it is obvious that the color of some areas outside the diagonal line is weakened. Figure 4 The middle (a) figure has higher discrimination, which indicates the increase of independence between data. Figure 4 The overall color of (b) is Figure 4The middle (a) figure is shallower because the original design of the model objective function includes two parts: expanding the distance between classes and reducing the distance within classes. The feature distribution of general traffic data is relatively sparse (many feature values ​​are 0), which leads to the tendency to expand the distance between classes in the original feature space during the mapping process to achieve the purpose of maximizing the objective function. In the task of characterizing actual network behavior, we generally only care about how to distinguish different network behaviors, which requires discarding most of the similar or identical feature quantities in the network data and finding the differences between different behavior categories. From this perspective, the flow spectrum theory we designed meets the needs of actual applications.

[0099] In summary, in the task of improving the network flow representation capability, starting from the two principles of data compression and classification, an interpretable construction method is implemented through spatial mapping to map the originally complex and degenerate traffic data into a homogeneous space with stronger representation capability according to different network behavior categories.

[0100] In some embodiments of the present invention, the selection of model parameters (such as the number of clusters, the number of network layers, etc.) must be determined according to the distribution characteristics of the network flow data, that is, different model parameters are selected according to different network environments. Otherwise, the model training efficiency will be greatly affected, and the distribution characteristics of the data cannot be fully utilized, thereby failing to achieve ideal data characterization capabilities.

[0101] In some embodiments of the present invention, the dimension of the input features needs to be considered, because the model is built on the basis of the original data, and too large a data dimension will lead to an increase in the size of the model. For the original traffic packet data captured from the network environment, the feature dimension can be reduced by principal component analysis, PCA dimensionality reduction, etc.; for the feature set obtained by feature engineering, the feature dimension can be reduced by deleting unimportant features according to the feature importance in the classification task.

[0102] Corresponding to the above method, the present invention also provides a network flow space mapping characterization device based on clustering compression, which includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0103] The invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described above are implemented.

[0104] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted on a transmission medium or a communication link via a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, and the like. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0105] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0106] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0107] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A network flow space mapping representation method based on clustering compression, It is characterized in that The method includes a training phase and a testing phase; The training phase consists of the following steps: A training sample acquisition step is used to acquire training samples of different cyberspace behavior categories to form a sample set, wherein the training samples of different cyberspace behavior categories are numerical matrices of different behavior categories generated from features of a predetermined number of dimensions extracted from captured network flow data of different cyberspace behavior categories; Clustering step: clustering the training samples of each behavior category in the training sample set using a clustering algorithm, clustering the samples of each behavior category into a predetermined number of clusters, and generating subcategory sample data in the sample space based on each cluster of different behavior categories; The multi-layer mapping network model construction step is as follows: the sub-category sample data in the sample space is input into the multi-layer mapping network model created based on the principle of maximizing the inter-class distance and minimizing the intra-class distance, and the model parameters including the expansion factor and the compression factor are calculated and updated layer by layer starting from the first layer of the model, and a filter is generated based on the updated model parameters for calculating the compression factor weight and weighting the result in the test phase, wherein the expansion factor is used to increase the inter-class distance of the overall sample space, and the compression factor is used to reduce the intra-class distance of each category of sample space; The testing phase includes the following steps: A test sample acquisition step for obtaining a predetermined number of test samples in the form of a numerical matrix in a sample space; In the testing step, the obtained sample set is input into the multi-layer mapping network model trained in the training phase, and the spatial mapping matrix is ​​output.

2. The method according to claim 1, It is characterized in that The clustering algorithm is a K-means clustering algorithm; The input of the K-means clustering algorithm is the numerical matrix of a single behavior category and the number of clusters of the current category. The clustering of the numerical matrix of each behavior category obtained by using the clustering algorithm includes: Randomly select a predetermined number of clusters from the cluster sample set formed by the numerical matrix of each behavior category as the mean vector of each cluster in the current category; According to the distance between each sample of the current behavior category and each mean vector, the samples are divided into the cluster where the nearest mean vector is located; Recalculate the mean vector of the corresponding cluster based on the currently divided cluster; Clustering is iterated based on the recalculated mean vector until the maximum number of iterations is reached or the mean vector no longer changes.

3. The method according to claim 1, It is characterized in that The method further includes: outputting each subspace matrix data in the output flow spectrum space matrix to a downstream network flow characterization task.

4. The method according to claim 1, It is characterized in that The expansion factor is expressed as: The compression factor is expressed as: Among them, the matrix is the input data of the lth layer The expansion factor, is the input data of the lth layer The compression factor of the j-th class sample in , ∈ is the error rate, represents the input matrix of the lth layer in the multi-layer mapping network model, Representation Matrix The adjoint matrix, I is the identity matrix, n is the number of features of the sample data, and m represents the sample space The total number of samples in tr(Π j ) is the matrix Π j The trace of j is a set of diagonal matrices used to screen sample data of the jth category.

5. The method according to claim 4, It is characterized in that The filter is constructed as: in, is a filter, which indicates the probability that the output data of the lth layer is filtered into the jth category, and λ is a constant.

6. The method according to claim 5, It is characterized in that The output data of each layer in the trained multi-layer mapping network model satisfies the following formula: Among them, η is the learning rate, γ j It is calculated during the training phase based on the following formula: Among them, tr(Π j ) is the matrix Π j The trace of represents the number of samples of category j.

7. The method according to claim 1, It is characterized in that Before generating the numerical matrix corresponding to the behavior category, the method further includes: In the case where the captured network flow data of different cyberspace behavior categories are original traffic packet data captured from a network environment, reducing the feature dimension by principal component analysis or PCA dimensionality reduction; In the case where the captured network flow data of different cyberspace behavior categories is a feature set obtained by feature engineering, some features are deleted according to the feature importance in the classification task to reduce the feature dimension.

8. The method according to claim 1, It is characterized in that During the testing phase, the input data of the multi-layer mapping network model is an n×m matrix, the expansion factor and compression factor are an m×m matrix, and the output data is an n×n matrix, where n is the number of features of the sample data and m is the total number of samples in the sample space Z.

9. A network flow space mapping representation device based on clustering compression, comprising a processor and a memory, It is characterized in that The memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the device implements the steps of the method as claimed in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images