Distributed network data processing system and method based on cloud computing

Abstract

The invention relates to the technical field of data processing, and more specifically discloses a distributed network data processing system and method based on cloud computing, and the method comprises the steps: obtaining historical network intrusion detection data and real-time network data, extracting feature vectors, fusing the feature vectors into an intrusion behavior monitoring feature matrix, and carrying out the recognition of the intrusion behavior. And analyzing and judging the feature matrix through a classifier to obtain a classification result which is used for representing whether a network intrusion behavior exists or not. Therefore, efficient network intrusion detection can be realized, and the method has important application value in the field of network security.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a distributed network data processing system and method based on cloud computing. Background Art

[0002] Cloud computing is a network-based computing resource sharing and allocation model that enables flexible resource allocation, rapid expansion, and low cost. Distributed computing is a method of breaking down computing tasks into multiple subtasks and executing them in parallel on multiple computing nodes. These two technologies can work together to achieve efficient processing of big data analysis. As a key technology supporting big data, cloud computing technology can achieve concurrent processing of huge data services through a distributed computing architecture, thereby realizing data-based services in many traditional business areas (such as industrial manufacturing, video surveillance, game operations, smart city services, telemedicine services, and digital financial services, etc.).

[0003] Distributed network data processing is a method of processing large-scale network data using distributed computing and network technology. It can be applied to multiple fields and industries. Distributed network data processing can be used in the field of network security, such as intrusion detection, threat intelligence analysis, malware detection, etc. With the rapid development of big data and artificial intelligence technology, the scale of network users continues to expand, introducing more network traffic and network security issues. Intrusion detection technology is the result of the continuous development of network technology. By monitoring and analyzing the information and behavior of computer networks and judging the existence of abnormal information, network attacks can be effectively detected.

[0004] Therefore, a distributed network data processing system and method based on cloud computing is desired. Summary of the invention

[0005] In order to solve the above technical problems, the present application is proposed. The embodiments of the present application provide a distributed network data processing system and method based on cloud computing, which realizes efficient network intrusion detection by extracting features and fusing data and using classifiers.

[0006] Accordingly, according to one aspect of the present application, a distributed network data processing system based on cloud computing is provided, comprising:

[0007] A network data acquisition module, used to obtain historical network intrusion detection data and real-time network data, wherein the historical network intrusion detection data includes unlabeled sample data, labeled sample data and labels corresponding to the labeled sample data;

[0008] A network data extraction module, used to extract a network intrusion detection associated feature vector from the historical network intrusion detection data, and encode the real-time network data to obtain an optimized real-time network data feature vector;

[0009] A network data fusion module, used for fusing the network intrusion detection associated feature vector and the optimized real-time network data feature vector to obtain an intrusion behavior monitoring feature matrix;

[0010] The network data analysis module is used to pass the intrusion behavior monitoring feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether there is network intrusion behavior.

[0011] According to another aspect of the present application, a distributed network data processing method based on cloud computing is provided, which includes:

[0012] Acquire historical network intrusion detection data and real-time network data, wherein the historical network intrusion detection data includes unlabeled sample data, labeled sample data, and labels corresponding to the labeled sample data;

[0013] Extracting a network intrusion detection associated feature vector from the historical network intrusion detection data, and encoding the real-time network data to obtain an optimized real-time network data feature vector;

[0014] The network intrusion detection associated feature vector and the optimized real-time network data feature vector are merged to obtain an intrusion behavior monitoring feature matrix;

[0015] The intrusion behavior monitoring feature matrix is ​​passed through a classifier to obtain a classification result, and the classification result is used to indicate whether there is network intrusion behavior.

[0016] Compared with the prior art, the present application provides a distributed network data processing system and method based on cloud computing, which obtains historical network intrusion detection data and real-time network data, extracts feature vectors, fuses them into an intrusion behavior monitoring feature matrix, analyzes and judges the feature matrix through a classifier, and obtains a classification result to indicate whether there is network intrusion behavior. This can achieve efficient network intrusion detection and has important application value in the field of network security. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 It is a block diagram of a distributed network data processing system based on cloud computing according to an embodiment of the present application.

[0019] Figure 2 It is a block diagram of a network data extraction module in a distributed network data processing system based on cloud computing according to an embodiment of the present application.

[0020] Figure 3 It is a block diagram of a network data conversion vector unit in a distributed network data processing system based on cloud computing according to an embodiment of the present application.

[0021] Figure 4 It is a block diagram of a one-dimensional arrangement unit of historical network data in a distributed network data processing system based on cloud computing according to an embodiment of the present application.

[0022] Figure 5 It is a block diagram of a historical network data two-dimensional arrangement unit in a distributed network data processing system based on cloud computing according to an embodiment of the present application.

[0023] Figure 6 The present invention is a flowchart of a distributed network data processing method based on cloud computing according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0025] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0026] In addition, in order to better illustrate the present application, numerous specific details are provided in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0027] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0028] Figure 1 FIG. 1 is a block diagram of a distributed network data processing system based on cloud computing according to an embodiment of the present application. Figure 1 As shown, according to the distributed network data processing system 100 based on cloud computing according to the embodiment of the present application, it includes: a network data acquisition module 110, which is used to obtain historical network intrusion detection data and real-time network data, wherein the historical network intrusion detection data includes unlabeled sample data, labeled sample data and labels corresponding to the labeled sample data; a network data extraction module 120, which is used to extract network intrusion detection associated feature vectors from the historical network intrusion detection data, and encode the real-time network data to obtain an optimized real-time network data feature vector; a network data fusion module 130, which is used to fuse the network intrusion detection associated feature vector and the optimized real-time network data feature vector to obtain an intrusion behavior monitoring feature matrix; a network data analysis module 140, which is used to pass the intrusion behavior monitoring feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether there is network intrusion behavior.

[0029] In an embodiment of the present application, the network data acquisition module 110 is used to obtain historical network intrusion detection data and real-time network data, wherein the historical network intrusion detection data includes unlabeled sample data, labeled sample data, and labels corresponding to the labeled sample data. It should be understood that the historical network intrusion detection data includes unlabeled sample data and labeled sample data, wherein the labeled sample data is data that has been marked as normal or intrusion and has corresponding labels. By using these labeled sample data, a classifier can be trained to learn the characteristics of intrusion behavior, so that real-time network data can be classified and judged. At the same time, unlabeled sample data can be used for further feature extraction and model training. Real-time network data is obtained in order to monitor and detect intrusion behavior in the network in real time, encode it into a feature vector, and merge it with historical data to more accurately determine whether there is network intrusion behavior. Integrating historical data and real-time data can improve the accuracy and reliability of the network intrusion detection system. Specifically, a large amount of historical network intrusion detection data can be obtained by collecting and analyzing log files generated by security devices (such as firewalls, intrusion detection systems). Using network traffic monitoring tools, such as network traffic analyzers or packet capture tools, real-time network data traffic can be captured and recorded. For historical network intrusion detection data, unlabeled sample data can be obtained by extracting unlabeled data records from historical logs. Labeled sample data can be labeled by known intrusion events or expert labeled data. The label corresponding to the labeled sample data indicates whether each data sample is normal or intrusion behavior.

[0030] In an embodiment of the present application, the network data extraction module 120 is used to extract the network intrusion detection associated feature vector from the historical network intrusion detection data, and encode the real-time network data to obtain an optimized real-time network data feature vector. It should be understood that by extracting these feature vectors, the historical network intrusion detection data can be analyzed and modeled to train an intrusion detection model. The model can learn the feature patterns corresponding to different types of intrusion behaviors, so that similar intrusion behaviors can be identified and detected in real-time network data. For real-time network data, the purpose of encoding it as a feature vector is to compare and match the real-time data with the historical data. The feature vector of the real-time network data can be obtained by the same feature extraction method. Then, by comparing the feature vector of the real-time data with the feature vector of the historical data, it can be determined whether the real-time data matches the known intrusion feature pattern, thereby determining whether there is an intrusion behavior. The process of encoding real-time network data as a feature vector can be carried out in real time so as to detect and respond to network intrusion in a timely manner. This method of real-time feature extraction and encoding can help build an efficient network intrusion detection system and improve the ability to identify and respond to network security threats.

[0031] Specifically, in one embodiment of the present application, Figure 2 FIG. 1 is a block diagram of a network data extraction module in a distributed network data processing system based on cloud computing according to an embodiment of the present application. Figure 2 As shown, in the above-mentioned distributed network data processing system 100 based on cloud computing, the network data extraction module 120 includes: a network data conversion vector unit 121, which is used to convert the historical network intrusion detection data into a historical network intrusion detection embedding vector and then obtain a plurality of historical network intrusion detection data feature vectors through a first context encoder based on a converter; a historical network data one-dimensional arrangement unit 122, which is used to extract features from the plurality of historical network intrusion detection data feature vectors after one-dimensional arrangement to obtain a multi-scale network intrusion detection feature vector; a historical network data two-dimensional arrangement unit 123, which is used to extract features from the plurality of historical network intrusion detection data feature vectors The vector is arranged in two dimensions and then convolutionally encoded to obtain a local feature vector for network intrusion detection; a fusion network intrusion detection data unit 124 is used to fuse the multi-scale network intrusion detection feature vector and the local feature vector for network intrusion detection to obtain the network intrusion detection associated feature vector; a real-time network data encoding unit 125 is used to convolutionally encode the real-time network data to obtain a real-time network data feature vector; a real-time network data feature optimization unit 126 is used to perform viewpoint metric perception optimization based on quantification of multi-angle feature relationships on the real-time network data feature vector based on the network intrusion detection associated feature vector to obtain an optimized real-time network data feature vector.

[0032] Accordingly, in a specific example of the present application, the network data conversion vector unit 121 is used to convert the historical network intrusion detection data into a historical network intrusion detection embedding vector and then obtain multiple historical network intrusion detection data feature vectors through a first context encoder based on a converter. It should be understood that the high-level semantic information and representation in the extracted data are taken into account. By using a converter model (such as Transformer) for encoding, the contextual relationship and semantic information in the historical network intrusion detection data can be captured. The converter model is a neural network model based on a self-attention mechanism, which has a strong modeling ability and can effectively learn dependencies in sequence data. The converter model converts the input historical network intrusion detection data into an embedding vector of fixed length through multiple layers of self-attention and feedforward neural network layers. These embedding vectors contain the semantic information and representation of the historical data. Then, the embedding vector can be further processed by the first context encoder based on the converter to generate multiple historical network intrusion detection data feature vectors. Converting the original data into an embedding vector and a feature vector can represent the data as a vector with a lower dimension, reducing the complexity and storage requirements of the data. Through the converter model and the context encoder, the semantic information of the data can be captured and the important features and patterns in the data can be extracted. Embedding vectors and feature vectors can better represent historical network intrusion detection data, making subsequent analysis and modeling tasks more efficient and accurate.

[0033] further, Figure 3 FIG. 1 is a block diagram of a network data conversion vector unit in a distributed network data processing system based on cloud computing according to an embodiment of the present application. Figure 3 As shown, in the network data extraction module 120 of the above-mentioned distributed network data processing system 100 based on cloud computing, the network data conversion vector unit 121 includes: an embedding conversion subunit 1211, which is used to pass the historical network intrusion detection data through an embedding layer to convert the historical network intrusion detection data of each dimension in the historical network intrusion detection data into a historical network intrusion detection data embedding vector to obtain a sequence of historical network intrusion detection data embedding vectors, wherein the embedding layer uses a learnable embedding matrix to embed the historical network intrusion detection data of each dimension; an encoding subunit 1212, which is used to pass the sequence of the historical network intrusion detection data embedding vectors through the first context encoder based on the converter to obtain the multiple historical network intrusion detection data feature vectors.

[0034] Specifically, the embedding conversion subunit 1211 is used to convert the historical network intrusion detection data of each dimension in the historical network intrusion detection data into a historical network intrusion detection data embedding vector to obtain a sequence of historical network intrusion detection data embedding vectors, wherein the embedding layer uses a learnable embedding matrix to embed the historical network intrusion detection data of each dimension. It should be understood that in order to convert the original data into a continuous, fixed-dimensional vector representation for subsequent analysis and modeling. The embedding layer is a learnable layer that embeds and encodes the historical network intrusion detection data of each dimension by using a learnable embedding matrix. The embedding matrix is ​​a parameter matrix in which each column corresponds to a dimension of historical network intrusion detection data. By multiplying the embedding matrix, the historical network intrusion detection data of each dimension is mapped to a low-dimensional embedding space to form a historical network intrusion detection data embedding vector. After the historical network intrusion detection data of each dimension is converted into an embedding vector, the data of all dimensions have the same dimension. This can eliminate the problems caused by dimensional differences, making the data easier to compare and analyze in subsequent processing. In the process of generating the embedding vector, the parameters in the embedding matrix are learnable, and the semantic information of the data can be learned through training. The embedding vector can better represent the features and patterns in the historical network intrusion detection data, thereby improving the accuracy of subsequent tasks. Embedding vectors are continuous vector representations, and they can be input into various machine learning algorithms for further analysis and modeling. Continuous representations help capture the similarities and correlations between data. By converting historical network intrusion detection data into a sequence of historical network intrusion detection data embedding vectors, a more compact and informative data representation can be provided, providing better input for subsequent data analysis and modeling tasks. This method of embedding coding can help extract important features and patterns in the data and improve the accuracy and effectiveness of network intrusion detection.

[0035] Further, the encoding subunit 1212 is used to pass the sequence of the historical network intrusion detection data embedding vectors through the first context encoder based on the converter to obtain the multiple historical network intrusion detection data feature vectors. It should be understood that the first context encoder based on the converter is a neural network model for sequence modeling, which encodes the input sequence through a multi-layer self-attention mechanism and a feedforward neural network layer. It can capture the dependencies and context information between different positions in the sequence, thereby generating a feature vector with richer semantic representation. The converter model can model the contextual relationship in the sequence, and it can capture the dependencies between different positions in the sequence through the self-attention mechanism. This can better understand the semantic information and context association in the sequence, thereby generating a more accurate feature vector representation. The first context encoder based on the converter can extract the semantic information in the sequence and encode it into a feature vector. These feature vectors can better represent the important features and patterns in the historical network intrusion detection data. Since the first context encoder based on the converter is a sequence modeling model, it can process the entire sequence instead of a single data point. In this way, the global information in the sequence can be taken into account during the encoding process, further improving the quality and expression ability of the feature vector.

[0036] Accordingly, in a specific example of the present application, the historical network data one-dimensional arrangement unit 122 is used to extract features from the multiple historical network intrusion detection data feature vectors after one-dimensional arrangement to obtain a multi-scale network intrusion detection feature vector. It should be understood that network intrusion detection data usually has features of different time scales and spatial scales. By arranging multiple historical network intrusion detection data feature vectors in one dimension, feature information of different scales can be integrated together. This can provide a more comprehensive and richer feature representation, thereby enhancing the detection capability of the network intrusion detection system for intrusion behaviors of various scales. After arranging multiple feature vectors in one dimension, they can be fused into a longer feature vector. This feature fusion can capture the mutual relationship and dependency between different features, thereby providing a more expressive feature representation. Through one-dimensional arrangement, multiple feature vectors can be merged into a longer vector. This can reduce the dimension of the feature, reduce the sparsity of the feature vector, and improve the compactness and effectiveness of the feature representation. By obtaining a multi-scale network intrusion detection feature vector, further processing and analysis can be performed on features of different scales in the subsequent feature extraction and analysis process. This helps the system to detect and identify intrusion behaviors of different scales in a more fine-grained manner.

[0037] further, Figure 4 FIG. 1 is a block diagram of a one-dimensional arrangement unit of historical network data in a distributed network data processing system based on cloud computing according to an embodiment of the present application. Figure 4As shown, in the network data extraction module 120 of the above-mentioned distributed network data processing system 100 based on cloud computing, the historical network data one-dimensional arrangement unit 122 includes: an arrangement vector subunit 1221, which is used to arrange the multiple historical network intrusion detection data feature vectors into the one-dimensional feature vector of the network intrusion detection data; a multi-scale extraction subunit 1222, which is used to pass the one-dimensional feature vector of the network intrusion detection data through a multi-scale neighborhood feature extraction module to obtain the multi-scale network intrusion detection feature vector.

[0038] Specifically, the arrangement vector subunit 1221 is used to arrange the multiple historical network intrusion detection data feature vectors into the one-dimensional feature vector of the network intrusion detection data. It should be understood that by arranging multiple historical network intrusion detection data feature vectors into one-dimensional feature vectors, they can be unified into the same representation form. This can simplify the data processing and analysis process, so that the features of different data have the same dimension and structure, which is convenient for subsequent feature extraction and model building. By arranging multiple historical network intrusion detection data feature vectors into one-dimensional feature vectors, the form of input data can be simplified. This helps to reduce storage and transmission overhead when processing large-scale data sets in practical applications.

[0039] Specifically, the multi-scale extraction subunit 1222 is used to pass the one-dimensional feature vector of the network intrusion detection data through a multi-scale neighborhood feature extraction module to obtain the multi-scale network intrusion detection feature vector. It should be understood that network intrusion behaviors usually exist at different time scales and spatial scales. By using a multi-scale neighborhood feature extraction module, features can be extracted from neighborhood information of different scales. In this way, the intrusion behavior features at different scales can be captured, thereby improving the detection capability of the network intrusion detection system for intrusion behaviors of various scales. Through multi-scale neighborhood feature extraction, context information around each feature point can be obtained. These context information can provide more comprehensive background knowledge, which helps to better understand and judge abnormal behaviors in network traffic. By fusing multi-scale context information, the recognition and classification capabilities of the intrusion detection system for complex intrusion behaviors can be improved. The multi-scale neighborhood feature extraction module can generate richer and more discriminative feature vectors by fusing feature information of different scales. This feature fusion and enhancement can improve the accuracy and robustness of the network intrusion detection system.

[0040] Further, the multi-scale extraction subunit includes: a first scale encoding secondary subunit, used to use the first convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the one-dimensional feature vector of the network intrusion detection data with a one-dimensional convolution kernel with a first scale to obtain a first-scale network intrusion detection data associated feature vector; a second scale encoding secondary subunit, used to use the second convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the one-dimensional feature vector of the network intrusion detection data with a one-dimensional convolution kernel with a second scale to obtain a second-scale network intrusion detection data associated feature vector; a multi-scale cascade secondary subunit, used to cascade the first-scale network intrusion detection data associated feature vector and the second-scale network intrusion detection data associated feature vector to obtain the multi-scale network intrusion detection feature vector.

[0041] Accordingly, in a specific example of the present application, the historical network data two-dimensional arrangement unit 123 is used to perform convolution coding on the multiple historical network intrusion detection data feature vectors after two-dimensional arrangement to obtain a local feature vector for network intrusion detection. It should be understood that network intrusion detection data usually has spatial correlation, that is, there is correlation between data points at adjacent positions. By arranging multiple historical network intrusion detection data feature vectors in two dimensions, they can be regarded as a two-dimensional image, so as to use the convolution operation to model the spatial correlation of the data. The convolution operation can capture the local structure and pattern in the data, which helps to extract discriminative local features. Convolution coding can extract local features from the input data through the sliding window operation of the convolution kernel. These local features can capture the details and local structure of the data, which helps to improve the accuracy of network intrusion detection. In addition, convolution coding can also reduce dimensionality through pooling operations, reduce the dimension of features, and improve computational efficiency. Convolution coding uses shared weights in the feature extraction process to reduce the number of parameters that need to be learned. This can reduce the complexity of the model and help avoid overfitting problems. Shared weights can share feature extraction information at different positions and improve the generalization ability of the model. The local feature vectors obtained by convolutional coding can better represent the local information of network intrusion detection data. These local features can capture the intrusion behavior patterns at different locations and provide more discriminative representations. The local feature vectors can be used in subsequent feature fusion, classification, and decision-making processes to improve the performance of network intrusion detection systems.

[0042] further, Figure 5 The figure shows a block diagram of a historical network data two-dimensional arrangement unit in a distributed network data processing system based on cloud computing according to an embodiment of the present application. Figure 5As shown, in the network data extraction module 120 of the above-mentioned distributed network data processing system 100 based on cloud computing, the historical network data two-dimensional arrangement unit 123 includes: an arrangement matrix subunit 1231, used to arrange the multiple historical network intrusion detection data feature vectors in two dimensions into a network intrusion detection data feature matrix; a convolutional coding subunit 1232, used to pass the network intrusion detection data feature matrix through a historical network intrusion detection data feature extractor based on a first convolutional neural network model to obtain the network intrusion detection local feature vector.

[0043] Specifically, the arrangement matrix subunit 1231 is used to arrange the plurality of historical network intrusion detection data feature vectors in two dimensions into a network intrusion detection data feature matrix. It should be understood that after the feature vectors are arranged into a feature matrix, the properties of the matrix can be used for related analysis and processing. For example, matrix operations and linear algebra methods can be used to perform operations such as feature extraction, dimensionality reduction, and clustering. These methods can effectively process matrix-structured data and improve the effect and performance of network intrusion detection. After the feature vectors are arranged into a feature matrix, the local neighborhood information of the matrix can be used to obtain context information. For example, convolution operations can be used to extract local features, or recurrent neural networks (RNNs) can be used to model the dependencies of time series. In this way, the context information of the data can be better utilized to improve the accuracy and robustness of network intrusion detection. The two-dimensional arrangement form of the feature matrix can more conveniently visualize and analyze data. The feature matrix can be visualized as a heat map or image to intuitively observe and analyze the patterns and trends of the data. This can help network security experts better understand network intrusion behavior and discover anomalies and threats.

[0044] Specifically, the convolutional coding subunit 1232 is used to pass the network intrusion detection data feature matrix through a historical network intrusion detection data feature extractor based on a first convolutional neural network model to obtain the network intrusion detection local feature vector. It should be understood that the convolutional neural network can effectively extract local features in the image data through convolution operations and pooling operations. Since the network intrusion detection data feature matrix can be regarded as a two-dimensional image, by inputting it into the convolutional neural network, the convolutional layer can be used to extract local features in the data. These local features can capture the details and local structure of the data, which contributes to the accuracy and robustness of network intrusion detection. The convolutional layer of the convolutional neural network uses a shared weight method to reduce the number of parameters that need to be learned. This can reduce the complexity of the model and help avoid overfitting problems. Shared weights can share feature extraction information at different positions and improve the generalization ability of the model. The convolutional neural network can automatically learn feature representation without manually designing a feature extractor. By training the network, the convolutional layer can automatically learn a feature extraction mode suitable for the network intrusion detection task, thereby improving the performance of network intrusion detection. Convolutional neural networks are usually composed of multiple convolutional layers and pooling layers, which can perform multi-level feature extraction. The lower convolutional layers can extract simpler local features, while the higher convolutional layers can extract more abstract and complex features. In this way, higher-level semantic information in the data can be gradually extracted, which is helpful for the discrimination and classification of network intrusion detection.

[0045] Specifically, the convolutional encoding subunit includes: a convolution secondary subunit, used to perform convolution processing on the input data to obtain a convolution feature map; a pooling secondary subunit, used to perform mean pooling of each feature matrix along the channel dimension of the convolution feature map to obtain a pooling feature map; an activation secondary subunit, used to perform nonlinear activation on the pooling feature map to obtain an activation feature map; wherein the output of the last layer of the first convolutional neural network model is the network intrusion detection local feature vector, and the input of the first layer of the first convolutional neural network model is the network intrusion detection data feature matrix.

[0046] Accordingly, in a specific example of the present application, the fusion network intrusion detection data unit 124 is used to fuse the multi-scale network intrusion detection feature vector and the network intrusion detection local feature vector to obtain the network intrusion detection associated feature vector. It should be understood that the intrusion behavior in network intrusion detection may have different time and space scales. By using a multi-scale network intrusion detection feature extractor, feature information can be extracted from different scales. In this way, the characteristic patterns of intrusion behaviors at different scales can be captured, and the coverage and accuracy of the network intrusion detection system can be improved. The multi-scale network intrusion detection feature vector and the network intrusion detection local feature vector represent feature information at different levels respectively. Fusion of these two feature vectors can combine information at different levels to form a richer and more comprehensive feature representation. In this way, more comprehensive feature information can be provided, which helps the network intrusion detection system to better distinguish normal traffic from malicious behavior. The intrusion behavior in network intrusion detection may be diverse and complex, and a single feature representation may not be able to fully capture this diversity. By fusing multi-scale feature vectors and local feature vectors, the diversity and robustness of features can be increased. In this way, the detection capability of the network intrusion detection system for unknown intrusion behaviors can be improved, and false positives and false negatives can be reduced. The fusion of multi-scale network intrusion detection feature vectors and network intrusion detection local feature vectors can generate a network intrusion detection correlation feature vector. This correlation feature vector can be used as a comprehensive decision-making basis for subsequent intrusion behavior classification and discrimination. By integrating feature information at different scales and levels, a more reliable and accurate decision-making basis can be provided, improving the performance of the network intrusion detection system.

[0047] Accordingly, in a specific example of the present application, the real-time network data encoding unit 125 is used to perform convolution encoding on the real-time network data to obtain a real-time network data feature vector. Specifically, the real-time network data is segmented and then passed through a second context encoder based on a converter to obtain the real-time network data feature vector. It should be understood that the text information in the real-time network data contains important semantic content, and the second context encoder based on the converter can effectively extract the semantic information of the text data. Through the self-attention mechanism, the converter model can automatically learn the association and context information between words, thereby capturing the semantic meaning in the text data. In this way, the text data can be converted into a feature vector with rich semantic information. The words in the text data usually need to be combined with the context to accurately understand their meaning. The second context encoder based on the converter can model the entire text sequence and enrich the representation of the words using context information. By considering the global contextual relationship, the meaning and context in the text data can be better understood, thereby generating a more accurate and useful feature representation. The text length in the real-time network data may not be fixed, and the second context encoder based on the converter can adapt to the input sequence of variable length. The converter model uses a self-attention mechanism to encode the input sequence, and does not require a fixed-length input. This makes it easy to process text data of different lengths and retains the complete information of the input sequence. The transformer-based second context encoder is usually initialized using a pre-trained model, such as BERT, GPT, etc. These pre-trained models are pre-trained on large-scale text data and have rich language knowledge and representation capabilities. By using these pre-trained models, better feature representation and semantic understanding capabilities can be obtained.

[0048] In particular, in the real-time network data feature optimization unit 126, it is considered that the real-time network data is the latest data, reflecting the current state of network activity; while the historical network intrusion detection data is based on past data for learning and feature extraction. This difference in timeliness may lead to inconsistency between the two in reflecting the network state. The historical network intrusion detection data feature vector extracts features through a first context encoder based on a converter and a multi-scale neighborhood feature extraction module, focusing on the learning and pattern recognition of historical data; while the real-time network data feature vector extracts features through a second context encoder, which may focus more on the rapid processing and analysis of real-time data. When fusing the network intrusion detection associated feature vector and the real-time network data feature vector, if the fusion strategy does not fully consider the time sensitivity and feature complementarity of the two, it may lead to the lack of a posteriori-priori causal association. In order to solve this problem, in the technical solution of the present application, based on the network intrusion detection associated feature vector, the real-time network data feature vector is optimized based on the viewpoint metric perception optimization of the multi-angle feature relationship quantification to obtain the optimized real-time network data feature vector.

[0049] Wherein, based on the network intrusion detection associated feature vector, the real-time network data feature vector is subjected to viewpoint metric perception optimization based on the quantification of the relationship between multi-angle features to obtain the optimized real-time network data feature vector, including: calculating the position difference between the real-time network data feature vector and the network intrusion detection associated feature vector to obtain a first viewpoint metric representation vector; calculating the position point multiplication between the real-time network data feature vector and the network intrusion detection associated feature vector to obtain a second viewpoint metric representation vector; calculating the position sum between the real-time network data feature vector and the network intrusion detection associated feature vector to obtain a third viewpoint metric representation vector. point metric representation vector; calculate the gradient amplitude of each position in the first viewpoint metric representation vector, the second viewpoint metric representation vector and the third viewpoint metric representation vector to obtain a first viewpoint metric perception vector, a second viewpoint metric perception vector and a third viewpoint metric perception vector; input the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector into a reverse learning optimization module to obtain a reverse multi-viewpoint perception optimization representation vector; calculate the position point multiplication between the reverse multi-viewpoint perception optimization representation vector and the real-time network data feature vector to obtain the optimized real-time network data feature vector.

[0050] The reverse learning optimization module is expressed as follows:

[0051]

[0052] in, and represents the eigenvalues ​​of the i-th position of the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, respectively, and μ represents the three eigenvalues ​​of the corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector , σ is the standard deviation of the three eigenvalues ​​at corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, represents the maximum value of the three eigenvalues ​​at the corresponding positions in the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector, ReLU represents a linear rectification function, softmax represents a normalized exponential function, and v ri Represents the eigenvalue of the i-th position of the inverse multi-view perceptual optimization representation vector.

[0053] Specifically, in the technical solution of the present application, based on the network intrusion detection associated feature vector, the real-time network data feature vector is optimized based on the viewpoint metric perception optimization based on the quantification of the relationship between multi-angle features, and the process includes first calculating the position difference, point multiplication and sum between the real-time network data feature vector and the network intrusion detection associated feature vector through multi-view analysis (Multi-View Analysis) to obtain three viewpoint metric representation vectors, and quantify the relationship between features from different angles. Then, the gradient amplitude of each position in these viewpoint metric representation vectors is calculated through backward error perception to identify key change points to provide a basis for feature optimization.

[0054] Then, the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector are input into a reverse learning optimization module, wherein the reverse learning optimization module captures the significant comprehensive ability factor of the feature importance perception (Feature Importance Perception) of the feature sub-dimension distribution of the corresponding positions in the first to third viewpoint metric perception vectors by adopting a squeeze-incentive mechanism composed of -functions. Then, the position point multiplication between the reverse multi-viewpoint perception optimization representation vector and the real-time network data feature vector is calculated to perform reverse optimization perception (Backward Optimization Perception) on the real-time network data feature vector to adjust the real-time network data feature vector so as to better associate with the network intrusion detection associated feature vector, thereby enhancing the causal relationship between the real-time network data feature vector and the network intrusion detection associated feature vector.

[0055] In an embodiment of the present application, the network data fusion module 130 is used to fuse the network intrusion detection associated feature vector and the optimized real-time network data feature vector to obtain an intrusion behavior monitoring feature matrix. It should be understood that the network intrusion detection associated feature vector provides a deep understanding of known attack patterns and can help identify traditional attack methods, while the optimized real-time network data feature vector can capture the current network status and emerging attack methods. By fusing the network intrusion detection associated feature vector and the optimized real-time network data feature vector, the intrusion detection system can more comprehensively analyze network traffic and identify potential threats.

[0056] In the embodiment of the present application, the network data analysis module 140 is used to pass the intrusion behavior monitoring feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether there is a network intrusion behavior. It should be understood that the intrusion behavior monitoring feature matrix is ​​the result of feature extraction and representation of network traffic data. By using a classifier, these features can be automatically classified to determine whether the network traffic belongs to an intrusion behavior. Compared with manual analysis and judgment, the use of a classifier can realize automated intrusion behavior monitoring and improve processing efficiency and accuracy. Intrusion behaviors usually have complex and diverse patterns, and different types of intrusion behaviors may have different characteristic manifestations. Classifiers can learn and identify these complex patterns to classify different types of intrusion behaviors. By training classifiers, a large amount of sample data and feature information can be used to improve the ability to identify intrusion behaviors. The classifier can process the intrusion behavior monitoring feature matrix in real time and quickly determine whether there is an intrusion behavior in the network traffic. In this way, intrusion behaviors can be discovered and responded to in a timely manner, reducing the losses caused by intrusions. Through real-time monitoring and classification, timely protection and response to network security can be achieved.

[0057] Accordingly, in one embodiment of the present application, the network data analysis module includes: a matrix expansion unit, used to expand the intrusion behavior monitoring feature matrix into a classification feature vector according to a row vector or a column vector; a fully connected encoding unit, used to use the fully connected layer of the classifier to fully connect the classification feature vector to obtain a fully connected encoded feature vector; a probability acquisition unit, used to pass the fully connected encoded feature vector through the Softmax classification function of the classifier to obtain a first probability attributable to the existence of network intrusion behavior and a second probability attributable to the absence of network intrusion behavior; a classification result determination unit, used to determine the classification result based on a comparison between the first probability and the second probability.

[0058] In summary, the cloud computing-based distributed network data processing system and method described in the embodiment of the present application obtains historical network intrusion detection data and real-time network data, extracts feature vectors, fuses them into an intrusion behavior monitoring feature matrix, analyzes and judges the feature matrix through a classifier, and obtains a classification result to indicate whether there is network intrusion behavior. This can achieve efficient network intrusion detection and has important application value in the field of network security.

[0059] As described above, the distributed network data processing system 100 based on cloud computing according to the embodiment of the present application can be implemented in various terminal devices, such as a server of a distributed network data processing system based on cloud computing. In one example, the distributed network data processing system 100 based on cloud computing can be integrated into the terminal device as a software module and / or a hardware module. For example, the distributed network data processing system 100 based on cloud computing can be a software module in the operating system of the terminal device, or can be an application developed for the terminal device; of course, the distributed network data processing system 100 based on cloud computing can also be one of the many hardware modules of the terminal device.

[0060] Alternatively, in another example, the cloud computing-based distributed network data processing system 100 and the terminal device may also be separate devices, and the cloud computing-based distributed network data processing system 100 may be connected to the terminal device via a wired and / or wireless network and transmit interactive information in accordance with an agreed data format.

[0061] Figure 6 FIG. 1 is a flow chart of a distributed network data processing method based on cloud computing according to an embodiment of the present application. Figure 6 As shown, according to the distributed network data processing method based on cloud computing according to the embodiment of the present application, the steps include: S110, obtaining historical network intrusion detection data and real-time network data, wherein the historical network intrusion detection data includes unlabeled sample data, labeled sample data and labels corresponding to the labeled sample data; S120, extracting network intrusion detection associated feature vectors from the historical network intrusion detection data, and encoding the real-time network data to obtain an optimized real-time network data feature vector; S130, fusing the network intrusion detection associated feature vector and the optimized real-time network data feature vector to obtain an intrusion behavior monitoring feature matrix; S140, passing the intrusion behavior monitoring feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether there is network intrusion behavior.

[0062] Here, those skilled in the art can understand that the specific operations of each step in the above-mentioned distributed network data processing method based on cloud computing have been referred to above. Figures 1 to 5 The description of the distributed network data processing system based on cloud computing has been described in detail, and therefore, its repeated description will be omitted.

[0063] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0064] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0065] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0066] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0067] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.

[0068] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit of the technical solution of the present invention.

Claims

1. A distributed network data processing system based on cloud computing, characterized in that: include: A network data acquisition module, used to obtain historical network intrusion detection data and real-time network data, wherein the historical network intrusion detection data includes unlabeled sample data, labeled sample data and labels corresponding to the labeled sample data; A network data extraction module, used to extract a network intrusion detection associated feature vector from the historical network intrusion detection data, and encode the real-time network data to obtain an optimized real-time network data feature vector; A network data fusion module, used for fusing the network intrusion detection associated feature vector and the optimized real-time network data feature vector to obtain an intrusion behavior monitoring feature matrix; The network data analysis module is used to pass the intrusion behavior monitoring feature matrix through a classifier to obtain a classification result, and the classification result is used to indicate whether there is network intrusion behavior.

2. The distributed network data processing system based on cloud computing according to claim 1, characterized in that: The network data extraction module comprises: A network data conversion vector unit, used to convert the historical network intrusion detection data into a historical network intrusion detection embedding vector and then pass it through a first context encoder based on a converter to obtain a plurality of historical network intrusion detection data feature vectors; A historical network data one-dimensional arrangement unit, used for performing feature extraction on the plurality of historical network intrusion detection data feature vectors after one-dimensional arrangement to obtain a multi-scale network intrusion detection feature vector; A historical network data two-dimensional arrangement unit, used for performing convolution coding on the plurality of historical network intrusion detection data feature vectors after two-dimensional arrangement to obtain a network intrusion detection local feature vector; A fusion network intrusion detection data unit, used for fusing the multi-scale network intrusion detection feature vector and the network intrusion detection local feature vector to obtain the network intrusion detection associated feature vector; A real-time network data encoding unit, used for performing convolution encoding on the real-time network data to obtain a real-time network data feature vector; The real-time network data feature optimization unit is used to perform viewpoint metric perception optimization based on multi-angle feature relationship quantification on the real-time network data feature vector based on the network intrusion detection associated feature vector to obtain the optimized real-time network data feature vector.

3. The distributed network data processing system based on cloud computing according to claim 2, characterized in that: The network data conversion vector unit includes: An embedding conversion subunit, used for converting the historical network intrusion detection data of each dimension in the historical network intrusion detection data into a historical network intrusion detection data embedding vector to obtain a sequence of historical network intrusion detection data embedding vectors through an embedding layer, wherein the embedding layer uses a learnable embedding matrix to embed the historical network intrusion detection data of each dimension; The encoding subunit is used to pass the sequence of historical network intrusion detection data embedding vectors through the converter-based first context encoder to obtain the multiple historical network intrusion detection data feature vectors.

4. The distributed network data processing system based on cloud computing according to claim 3 is characterized in that: The historical network data one-dimensional arrangement unit includes: A vector arrangement subunit, used for arranging the plurality of historical network intrusion detection data feature vectors into a one-dimensional feature vector of the network intrusion detection data; The multi-scale extraction subunit is used to obtain the multi-scale network intrusion detection feature vector by passing the one-dimensional feature vector of the network intrusion detection data through a multi-scale neighborhood feature extraction module.

5. The distributed network data processing system based on cloud computing according to claim 4, characterized in that: The multi-scale extraction subunit comprises: A first scale encoding secondary subunit, configured to use the first convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the one-dimensional feature vector of the network intrusion detection data with a one-dimensional convolution kernel having a first scale to obtain a first scale network intrusion detection data associated feature vector; A second-scale encoding secondary subunit, configured to use the second convolution layer of the multi-scale neighborhood feature extraction module to perform one-dimensional convolution encoding on the one-dimensional feature vector of the network intrusion detection data with a one-dimensional convolution kernel having a second scale to obtain a second-scale network intrusion detection data associated feature vector; The multi-scale cascade secondary subunit is used to cascade the first-scale network intrusion detection data associated feature vector and the second-scale network intrusion detection data associated feature vector to obtain the multi-scale network intrusion detection feature vector.

6. The distributed network data processing system based on cloud computing according to claim 5, characterized in that: The historical network data two-dimensional arrangement unit includes: An arrangement matrix subunit, used for arranging the plurality of historical network intrusion detection data feature vectors in two dimensions into a network intrusion detection data feature matrix; The convolutional coding subunit is used to pass the network intrusion detection data feature matrix through a historical network intrusion detection data feature extractor based on a first convolutional neural network model to obtain the network intrusion detection local feature vector.

7. The distributed network data processing system based on cloud computing according to claim 6, characterized in that: The real-time network data encoding unit comprises: Performing word segmentation processing on the real-time network data to obtain a real-time network word sequence; The real-time network word sequence is passed through a second context encoder based on a converter to obtain a real-time network data feature vector.

8. The distributed network data processing system based on cloud computing according to claim 7, characterized in that: The real-time network data feature optimization unit comprises: Calculating a position-based difference between the real-time network data feature vector and the network intrusion detection associated feature vector to obtain a first viewpoint metric representation vector; Calculate the position point multiplication between the real-time network data feature vector and the network intrusion detection associated feature vector to obtain a second viewpoint metric representation vector; Calculating the positional sum between the real-time network data feature vector and the network intrusion detection associated feature vector to obtain a third viewpoint metric representation vector; Calculating the gradient magnitude of each position in the first viewpoint metric representation vector, the second viewpoint metric representation vector, and the third viewpoint metric representation vector to obtain a first viewpoint metric perception vector, a second viewpoint metric perception vector, and a third viewpoint metric perception vector; Inputting the first viewpoint metric perception vector, the second viewpoint metric perception vector and the third viewpoint metric perception vector into a reverse learning optimization module to obtain a reverse multi-viewpoint perception optimization representation vector; The optimized real-time network data feature vector is obtained by calculating the position point multiplication between the reverse multi-view perception optimization representation vector and the real-time network data feature vector.

9. The distributed network data processing system based on cloud computing according to claim 8, characterized in that: The network data analysis module comprises: A matrix expansion unit, used for expanding the intrusion behavior monitoring feature matrix into a classification feature vector according to a row vector or a column vector; A fully connected encoding unit, configured to perform fully connected encoding on the classification feature vector using the fully connected layer of the classifier to obtain a fully connected encoded feature vector; A probability obtaining unit, used for passing the fully connected encoded feature vector through the Softmax classification function of the classifier to obtain a first probability attributable to the existence of network intrusion behavior and a second probability attributable to the absence of network intrusion behavior; A classification result determination unit is used to determine the classification result based on a comparison between the first probability and the second probability.

10. A distributed network data processing method based on cloud computing, characterized in that: include: Acquire historical network intrusion detection data and real-time network data, wherein the historical network intrusion detection data includes unlabeled sample data, labeled sample data, and labels corresponding to the labeled sample data; Extracting a network intrusion detection associated feature vector from the historical network intrusion detection data, and encoding the real-time network data to obtain an optimized real-time network data feature vector; The network intrusion detection associated feature vector and the optimized real-time network data feature vector are merged to obtain an intrusion behavior monitoring feature matrix; The intrusion behavior monitoring feature matrix is ​​passed through a classifier to obtain a classification result, and the classification result is used to indicate whether there is network intrusion behavior.

Images