Network data transmission monitoring system and method based on big data analysis

Abstract

The present application provides a network data transmission monitoring system and method based on big data analysis, which relates to the field of network data transmission technology, including: a data monitoring and acquisition module, used to obtain network transmission perception data streams; a big data analysis module, used to obtain multiple network transmission abnormality feature sets; an abnormality identification module, used to perform abnormality identification and obtain network transmission abnormality feature analysis results; an early warning response module, used to trigger a security defense mechanism, perform early warning response and security defense. This application can solve the problem that the existing technology cannot process large-scale network data and has a low accuracy rate for abnormality identification, resulting in the network data transmission monitoring system being unable to monitor abnormalities in a timely manner, further affecting the security and stability of the network. It can effectively respond to abnormal events in network transmission, protect the network from attacks, and ensure business continuity and data security.

Description

Technical Field

[0001] The present application relates to the technical field of network data transmission, and in particular to a network data transmission monitoring system and method based on big data analysis. Background Art

[0002] With the rapid development of Internet technology, network data transmission has become an important way of information flow. New applications such as e-commerce, cloud computing services, and social networks continue to emerge, generating massive amounts of network data. The amount of network data is growing exponentially, and traditional network monitoring methods are difficult to cope with such a huge amount of data. Big data technology makes it possible to process and analyze massive amounts of data. It can extract valuable information from massive amounts of network data, thereby monitoring and identifying abnormal behaviors in network transmission. However, the existing technology has a low accuracy rate in monitoring abnormal behaviors.

[0003] At present, the existing technology is unable to process large-scale network data and has a low accuracy rate in identifying anomalies, resulting in the network data transmission monitoring system being unable to detect anomalies in a timely manner, further affecting the security and stability of the network. Summary of the invention

[0004] The purpose of this application is to provide a network data transmission monitoring system and method based on big data analysis, so as to solve the problem that the existing technology is unable to process large-scale network data and has a low accuracy rate in identifying anomalies, resulting in the network data transmission monitoring system being unable to detect anomalies in a timely manner, further affecting the security and stability of the network.

[0005] In view of the above problems, the present application provides a network data transmission monitoring system and method based on big data analysis.

[0006] In a first aspect, the present application provides a network data transmission monitoring system based on big data analysis, which is used to execute the network data transmission monitoring method based on big data analysis, wherein the network data transmission monitoring system based on big data analysis includes: a data monitoring and acquisition module, which is used to deploy perception nodes on the target network, obtain a set of network perception nodes, perform transmission monitoring and acquisition on the target network through the network perception node set, and obtain a network transmission perception data stream; a big data analysis module, which is used to collect and obtain transmission monitoring data sets of multiple networks through big data technology, extract abnormal features from the transmission monitoring data sets, and obtain multiple network transmission abnormal feature sets; an abnormality identification module, which is used to train and fuse recurrent neural networks based on the multiple network transmission abnormality feature sets, construct a global network transmission abnormality analysis model, use the global network transmission abnormality analysis model to perform abnormality identification on the network transmission perception data stream, and obtain a network transmission abnormality feature analysis result; an early warning response module, which is used to trigger a security defense mechanism, match and analyze the network transmission abnormality feature analysis result with the security defense mechanism, determine a network transmission security defense strategy, and perform early warning response and security defense on the target network through the network transmission security defense strategy.

[0007] In a second aspect, the present application provides a network data transmission monitoring method based on big data analysis, which is implemented by the network data transmission monitoring system based on big data analysis, wherein the network data transmission monitoring method based on big data analysis includes: deploying perception nodes on the target network to obtain a set of network perception nodes, performing transmission monitoring and collection on the target network through the network perception node set to obtain a network transmission perception data stream; acquiring transmission monitoring data sets of multiple networks through big data technology collection, extracting abnormal features of the transmission monitoring data sets, and obtaining multiple network transmission abnormal feature sets; training and fusing recurrent neural networks based on the multiple network transmission abnormal feature sets to construct a global network transmission abnormality analysis model, using the global network transmission abnormality analysis model to identify abnormalities on the network transmission perception data stream, and obtaining network transmission abnormality feature analysis results; triggering a security defense mechanism, matching and parsing the network transmission abnormality feature analysis results with the security defense mechanism, determining a network transmission security defense strategy, and performing early warning response and security defense on the target network through the network transmission security defense strategy.

[0008] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0009] The data monitoring and acquisition module is used to deploy sensing nodes on the target network, obtain a set of network sensing nodes, and perform transmission monitoring and acquisition on the target network through the network sensing node set to obtain a network transmission sensing data stream; the big data analysis module is used to collect and acquire transmission monitoring data sets of multiple networks through big data technology, extract abnormal features from the transmission monitoring data sets, and obtain multiple network transmission abnormal feature sets; the abnormality identification module is used to train and fuse the recurrent neural network based on the multiple network transmission abnormal feature sets, build a global network transmission abnormality analysis model, and use the global network transmission abnormality analysis model to identify abnormalities of the network transmission sensing data stream , obtain the results of the network transmission anomaly feature analysis; an early warning response module is used to trigger the security defense mechanism, match and parse the network transmission anomaly feature analysis results with the security defense mechanism, determine the network transmission security defense strategy, and perform early warning response and security defense on the target network through the network transmission security defense strategy, which effectively solves the problem that the prior art cannot process large-scale network data and has a low accuracy rate for anomaly identification, resulting in the network data transmission monitoring system being unable to detect anomalies in a timely manner, further affecting the security and stability of the network, and can effectively respond to abnormal events in network transmission, protect the network from attacks, and ensure business continuity and data security.

[0010] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented according to the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are specifically cited below. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easy to understand through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0012] Figure 1 This is a structural diagram of the network data transmission monitoring system based on big data analysis in this application.

[0013] Figure 2 This is a flow chart of the network data transmission monitoring method based on big data analysis in this application.

[0014] Description of reference numerals:

[0015] Data monitoring and collection module 11, big data analysis module 12, anomaly identification module 13, and early warning response module 14. DETAILED DESCRIPTION

[0016] The present application provides a network data transmission monitoring system and method based on big data analysis, which solves the problem that the prior art is unable to process large-scale network data and has a low accuracy rate in identifying anomalies, resulting in the network data transmission monitoring system being unable to detect anomalies in a timely manner, further affecting the security and stability of the network. The application can effectively respond to abnormal events in network transmission, protect the network from attacks, and ensure business continuity and data security.

[0017] Below, the technical solutions in the present application will be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments of the present application. It should be understood that the present application is not limited to the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the accompanying drawings, rather than all of them.

[0018] Embodiment 1: This application provides a network data transmission monitoring system based on big data analysis. Figure 1 , the network data transmission monitoring system based on big data analysis includes:

[0019] The data monitoring and collection module 11 is used to deploy sensing nodes on the target network, obtain a network sensing node set, perform transmission monitoring and collection on the target network through the network sensing node set, and obtain network transmission sensing data flow.

[0020] Specifically, according to the characteristics of the target network, select appropriate sensing nodes. These nodes can be sensors, network devices such as switches and routers. Install sensors or configure network devices at key locations of the network, such as core switches, server clusters, etc., for data capture. In a virtual network or cloud environment, you can deploy virtual sensors or use APIs to collect data. Ensure that all sensing nodes are correctly configured, including setting monitoring parameters, data transmission protocols, and data storage solutions. Manage the operating status of all nodes to ensure that they can collect data continuously and stably. Sensing nodes capture all network traffic passing through them, including inbound and outbound traffic. Extract useful information from the captured traffic, such as source / destination IP addresses, port numbers, protocol types, packet sizes, etc. Organize the extracted data into data streams, which can be real-time data streams or batch data streams. Determine the type of data and network areas that need to be collected according to the monitoring purpose. Select tools that can meet the monitoring needs, such as Wireshark, Snort, Zeek, etc. Install and configure the selected tools in the network to ensure that they can collect and transmit data correctly. Before the data is sent to the big data analysis module, perform necessary cleaning and formatting to reduce noise and improve data quality.

[0021] The big data analysis module 12 is used to collect and obtain transmission monitoring data sets of multiple networks through big data technology, extract abnormal features from the transmission monitoring data sets, and obtain multiple network transmission abnormal feature sets.

[0022] Specifically, the data sets collected from different networks and sensing nodes are integrated into a unified data storage system, such as the Hadoop distributed file system. The collected data is preprocessed by cleaning, removing duplicates, filling missing values, etc. to ensure data quality. Features that help identify abnormal behavior, such as traffic size, access mode, communication frequency, packet type, etc., are extracted from the preprocessed data. Statistical methods or machine learning algorithms are used to select the most discriminative features, reduce feature dimensions, and improve analysis efficiency. The extracted features are transformed as necessary, such as standardization and normalization, to facilitate subsequent analysis and modeling. By analyzing the data of normal behavior, the characteristic pattern of normal behavior is established. Real-time data is compared with the normal behavior pattern to identify abnormal behavior that deviates from the normal pattern. The identified abnormal behavior features are organized into a feature set for subsequent anomaly identification and prediction.

[0023] The anomaly identification module 13 is used to train and fuse the recurrent neural network based on the multiple network transmission anomaly feature sets, build a global network transmission anomaly analysis model, use the global network transmission anomaly analysis model to identify anomalies in the network transmission perception data stream, and obtain network transmission anomaly feature analysis results.

[0024] Specifically, multiple sets of network transmission anomaly features are organized into a data format suitable for training RNN, including time series data. Appropriate RNN variants are selected, such as long short-term memory networks or gated recurrent units, and these models are used to handle the long-term dependency problem of sequence data. The RNN model is trained using the anomaly feature set, and the prediction error is minimized by adjusting the model parameters. If multiple networks or different types of anomaly features are included, multiple RNN models are fused into a global model to improve the accuracy and generalization of recognition. The performance of the model is evaluated through a validation set to ensure that the model has good generalization ability. The network transmission perception data stream is input into the trained global network transmission anomaly analysis model. The model analyzes the data stream and identifies abnormal behaviors that are inconsistent with normal behavior patterns. Output the anomaly recognition results, including the severity, type, possible impact, etc. of the anomaly.

[0025] The early warning response module 14 is used to trigger the security defense mechanism, match and analyze the network transmission abnormality feature analysis results with the security defense mechanism, determine the network transmission security defense strategy, and perform early warning response and security defense on the target network through the network transmission security defense strategy.

[0026] Specifically, when the anomaly identification module detects abnormal behavior in network transmission, the early warning response module is activated. The early warning response module decides whether to trigger the security defense mechanism based on the severity and type of the anomaly. The early warning response module compares the results of the network transmission anomaly feature analysis with the preset security defense mechanism. Through matching analysis, determine which security defense measures are applicable to the currently detected anomaly. Based on the results of matching analysis, formulate specific defense strategies, such as isolating malicious traffic, restricting access to specific IPs, increasing monitoring frequency, etc. The formulation of strategies needs to consider the nature of the anomaly, potential risks, and impact on the business. Send an alert to the network administrator or security operation center to provide detailed information about the anomaly. Automatically execute defense strategies, such as adjusting firewall rules, starting intrusion prevention systems, etc.

[0027] Furthermore, the data monitoring and collection module 11 is also used for:

[0028] Identify key points of the target network to obtain a set of network transmission key points; perform coverage analysis on the target network based on the set of network transmission key points to obtain a set of network transmission monitoring points; perform transmission demand analysis on the set of network transmission monitoring points to obtain a set of data transmission monitoring requirements, wherein the set of data transmission monitoring requirements includes data type, transmission frequency, data volume, and real-time requirements; perform node performance configuration on the set of network transmission monitoring points based on the set of data transmission monitoring requirements to obtain the set of network perception nodes.

[0029] Specifically, identify key points in the network, including core routers, data centers, server clusters, etc. Analyze the network topology and identify key nodes and links in the network. Monitor network traffic and identify nodes with large traffic and frequent data transmission. According to business needs, evaluate which nodes are most critical to business continuity. Organize the identified key points into a set of network transmission key points. Ensure that the monitoring points can fully cover the key points in the network to effectively monitor network transmission. Map the key points of network transmission to the monitoring network, plan the location and number of monitoring points, ensure that the key points are fully covered, evaluate whether the layout of monitoring points meets the coverage requirements, make adjustments if necessary, determine the final layout of monitoring points, and form a set of network transmission monitoring points. Collect information such as the data type, transmission frequency, data volume, and real-time requirements of each monitoring point, analyze the collected information, determine the specific monitoring requirements of the monitoring point, and organize the analysis results into a set of data transmission monitoring requirements. Configure the performance of the monitoring node according to the monitoring requirements to ensure that the node can meet the requirements of the monitoring task. According to the set of data transmission monitoring requirements, allocate appropriate computing resources, storage space, and bandwidth to each monitoring node, adjust the node configuration, and optimize data processing speed and efficiency. After the configuration is completed, the monitoring nodes are tested to ensure that their performance meets the requirements, and the configured monitoring nodes are integrated into a set of network perception nodes.

[0030] Furthermore, the big data analysis module 12 is also used for:

[0031] Perform label vector identification on each transmission monitoring data in the transmission monitoring data set to obtain a data feature vector label set; divide the transmission monitoring data set according to a preset ratio to obtain a training transmission monitoring data set and a test transmission monitoring data set; identify the training transmission monitoring data set and the test transmission monitoring data set based on the data feature vector label set to obtain a training transmission monitoring data label and a test transmission monitoring data label; use the training transmission monitoring data set and the training transmission monitoring data label as training data for model training, and perform model verification through the test transmission monitoring data set and the test transmission monitoring data label to obtain a network transmission anomaly feature identifier; perform anomaly feature extraction on the transmission monitoring data set based on the network transmission anomaly feature identifier to obtain the multiple network transmission anomaly feature sets.

[0032] Specifically, extract relevant features from the original monitoring data, such as traffic size, protocol type, source / destination address, etc. Construct a multidimensional label vector based on the feature type and category, where each dimension represents a feature or category. Associate the label vector with the corresponding transmission monitoring data to form a data feature vector label set. Select an appropriate partition ratio based on the size of the data set and model requirements. Usually, the training set accounts for a larger proportion, such as 70-80%. Use a random partitioning method to ensure the randomness of data distribution. If the data set categories are unbalanced, use stratified sampling to keep the proportion of each category consistent in the training set and the test set. According to the data feature vector label set, assign corresponding labels to each data in the training set and the test set. Ensure that both the training transmission monitoring data set and the test transmission monitoring data set have correct label information. Select a machine learning model suitable for the problem, such as decision tree, support vector machine, neural network, etc. Use the training transmission monitoring data set and the training transmission monitoring data label to train the model. Use the test transmission monitoring data set and the test transmission monitoring data label to verify the model and evaluate the performance of the model. Apply the trained network transmission anomaly feature identifier to the entire transmission monitoring data set. The model identifies the anomaly instances in the data set and extracts their features. The identified anomaly features are organized into multiple network transmission anomaly feature sets for subsequent anomaly analysis and defense strategy formulation.

[0033] Furthermore, the big data analysis module 12 is also used for:

[0034] According to the network data transmission attribute set, a data transmission feature coordinate system is constructed, and the data transmission feature coordinate system uses the network data transmission attribute set as a multi-dimensional coordinate axis; the data transmission feature coordinate system is regionally labeled and classified to obtain a coordinate system regional label classification result, and the coordinate system regional label classification result includes whether it is abnormal, the type of abnormality, and the degree of abnormality; each transmission monitoring data in the transmission monitoring data set is mapped to the data transmission feature coordinate system to obtain a transmission monitoring data vector set; feature matching is performed based on the transmission monitoring data vector set and the coordinate system regional label classification result to obtain the data feature vector label set.

[0035] Specifically, determine the key attributes of network data transmission, such as source IP address, destination IP address, port number, traffic size, protocol type, timestamp, transmission speed, packet loss rate, delay time and other related transmission characteristics. Define each attribute as a dimension of the coordinate system to form a multi-dimensional coordinate axis. According to security policies and business needs, formulate standards for regional classification, including normal behavior, abnormal behavior, abnormal type and degree of abnormality. Divide different areas in the coordinate system and mark each area as a specific label classification. Convert each transmission monitoring data into a data feature vector, and the value of each dimension corresponds to the data transmission attribute. Map the data feature vector to the data transmission feature coordinate system to determine its position in the coordinate system. According to the regional label classification results, formulate matching rules to determine how to map the data vector to a specific label. According to the matching rules, assign one or more labels to each vector in the transmission monitoring data vector set to form a data feature vector label set.

[0036] Furthermore, the big data analysis module 12 is also used for:

[0037] The training transmission monitoring data set and the training transmission monitoring data label are used as training data for model training to obtain an initial abnormal feature identifier; the output result of the initial abnormal feature identifier is verified by the test transmission monitoring data set and the test transmission monitoring data label to obtain the identifier output accuracy; if the identifier output accuracy does not reach the preset classification accuracy, the transmission attribute information in the transmission monitoring data set is subjected to main feature analysis to obtain reduced-dimensionality data transmission attributes; based on the reduced-dimensionality data transmission attributes, the initial abnormal feature identifier is corrected and trained to obtain the network transmission abnormal feature identifier.

[0038] Specifically, ensure that the training transmission monitoring data set and training transmission monitoring data labels are ready. Select an appropriate machine learning model or deep learning model for training. Use the training data set and labels to train the model until the model converges. Test the model using the test transmission monitoring data set and test transmission monitoring data labels. Compare the output results of the model with the actual labels, calculate the accuracy, and evaluate the performance of the model based on the preset classification accuracy standard. Use feature selection techniques, such as recursive feature elimination and principal component analysis, to evaluate the importance of each attribute in the transmission monitoring data set. Select the most important features, reduce the data dimension, and obtain the reduced-dimensional data transmission attributes. Reconstruct the training set using the reduced-dimensional data transmission attributes, retrain the initial anomaly feature identifier using the new training set, and adjust the model parameters, such as adding hidden layer nodes, changing the learning rate, etc., to improve model performance.

[0039] Furthermore, the abnormality identification module 13 is also used for:

[0040] Based on the multiple network transmission anomaly feature sets, recurrent neural networks are distributedly trained to construct multiple network transmission anomaly feature analysis models; model parameters are extracted for the multiple network transmission anomaly feature analysis models to obtain multiple analysis model parameter information; the multiple analysis model parameter information is encrypted and sent to a trusted central server; based on the trusted central server, model update aggregation is performed on the multiple analysis model parameter information, and it is repeated to a preset termination condition to construct the global network transmission anomaly analysis model.

[0041] Specifically, multiple sets of network transmission anomaly features are assigned to different training nodes, and a recurrent neural network model is built on each training node. Distributed computing frameworks such as Apache Spark and TensorFlow Distributed are used for parallel training. After training, the weights, biases and other parameters of the model are obtained from each training node, and the parameter information of each model is sorted out for subsequent aggregation. Encrypted communication protocols such as HTTPS and SSL are used to ensure the security of data transmission. The encrypted model parameter information is sent to a trusted central server. Aggregation algorithms such as the aggregation method in federated learning are implemented on the trusted central server, and the aggregated parameters are used to update the global model. The above process is repeated until the preset termination conditions are met, such as model performance convergence or the upper limit of the number of iterations is reached.

[0042] Furthermore, the abnormality identification module 13 is also used for:

[0043] The security defense mechanism is analyzed for application scenarios to obtain a set of abnormal application scenario parameters, and the security defense mechanism is clustered and calibrated through the set of abnormal application scenario parameters to obtain a network transmission security defense policy library; the results of the network transmission abnormal feature analysis are classified into applicable scenarios to determine applicable scenario parameters; based on the applicable scenario parameters, the parameters are matched with the network transmission security defense policy library to determine the network transmission security defense policy.

[0044] Specifically, analyze the abnormal situations that may occur in network transmission, such as DDoS attacks, port scanning, malicious code transmission, etc. For each abnormal situation, determine the key parameters that affect the effectiveness of the security defense mechanism and form a set of abnormal application scenario parameters. Select appropriate clustering algorithms, such as K-means, hierarchical clustering, etc. to classify the security defense mechanism. According to the abnormal application scenario parameter set, cluster the security defense mechanism into different categories to form a network transmission security defense policy library. According to the abnormal feature analysis results, determine the applicable scenario of each abnormality, extract the parameters of the applicable scenario from the abnormal features, form the applicable scenario parameters, and match the applicable scenario parameters with the network transmission security defense policy library. According to the matching results, select the most appropriate network transmission security defense strategy for implementation.

[0045] For Example 2, please refer to the attached Figure 2 The present application provides a network data transmission monitoring method based on big data analysis, wherein the network data transmission monitoring method based on big data analysis is applied to a network data transmission monitoring system based on big data analysis, and the network data transmission monitoring method based on big data analysis specifically includes the following steps:

[0046] Deploy sensing nodes on the target network to obtain a network sensing node set, monitor and collect transmission on the target network through the network sensing node set, and obtain network transmission sensing data streams.

[0047] The transmission monitoring data sets of multiple networks are collected and acquired through big data technology, and abnormal features are extracted from the transmission monitoring data sets to obtain multiple network transmission abnormal feature sets.

[0048] Based on the multiple network transmission anomaly feature sets, the recurrent neural network is trained and integrated to build a global network transmission anomaly analysis model. The global network transmission anomaly analysis model is used to identify anomalies in the network transmission perception data stream to obtain a network transmission anomaly feature analysis result.

[0049] The security defense mechanism is triggered, and the network transmission security defense strategy is determined based on the matching analysis result of the network transmission abnormality characteristics with the security defense mechanism, and the target network is given an early warning response and security defense through the network transmission security defense strategy.

[0050] Further, obtaining the network perception node set includes:

[0051] Identify key points of the target network to obtain a set of network transmission key points; perform coverage analysis on the target network based on the set of network transmission key points to obtain a set of network transmission monitoring points; perform transmission demand analysis on the set of network transmission monitoring points to obtain a set of data transmission monitoring requirements, wherein the set of data transmission monitoring requirements includes data type, transmission frequency, data volume, and real-time requirements; perform node performance configuration on the set of network transmission monitoring points based on the set of data transmission monitoring requirements to obtain the set of network perception nodes.

[0052] Further, the obtaining of multiple network transmission abnormality feature sets includes:

[0053] Perform label vector identification on each transmission monitoring data in the transmission monitoring data set to obtain a data feature vector label set; divide the transmission monitoring data set according to a preset ratio to obtain a training transmission monitoring data set and a test transmission monitoring data set; identify the training transmission monitoring data set and the test transmission monitoring data set based on the data feature vector label set to obtain a training transmission monitoring data label and a test transmission monitoring data label; use the training transmission monitoring data set and the training transmission monitoring data label as training data for model training, and perform model verification through the test transmission monitoring data set and the test transmission monitoring data label to obtain a network transmission anomaly feature identifier; perform anomaly feature extraction on the transmission monitoring data set based on the network transmission anomaly feature identifier to obtain the multiple network transmission anomaly feature sets.

[0054] Further, the obtaining of the data feature vector label set includes:

[0055] According to the network data transmission attribute set, a data transmission feature coordinate system is constructed, and the data transmission feature coordinate system uses the network data transmission attribute set as a multi-dimensional coordinate axis; the data transmission feature coordinate system is regionally labeled and classified to obtain a coordinate system regional label classification result, and the coordinate system regional label classification result includes whether it is abnormal, the type of abnormality, and the degree of abnormality; each transmission monitoring data in the transmission monitoring data set is mapped to the data transmission feature coordinate system to obtain a transmission monitoring data vector set; feature matching is performed based on the transmission monitoring data vector set and the coordinate system regional label classification result to obtain the data feature vector label set.

[0056] Further, the obtaining of the network transmission abnormality feature identifier includes:

[0057] The training transmission monitoring data set and the training transmission monitoring data label are used as training data for model training to obtain an initial abnormal feature identifier; the output result of the initial abnormal feature identifier is verified by the test transmission monitoring data set and the test transmission monitoring data label to obtain the identifier output accuracy; if the identifier output accuracy does not reach the preset classification accuracy, the transmission attribute information in the transmission monitoring data set is subjected to main feature analysis to obtain reduced-dimensionality data transmission attributes; based on the reduced-dimensionality data transmission attributes, the initial abnormal feature identifier is corrected and trained to obtain the network transmission abnormal feature identifier.

[0058] Further, the construction of the global network transmission anomaly analysis model includes:

[0059] Based on the multiple network transmission anomaly feature sets, recurrent neural networks are distributedly trained to construct multiple network transmission anomaly feature analysis models; model parameters are extracted for the multiple network transmission anomaly feature analysis models to obtain multiple analysis model parameter information; the multiple analysis model parameter information is encrypted and sent to a trusted central server; based on the trusted central server, model update aggregation is performed on the multiple analysis model parameter information, and it is repeated to a preset termination condition to construct the global network transmission anomaly analysis model.

[0060] Further, the determining of the network transmission security defense strategy includes:

[0061] The security defense mechanism is analyzed for application scenarios to obtain a set of abnormal application scenario parameters, and the security defense mechanism is clustered and calibrated through the set of abnormal application scenario parameters to obtain a network transmission security defense policy library; the results of the network transmission abnormal feature analysis are classified into applicable scenarios to determine applicable scenario parameters; based on the applicable scenario parameters, the parameters are matched with the network transmission security defense policy library to determine the network transmission security defense policy.

[0062] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0063] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application belong to the scope of the present application and its equivalent technology, the present application is also intended to include these modifications and variations.

Claims

1. A network data transmission monitoring system based on big data analysis, characterized in that: include: A data monitoring and collection module is used to deploy sensing nodes on the target network, obtain a set of network sensing nodes, perform transmission monitoring and collection on the target network through the set of network sensing nodes, and obtain network transmission sensing data streams; A big data analysis module is used to collect and obtain transmission monitoring data sets of multiple networks through big data technology, extract abnormal features from the transmission monitoring data sets, and obtain multiple network transmission abnormal feature sets; An anomaly identification module is used to train and fuse the recurrent neural network based on the multiple network transmission anomaly feature sets, build a global network transmission anomaly analysis model, use the global network transmission anomaly analysis model to perform anomaly identification on the network transmission perception data stream, and obtain a network transmission anomaly feature analysis result; An early warning response module is used to trigger a security defense mechanism, match and analyze the network transmission abnormality feature analysis result with the security defense mechanism, determine a network transmission security defense strategy, and perform early warning response and security defense on the target network through the network transmission security defense strategy; Wherein, the data monitoring and collection module is also used for: Identifying key points of the target network to obtain a set of network transmission key points; Performing coverage analysis on the target network based on the network transmission key point set to obtain a network transmission monitoring point set; Performing transmission requirement analysis on the network transmission monitoring point set to obtain a data transmission monitoring requirement set, wherein the data transmission monitoring requirement set includes data type, transmission frequency, data volume, and real-time requirements; The network transmission monitoring point set is configured for node performance based on the data transmission monitoring requirement set to obtain the network perception node set.

2. The network data transmission monitoring system based on big data analysis according to claim 1, characterized in that: The big data analysis module is also used for: Performing label vector identification on each transmission monitoring data in the transmission monitoring data set to obtain a data feature vector label set; Dividing the transmission monitoring data set according to a preset ratio to obtain a training transmission monitoring data set and a test transmission monitoring data set; The training transmission monitoring data set and the test transmission monitoring data set are labeled based on the data feature vector label set to obtain training transmission monitoring data labels and test transmission monitoring data labels; The training transmission monitoring data set and the training transmission monitoring data label are used as training data to perform model training, and the test transmission monitoring data set and the test transmission monitoring data label are used to perform model verification to obtain a network transmission anomaly feature identifier; Based on the network transmission abnormality feature identifier, abnormality features are extracted from the transmission monitoring data set to obtain the multiple network transmission abnormality feature sets.

3. The network data transmission monitoring system based on big data analysis according to claim 2, characterized in that: The big data analysis module is also used for: According to the network data transmission attribute set, a data transmission feature coordinate system is constructed, wherein the data transmission feature coordinate system uses the network data transmission attribute set as a multi-dimensional coordinate axis; Performing regional label classification on the data transmission feature coordinate system to obtain a coordinate system regional label classification result, wherein the coordinate system regional label classification result includes whether it is abnormal, the type of abnormality, and the degree of abnormality; Mapping each transmission monitoring data in the transmission monitoring data set to the data transmission characteristic coordinate system to obtain a transmission monitoring data vector set; The data feature vector label set is obtained by performing feature matching based on the transmission monitoring data vector set and the coordinate system area label classification result.

4. The network data transmission monitoring system based on big data analysis according to claim 2, characterized in that: The big data analysis module is also used for: Using the training transmission monitoring data set and the training transmission monitoring data label as training data to perform model training to obtain an initial abnormal feature identifier; Verifying the output result of the initial abnormal feature identifier by using the test transmission monitoring data set and the test transmission monitoring data label to obtain the output accuracy of the identifier; If the accuracy of the output of the identifier does not reach the preset classification accuracy, main feature analysis is performed on the transmission attribute information in the transmission monitoring data set to obtain reduced-dimensional data transmission attributes; The initial abnormal feature identifier is corrected and trained based on the reduced-dimensional data transmission attribute to obtain the network transmission abnormal feature identifier.

5. The network data transmission monitoring system based on big data analysis according to claim 1, characterized in that: The abnormality identification module is also used for: Based on the multiple network transmission abnormality feature sets, distributed training is performed on the recurrent neural network to construct multiple network transmission abnormality feature analysis models; Extracting model parameters of the plurality of network transmission abnormality feature analysis models respectively to obtain a plurality of analysis model parameter information; Encrypting the plurality of analysis model parameter information and sending them to a trusted central server; Based on the trusted central server, the multiple analysis model parameter information is aggregated for model update, and it is iterated repeatedly until a preset termination condition is reached to construct the global network transmission anomaly analysis model.

6. The network data transmission monitoring system based on big data analysis according to claim 5, characterized in that: The abnormality identification module is also used for: Performing application scenario analysis on the security defense mechanism to obtain an abnormal application scenario parameter set, and clustering and calibrating the security defense mechanism through the abnormal application scenario parameter set to obtain a network transmission security defense strategy library; Classifying applicable scenarios for the network transmission abnormality feature analysis results and determining applicable scenario parameters; The network transmission security defense strategy is determined based on matching the applicable scenario parameters with the network transmission security defense strategy library.

7. A network data transmission monitoring method based on big data analysis, characterized in that: The network data transmission monitoring method based on big data analysis is implemented by the network data transmission monitoring system based on big data analysis according to any one of claims 1 to 6, comprising: Deploy sensing nodes on the target network to obtain a set of network sensing nodes, and perform transmission monitoring and collection on the target network through the set of network sensing nodes to obtain a network transmission sensing data stream; Acquire transmission monitoring data sets of multiple networks through big data technology, extract abnormal features from the transmission monitoring data sets, and obtain multiple network transmission abnormal feature sets; Based on the multiple network transmission anomaly feature sets, a recurrent neural network is trained and integrated to construct a global network transmission anomaly analysis model, and the global network transmission anomaly analysis model is used to identify anomalies in the network transmission perception data stream to obtain a network transmission anomaly feature analysis result; The security defense mechanism is triggered, and the network transmission security defense strategy is determined based on the matching analysis result of the network transmission abnormality characteristics with the security defense mechanism, and the target network is given an early warning response and security defense through the network transmission security defense strategy.

Images