IPv6 flow monitoring platform based on big data analysis

Abstract

The invention belongs to the technical field of traffic monitoring, and particularly relates to an IPv6 traffic monitoring platform based on big data analysis, comprising an IPv6 traffic acquisition subsystem which acquires dynamic traffic in real time through a built-in dynamic traffic acquisition method based on big data and provides a data basis for subsequent traffic monitoring; the traffic encryption subsystem provides hardware support for an IPv6 security encryption method, realizes security encryption for network IPv6 traffic, and ensures traffic transmission and storage security; according to the intelligent encrypted traffic identification and abnormal traffic detection method, intelligent identification of encrypted traffic and accurate and efficient detection of abnormal traffic are realized under the support of an intelligent algorithm processing platform by designing an intelligent calculation algorithm; the IPv6 traffic visual monitoring platform realizes front-end interface design and functional module editing through Qt, Pycharm and MATLAB software compiling platforms, and displays traffic monitoring results in real time by combining a digital twinborn technology, including traffic information, a traffic identification result and an abnormal traffic detection result.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of flow monitoring, and particularly relates to an IPv6 flow monitoring platform based on big data analysis. BACKGROUND

[0002] With the continuous development of Internet technology, the scale of data is showing an explosive growth. IPv6 (Internet Protocol version 6) as the next generation of Internet protocol, with its huge address space, higher security and better scalability, is gradually replacing IPv4 to become the network infrastructure. According to relevant statistical data, as of early 2020, nearly 80 provincial and ministerial websites in China support IPv6, more than 90 central enterprise websites support IPv6, and China Telecom, China Unicom, China Mobile and other operators have completed IPv6 transformation. There are a large number of self-operated IPv6-based Apps running. This trend shows that the popularization and application of IPv6 are gradually deepening, and it is particularly urgent to monitor and analyze the network traffic under the IPv6 environment.

[0003] IPv6 not only solves the problem of IPv4 address space shortage, but also improves the overall performance and security of the network by improving the protocol structure and adding security features. However, with the rapid development of IPv6, various new applications and new attacks have also emerged, which poses a serious challenge to the security and performance of the next generation of Internet. IPv4 and IPv6 protocols are independent of each other, IPv6 has made a lot of improvements on IPv4 protocol, so that the traffic behavior under IPv6 network changes, and the characteristics of new attacks also change. The existing network traffic monitoring and analysis system is mostly designed based on IPv4 protocol, which cannot effectively parse IPv6 protocol, and cannot effectively detect new attack forms. Therefore, it is of great practical application value and theoretical research significance to study a flow monitoring and analysis platform with high overall performance under high flow and complex IPv6 network.

[0004] The practical significance of the IPv6 flow monitoring platform research mainly includes the following aspects:

[0005] (1) Improve the accuracy and efficiency of network traffic analysis under IPv6 environment The IPv6 traffic monitoring platform based on big data analysis can realize the real-time collection, storage and analysis of massive data in the IPv6 environment. Through in-depth analysis of network traffic, rich network behavior characteristics can be obtained, including packet bytes, server response time, retransmission rate, failed connection number, packet size, etc. These information is of great significance for evaluating network performance, detecting network anomalies and ensuring network security. Traditional network traffic analysis systems often rely on eigenvalue identification technology, which requires maintaining a library of eigenvalues and constantly updating it, and cannot identify and judge new or variant attack methods. The IPv6 traffic monitoring platform based on big data analysis can monitor and alarm suspicious network traffic and behavior in real time through network behavior analysis technology, including host scanning, port scanning, worm virus, suspicious connection, etc., thereby improving the accuracy and efficiency of network traffic analysis.

[0006] (2) Promote the popularization and application of IPv6 technology The popularization and application of IPv6 technology cannot be separated from a perfect network traffic monitoring and analysis system. The IPv6 traffic monitoring platform based on big data analysis can provide comprehensive network traffic monitoring and analysis capabilities, helping enterprises and individuals to real-time grasp the running status of IPv6 network, user experience speed, network application performance and network resource utilization, etc. These information is of great significance for optimizing network configuration, improving user experience, ensuring network security and stability. By promoting the popularization and application of IPv6 technology, the overall performance and security of the Internet can be further improved, and the healthy development of the Internet industry can be promoted.

[0007] (3) Promote data-driven decision-making The IPv6 traffic monitoring platform based on big data analysis not only can provide real-time network traffic monitoring and analysis capabilities, but also can conduct deep mining and intelligent analysis of network traffic data through data mining, machine learning and other advanced technologies. Through deep mining of network traffic data, the rules and trends in network traffic can be discovered, providing strong support for network optimization and decision-making. At the same time, intelligent analysis technology based on machine learning can automatically identify abnormal patterns and potential threats in network traffic, improving the warning and response capabilities of network security. These functions help enterprises and individuals to realize data-driven decision-making, improve business development and competitive advantage.

[0008] (4) Ensure network health and information security With the widespread application of IPv6, network security issues have become increasingly prominent. New forms of attacks in IPv6 network environments continue to emerge, posing a serious threat to network security. An IPv6 traffic monitoring platform based on big data analysis can monitor and alert abnormal traffic and behavior in the network in real time, promptly discovering and resolving network security risks. At the same time, through the retrospective analysis of network traffic, the source and path of attacks can be traced, providing strong support for the investigation and handling of network security incidents. In addition, the IPv6 traffic monitoring platform based on big data analysis can provide more than 100 kinds of indicators and object data based on packet, network usage, transmission efficiency, network performance, connection information, application access, application performance, user experience, etc. for third-party data platform analysis. These data are of great significance for evaluating network health, optimizing network configuration, and improving network security level.

[0009] (5) Promote the development of related industries The research and application of IPv6 traffic monitoring platform based on big data analysis not only can promote the development of Internet industry, but also can drive the development of related industries. For example, with the rapid growth of IPv6 traffic, the demand for network storage and management technology is increasing. Research on efficient storage and management technology of massive data in IPv6 environment is of great significance to promote the development of storage industry. At the same time, the research and application of IPv6 traffic monitoring platform based on big data analysis also requires a large amount of data collection, preprocessing and analysis technology, which will promote the development of data processing and analysis industry. In addition, the research and application of IPv6 traffic monitoring platform based on big data analysis will also drive the development of network security industry, promote the innovation and upgrading of network security technology.

[0010] The prior art CN118734134A discloses a small sample abnormal flow detection method and system based on a meta-learning framework, which learns the difference between known abnormal flow and normal flow to distinguish new unknown abnormal flow and normal flow with less sample number. The application is mainly used to solve the problem that the traditional deep learning model is difficult to identify small sample data, and improve the generalization ability of the algorithm. However, the application only solves the problem of abnormal flow detection and identification, and does not provide a good solution for normal flow monitoring. Moreover, the selection of meta-learning and small sample to improve the generalization ability of the algorithm and solve the data shortage dilemma will inevitably lead to the reduction of the algorithm performance. The prior art CN118714080A discloses a distributed network flow monitoring method, which comprises the following steps: each branch flow monitoring node respectively monitors the flow of itself; if the branch flow monitoring node monitors the flow of itself, and the flow reaches the flow detection threshold of the current flow ladder, the branch flow monitoring node reports to the total flow monitoring node and applies to upgrade the flow ladder; if the branch flow monitoring node monitors the flow of itself, and the flow drops below the flow detection threshold of the next flow ladder of the current flow ladder, the branch flow monitoring node reports to the total flow monitoring node and applies to reduce the flow ladder. The application has the beneficial effect of providing a distributed network flow monitoring method which can realize dynamic monitoring of node flow, reasonably utilize idle flow, save flow resources, and improve work efficiency. The application focuses on realizing flexible and reasonable scheduling of resources based on the existing flow monitoring capability of each branch, and does not provide an effective solution for improving the flow monitoring performance of each branch, which still has room for improvement. SUMMARY

[0011] To solve the above problems, the application provides an IPv6 flow monitoring platform based on big data analysis to solve the problems existing in the prior art.

[0012] To achieve the above purpose, the application provides the following technical scheme: an IPv6 flow monitoring method based on big data analysis, applied to an IPv6 flow monitoring platform based on big data analysis, comprising a dynamic flow acquisition method based on big data, an IPv6 secure flow encryption method, an intelligent identification method of IPv6 encrypted flow based on machine learning, and an efficient detection method of abnormal flow based on big data analysis.

[0013] Further, the dynamic flow acquisition method based on big data is the data basis of the IPv6 flow monitoring method based on big data analysis, which acquires IPv6 flow data from different networks such as video flow, voice flow and game flow through big data technology, and provides support for subsequent flow intelligent identification and prediction. The specific steps include: S10: Determine the data source, obtain the current network transmission architecture, including the layout and configuration of network devices (such as routers, switches), security devices (such as firewalls, intrusion detection systems) and application servers, determine the possible entry and exit points of IPv6 traffic; S101: Maintain and update the IP address database, including the allocation and use of IPv6 addresses, track and locate the source and target addresses of IPv6 traffic through the IP address management system; S102: Deploy traffic monitoring tools at key traffic nodes, use deep packet inspection technology to analyze the content of data packets, and analyze log files to obtain detailed information of IPv6 traffic; S103: Use protocol obfuscation identification and geographic location positioning technology to further analyze and determine the source of traffic; S104: Correlate and analyze the collected data, and verify and test to ensure the accuracy and reliability of the analysis results; S11: Deploy collection tools, including NetFlow, sFlow or IPFIX, configure the collection tools to capture IPv6 network traffic data in real time, convert it into standardized traffic records, and ensure data integrity and accuracy; S12: Data preprocessing, preprocess the collected raw traffic data, including data cleaning, format conversion and deduplication, to ensure data quality; extract key fields related to IPv6 traffic, including source IP address, destination IP address, protocol type, packet size, timestamp; S13: Select a storage solution, select an appropriate storage solution based on data volume and processing requirements, such as distributed file system (such as Hadoop HDFS), database (such as MongoDB, Cassandra) or data warehouse (such as Hive, Spark SQL), to ensure that the storage solution can support large data volume storage and efficient data access; S14: Data import and storage, import the preprocessed IPv6 traffic data into the selected storage system, and establish indexing and partitioning strategies to improve data query, call and analysis efficiency; S15: Visualization processing, through digital twin technology, use data visualization tools such as Tableau, Power BI to present the real-time acquired IPv6 traffic data in the form of charts, reports, etc., more intuitively reflect the IPv6 traffic data acquisition situation, ensure that the acquired data can provide support for subsequent traffic identification.

[0014] Further, the IPv6 secure flow encryption method is mainly used to solve the security problem of large-scale IPv6 flow data obtained by big data in network transmission and node storage, and to reduce the probability of data leakage and network attack malicious events as much as possible. The specific steps include: S20: Key generation, a pair of keys including a public key (PK) and a private key (SK) are generated using the RSA encryption algorithm, the public key is used to encrypt the symmetric key, that is, the triple data encryption (3DES) key, and the private key is used to decrypt the symmetric key; S21: Selection of initialization vector (IV), the initialization vector (IV) is used for the CBC mode (Cipher Block Chaining) of 3DES encryption, the length of the initialization vector is usually 8 bytes, and it must be random to ensure the randomness and security of encryption; S22: Plain text data segmentation, the plain text data to be encrypted is segmented according to the block size of the triple data encryption algorithm to obtain multiple data blocks; S23: 3DES encryption, the generated 3DES key and IV are used to encrypt each data block, the 3DES encryption process includes three DES encryption operations, which can adopt EDE mode (encryption-decryption-encryption) or EEE mode (encryption-encryption-encryption), and after encryption, the ciphertext data block is obtained; S24: RSA encryption key and IV, the 3DES key and IV are encrypted using the RSA public key, and after encryption, the ciphertext of the key and IV is obtained; S25: Combination of ciphertext and encrypted key, the 3DES encrypted ciphertext data block and the RSA encrypted key and IV ciphertext are combined to form the final encrypted data packet.

[0015] It should be noted that the generation of the above-mentioned 3DES key needs to ensure at least 24 bytes (192 bits) of length to meet the security requirements; Optionally, the final encrypted data packet formed above can include the sequence information of the ciphertext data block, the ciphertext data block itself, the RSA encrypted key ciphertext and the IV ciphertext.

[0016] Further, the IPv6 encryption flow identification method based on flow characteristics and behavior is mainly used to identify various encryption flow types and their characteristics in the IPv6 network. In the identification process, different application programs and services have specific characteristics when transmitting flow. By analyzing the flow characteristics, more information about the transmission data characteristics can be obtained, so as to realize accurate identification of encrypted flow. The specific steps include: S30: Data preprocessing, parsing the collected data packets, extracting key information, including source IP address, destination IP address, source port, destination port, protocol type, packet size, packet timestamp, then removing irrelevant or redundant information through de-duplication and filtering processing; S31: Flow feature extraction, analyzing the length, interval, frequency statistical characteristics of data packets, extracting the flag bit, checksum field information of data packets, analyzing the transmission layer protocol and application layer protocol information of data packets; S32: Behavior feature extraction, analyzing the transmission mode of data packets, including whether there is burst transmission, periodic transmission; measuring the behavior of network connection, including connection establishment, disconnection, retransmission behavior; analyzing the directionality of network connection, including whether there is bidirectional communication or unidirectional communication; S33: Constructing a decision tree intelligent computing model, using known types of encrypted traffic data as a training set to train the model, and constantly adjusting the parameters and feature selection of the model during the training process to improve the recognition accuracy of the model; S34: Test the model using test set data to evaluate the recognition performance of the model, the test set data includes different types of encrypted traffic and normal traffic to verify the generalization ability of the model; S35: According to the test results, optimize the model, including adjusting the feature weight, adding new features, improving the model structure, repeating the test and optimization process until the model reaches satisfactory recognition performance; S36: Use the trained model to identify encrypted network traffic.

[0017] In a preferred embodiment, step S33 can also be implemented using a deep learning model-based method. In the identification process, the deep learning model-based IPv6 traffic intelligent identification method will use machine learning algorithms to train and learn network encrypted traffic data, thereby extracting various traffic type features. These features may include packet header information, payload content, transmission speed, arrival time interval, etc. By comparing and analyzing these features, the technology can accurately identify various traffic types in the network and classify and manage them. The specific steps include: Data cleaning, the original encrypted traffic data is composed of PCAP files, each PCAP file contains multiple data packets, however, these data packets do not all contain relevant features of specific applications, such as SSDP, LLMNR, DNS, ARP, NetBIOS, ICMP, IGMP protocol packets, in order to eliminate its influence on the accuracy of traffic identification, first use the built-in editcap tool in Wireshark software to filter out the data packets of the above protocols; the original data packets after filtering out the protocol data packets are aggregated into flows according to the five tuple information, which are used as target objects for feature extraction, the clear text information that can be used as features in the flow includes source IP address, destination IP address, source port, destination port, transmission layer protocol, packet length, window value, flag bit; the frame length and window value of each data packet in a flow are extracted to form a packet length sequence and a window sequence Then and are combined into the final input data sequence; data enhancement, due to the large difference in sample size between different categories of encrypted data sources, in order to avoid the problem of poor recognition ability of minority class encrypted data caused by serious class encrypted data sample imbalance, it is necessary to expand the minority encrypted class sample data through data enhancement, so that all categories remain relatively balanced, the specific operation is: Use k-neighbor algorithm to calculate K nearest neighbors of each minority class sample Set a sampling rate N, randomly select one sample from K nearest neighbors, repeat N times; for the K nearest neighbor samples randomly selected, use the following formula to synthesize new samples respectively, and add them to the sample set;

[0018] Repeat the above process until the specified number of samples is generated or the class balance is reached, and end the algorithm, return the synthesized new data set; Map the data after data enhancement to a high-dimensional vector space, convert it to a continuous vector through vector embedding, so that the neural network can discover more detailed features;

[0019] A neural network model TrCNN combining a convolutional neural network and a Transformer is constructed, the multi-head attention mechanism of the Transformer neural network is used as the first layer of the network, and the Encoder is used as the second layer to obtain the global dependency of the data and the hidden dependency between the context of the continuous data stream; in order to avoid the gradient disappearance and gradient explosion phenomenon caused by deepening the model layer, a residual skip connection structure is introduced to maintain the features extracted by the neural network; in order to make up for the short board of the Transformer neural network in capturing local spatial features, a convolution feature extraction submodule is constructed, which includes two two-dimensional convolution layers, two maximum pooling layers and two fully connected layers, which can effectively extract the key local features of the data; The processed data is divided into a training set, a test set and a verification set in a ratio of 7:2:1 according to different categories, the model input is encrypted data, and the label data is the original data before encryption, that is, the target output of the neural network; The constructed neural network is trained on the training data set and the verification data set, the optimizer is selected as the Adam optimizer, and the loss function is selected as the cross entropy loss function (Cross Entropy Loss); after the model training is completed, the performance test is completed on the test set, and the model with good test performance is selected for deployment.

[0020] The above differences between different sample data and data sources have a great relationship, for example, different applications have different sizes, different functions, different use frequencies, resulting in a large gap between data samples belonging to different applications; The above-mentioned Adam optimizer is an effective optimizer in neural network training, which adjusts the learning rate of each parameter by calculating the first moment (mean) and second moment (uncentered variance) of the gradient, and is suitable for processing non-stationary objective functions and very large data sets or parameter quantities.

[0021] Further, the abnormal traffic efficient detection method based on big data analysis is combined with big data analysis and deep learning technology to quickly and quickly troubleshoot traffic that does not belong to normal behavior patterns in a running system, so that system managers can locate network problems, timely repair system vulnerabilities, and maintain the normal operation of the system. The specific steps include: S40: Construct an abnormal traffic identification neural network combining one-dimensional convolutional neural network (1D CNN) and BERT neural network model, specifically including two one-dimensional convolutional layers, two pooling layers, one BERT layer, one fully connected layer, and finally an output layer, respectively learning abnormal traffic data features from time and space dimensions, and realizing efficient and accurate detection of abnormal traffic; S41: Connect a normalization function after each max pooling layer to speed up neural network convergence and improve model generalization ability, and the calculation formula is: ; ; ; wherein, represents the input value for a certain training batch, m represents the total amount of data in a batch, is an arbitrarily small value in a mathematical sense, and are affine parameters.

[0022] S42: Data collection, collect abnormal traffic data through public abnormal traffic detection data set or crawler algorithm as the big data basis of the technology; S43: Data cleaning, the original data collected in step S42 is usually stored in the form of PCAP file, data cleaning mainly filters the error information and missing values in it, and removes some useless information such as domain name service DNS for addressing, reducing the data sample set; S44: Data labeling, convert the PCAP file used for training into a time series session group through the pkt2flow tool to separate different flows into different files; after classifying the PCAP file, extract the corresponding features according to the type of different flows and label them, finally form a CSV format file; S45: Data format conversion of CSV file through one-hot encoding to ensure that all data in CSV file have uniform format measurement; S46: Data normalization, compress and convert the data to limit it within a certain range to speed up the algorithm convergence speed, and the compression calculation formula is: ;; wherein, represents the value of the i-th feature component, represents the compressed value of the i-th feature component, ranging from 0 to 1, represents the minimum value of the i-th feature component, represents the maximum value of the i-th feature component.

[0023] S47: The data processed in step S46 is divided into a training set, a validation set and a test set according to the proportion and different abnormal traffic categories, and the neural network model constructed in steps S40 and S41 is trained on the training set and the validation set, wherein the model input is the processed data, and the label is whether it is abnormal traffic or the abnormal traffic category. After the model training on the training set and the validation set is completed, the model performance test on the test set is completed.

[0024] In another aspect, an IPv6 traffic monitoring platform based on big data analysis is provided for any of the IPv6 traffic monitoring methods based on big data analysis, and the IPv6 traffic monitoring platform based on big data analysis comprises: IPv6 traffic acquisition subsystem: The IPv6 traffic acquisition subsystem is built-in with a dynamic traffic acquisition method based on big data, which acquires dynamic traffic in real time by deploying NetFlow, sFlow or IPFIX acquisition tools in the data transmission network, analyzes the content of the data packet by deploying traffic monitoring tools at key traffic nodes, and analyzes log files to obtain detailed information of IPv6 traffic; Traffic encryption subsystem: The traffic encryption subsystem provides hardware support for the IPv6 security encryption method, provides the basic computing power required for the security traffic encryption algorithm, provides the necessary data storage space for traffic and encryption passwords, and realizes the secure transmission of encrypted traffic through various layers of transmission protocols; Intelligent algorithm processing platform: The intelligent algorithm processing platform is built-in with a large computing power GPU board, which provides necessary computing power support for intelligent computing algorithms and neural network model training and testing in the encrypted traffic recognition and abnormal traffic detection methods, and supports basic data preprocessing; IPv6 traffic visualization monitoring platform: The IPv6 traffic visualization monitoring platform is built-in on the display of a computer device, realizes front-end interface design and function module editing through Qt, Pycharm and MATLAB software compilation platforms, and realizes real-time display of traffic monitoring results including traffic information, traffic recognition results and abnormal traffic detection results by combining digital twin technology.

[0025] Compared with the prior art, the present application has the following advantages: 1. The present application proposes an IPv6 traffic acquisition method based on big data acquisition and analysis technology, which can effectively acquire IPv6 traffic data at key nodes and preliminarily analyze and distinguish data content and other information, providing an important basis for subsequent traffic recognition and abnormal traffic detection; 2. The present application proposes an IPv6 security encryption method, which realizes fast and secure encryption of IPv6 traffic through the joint mode of RSA encryption algorithm and 3DES encryption algorithm, and guarantees the safety of IPv6 traffic acquisition and transmission process; 3. The application provides an IPv6 encrypted traffic recognition method based on traffic characteristics and behaviors, which realizes intelligent and accurate recognition of encrypted traffic by combining intelligent computing algorithms, and solves the problems of low recognition accuracy and slow speed in existing encrypted traffic recognition. 4. The application provides an abnormal traffic efficient detection method based on big data analysis, which realizes accurate and efficient detection of abnormal traffic in a large amount of IPv6 data under the dual assistance of data and computing power by combining deep learning technology, and provides protection for network traffic security. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The application provides an IPv6 traffic monitoring method based on big data analysis;

[0027] Figure 2 The application provides an IPv6 traffic monitoring platform structure based on big data analysis; DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the application.

[0029] In the description of the application, the terms "first", "second" are used only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first", "second" can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0030] In the description of the application, the term "for example" is used to indicate "as an example, illustration or explanation". Any embodiment described as "for example" in the application is not necessarily interpreted as more preferred or more advantageous than other embodiments. The following description is given to enable any person skilled in the art to implement and use the application. In the following description, details are listed for the purpose of explanation. It should be understood that those skilled in the art can realize the application without using these specific details. In other examples, well-known structures and processes will not be described in detail to avoid unnecessary details making the description of the application obscure. Therefore, the application is not intended to be limited to the shown embodiments, but is consistent with the broadest scope of principles and features disclosed in the application.

[0031] The embodiment of the application discloses an IPv6 flow monitoring method based on big data analysis, and specifically refers to the accompanying Figure 1 The application is applied to an IPv6 flow monitoring platform based on big data analysis, and comprises a dynamic flow acquisition method based on big data, an IPv6 secure flow encryption method, an IPv6 encrypted flow identification method based on flow characteristics and behaviors, and an abnormal flow efficient detection method based on big data analysis.

[0032] The dynamic flow acquisition method based on big data is the data basis of the IPv6 flow monitoring method based on big data analysis, and acquires IPv6 flow data such as video flow, voice flow and game flow from different networks through big data technology, thereby providing support for subsequent flow intelligent identification and prediction. The IPv6 secure flow encryption method is mainly used for solving the security problem of large-scale IPv6 flow data acquired by big data during transmission in a network and storage in a node, and reducing the probability of data leakage and network attack malicious events as much as possible. The IPv6 encrypted flow identification method based on flow characteristics and behaviors is mainly used for identifying various encrypted flow types and characteristics in an IPv6 network. The abnormal flow efficient detection method based on big data analysis is combined with big data analysis and deep learning technology, and can timely and quickly troubleshoot flow not belonging to a normal behavior mode in a running system, so that a system manager can locate network problems, timely repair system vulnerabilities and maintain normal operation of the system.

[0033] The application further provides an IPv6 flow monitoring platform based on big data analysis, which comprises an IPv6 flow acquisition subsystem, a flow encryption subsystem, an intelligent algorithm processing platform and an IPv6 flow visualization monitoring platform. Figure 2 The IPv6 flow acquisition subsystem is internally provided with the dynamic flow acquisition method based on big data, and acquires dynamic flow in real time through deployment of a NetFlow, sFlow or IPFIX acquisition tool in a data transmission network.

[0034] ​The traffic encryption subsystem provides hardware support for the IPv6 security encryption method, provides the basic computing power required for the security traffic encryption algorithm, provides the necessary data storage space for traffic and encryption passwords, and realizes the safe transmission of encrypted traffic through various layers of transmission protocols. The intelligent algorithm processing platform is built-in with a large computing power GPU board, which provides the necessary computing power support for the intelligent computing algorithm and neural network model training and testing in the encrypted traffic recognition and abnormal traffic detection method, and supports the realization of basic data preprocessing. The IPv6 traffic visualization monitoring platform is built-in on the computer device display, realizes the front-end interface design and function module editing through Qt, Pycharm, MATLAB software compilation platform, and combines digital twin technology to realize real-time display of traffic monitoring results, including traffic information, traffic recognition results and abnormal traffic detection results.

[0035] In specific embodiments, the IPv6 traffic monitoring engineer deploys traffic monitoring tools at key traffic nodes to obtain IPv6 traffic data from different devices and applications through the dynamic traffic acquisition method based on big data built-in the IPv6 traffic acquisition subsystem, including video traffic, voice traffic, and game traffic from various devices and applications, and performs data cleaning, format conversion and deduplication on the captured traffic data to improve the quality of IPv6 traffic data. Finally, the high-quality traffic data is stored in the system, a complete indexing and partitioning strategy is established, and the real-time, intuitive and visual monitoring of IPv6 traffic is realized through digital twin technology and Tableau visualization tools; To ensure the security of IPv6 traffic data transmission in the network and storage in the node, reduce the risk of data leakage when the system is attacked, the engineer uses the IPv6 security encryption method built-in the traffic encryption subsystem to securely encrypt the IPv6 traffic data in the network, and transmits and stores the encrypted data in the network; To further realize accurate monitoring and control of IPv6 traffic in the network, the engineer completes data preprocessing, intelligent computing algorithm training and testing based on the encrypted traffic recognition method and the support of the intelligent algorithm processing platform, and deploys the well-performing model in the platform to accurately identify encrypted traffic information, including traffic address, traffic type, traffic content and source device information in encrypted traffic; To effectively deal with abnormal traffic attacks in the network, discover attack types and abnormal situations as early as possible, reduce the probability of false alarm and false report, and take timely defense measures, the engineer builds a neural network model through the abnormal traffic detection method, realizes real-time detection of abnormal traffic based on the intelligent algorithm processing platform, accurately discovers, identifies and locates abnormal traffic, and improves the stability and security of network traffic transmission.

[0036] Embodiment 1

[0037] In one embodiment, the traffic monitoring engineer collects large-scale IPv6 network traffic data through big data technology, providing data sources and guarantees for subsequent IPv6 traffic monitoring. First, the engineer determines the network transmission architecture through the system hardware architecture diagram, including one switch, two routers, four terminal devices and servers. The switch is the overall data source for system network data exchange. The two routers are connected to the switch and serve as two data distribution and relay interfaces. The four terminal devices and servers are connected to the two routers through optical cables to obtain traffic data transmission links. Various commonly used software applications are installed on the four terminal devices and servers, and a firewall and intrusion detection system are also installed. The four terminal devices can realize data interaction through optical cables and routers. Thus, the engineer can determine that the possible entry and exit points of IPv6 traffic are the switch and external network data exchange connection nodes, the switch and two router data exchange nodes, and the two routers and four terminal devices and server data exchange nodes. According to the above hardware system architecture analysis, the engineer maintains and updates the IP address database of different devices, optimizes the allocation and use of IPv6 addresses, and facilitates the tracking and positioning of IPv6 traffic source and target addresses.

[0038] Subsequently, the engineer deploys traffic monitoring tools at the above key traffic nodes, uses deep packet inspection technology to analyze the content of data packets, and analyzes log files to obtain detailed information of IPv6 traffic. Further analysis and determination of the source of the traffic are made using protocol obfuscation recognition and geographic location positioning techniques. To ensure the accuracy of data source positioning, the engineer correlates and analyzes various collected data and passes the verification and testing. After completing the tracking of captured IPv6 traffic addresses, the engineer configures and deploys NetFlow, sFlow and IPFIX collection tools at the above key nodes to capture IPv6 network traffic data in real time, converts it into standardized traffic records, and ensures the integrity and accuracy of the data. After completing data collection and capture, to ensure the effectiveness and availability of the collected data, the engineer pre-processes the raw traffic data, including data cleaning, format conversion and deduplication, to ensure data quality. Key fields related to IPv6 traffic are extracted, including source IP address, destination IP address, protocol type, packet size and timestamp.

[0039] Since the traffic monitoring task requirements are different in different scenarios and time periods, engineers need to store the collected and processed IPv6 network data. Usually, engineers select appropriate storage solutions, such as distributed file systems (e.g., Hadoop HDFS), databases (e.g., MongoDB, Cassandra), or data warehouses (e.g., Hive, SparkSQL), according to data volume and processing requirements, to ensure that the storage solution can support large data volume storage and efficient data access. After determining the storage solution, engineers import the preprocessed IPv6 traffic data into the selected storage system and establish indexing and partitioning strategies to improve data query, call, and analysis efficiency. Finally, to facilitate real-time and rapid monitoring of IPv6 traffic, engineers use digital twinning technology and data visualization tools such as Tableau or Power BI to present the real-time IPv6 traffic data in the form of charts and reports, making the IPv6 traffic data capture more intuitive and ensuring that the data obtained can support subsequent traffic identification.

[0040] Embodiment 2

[0041] In one embodiment, to ensure the security of IPv6 traffic data transmission and storage in the network, engineers use an IPv6 secure traffic encryption method to achieve IPv6 traffic data security encryption. Engineers use the RSA encryption algorithm to generate a pair of encryption keys, including a public key and a private key. Engineers use the public key, i.e., the triple data encryption key, to encrypt the symmetric key. To ensure security, engineers reserve a 24-byte length space to generate the triple data encryption key. The private key is used to decrypt the symmetric key. Subsequently, engineers randomly select an initialization vector with a length of 8 bytes for the CBC mode of triple data encryption. After selecting the initialization vector, engineers split the plaintext data to be encrypted according to the block size of the triple data encryption algorithm, obtaining multiple data blocks. Engineers use the generated 3DES key and IV to perform 3DES encryption on each data block using the EDE mode (encryption-decryption-encryption) or EEE mode (encryption-encryption-encryption) through three DES encryption operations, obtaining ciphertext data blocks. Then, engineers encrypt the 3DES key and IV using the RSA public key. After encryption, the key and IV ciphertext are obtained. Finally, engineers combine the 3DES encrypted ciphertext data blocks and the RSA encrypted key and IV ciphertext to form the final encrypted data packet. The encrypted data packet includes the order information of the ciphertext data blocks, the ciphertext data blocks themselves, the RSA encrypted key ciphertext, and the IV ciphertext.

[0042] Embodiment 3

[0043] In one embodiment, the engineer identifies the encrypted traffic based on the traffic characteristics and behaviors through an IPv6 encrypted traffic identification method based on different applications and services, extracts and analyzes various encrypted traffic types and their characteristics in the IPv6 network, and obtains more information about the transmission data characteristics, so as to realize accurate identification of encrypted traffic, and the specific steps include:

[0044] The engineer analyzes the collected data packets, extracts key information including source IP address, destination IP address, source port, destination port, protocol type, packet size, and packet timestamp, and then performs de-duplication and filtering processing on the data packets to remove irrelevant or redundant information; Then, the length, interval, and frequency statistical characteristics of the data packets are analyzed, the flag bit and checksum field information of the data packets are extracted, the transmission layer protocol and application layer protocol information of the data packets are analyzed, the behavior characteristics are extracted, the transmission mode of the data packets is analyzed, including whether there is burst transmission or periodic transmission, the behavior of network connection is measured, including connection establishment, disconnection, and retransmission behavior, and the directionality of network connection is analyzed, including whether there is bidirectional communication or unidirectional communication; A decision tree intelligent computing model is constructed, known types of encrypted traffic data are used as a training set to train the model, and in the training process, the parameters and feature selection of the model are constantly adjusted to improve the recognition accuracy of the model; The model is tested using test set data to evaluate the recognition performance of the model, and the test set data includes different types of encrypted traffic and normal traffic to verify the generalization ability of the model; According to the test results, the model is optimized, including adjusting the feature weight, adding new features, and improving the model structure, and the test and optimization process is repeated until the model reaches satisfactory recognition performance; finally, the trained model is used to identify encrypted network traffic.

[0045] Embodiment 4

[0046] In one embodiment, the engineer uses an intelligent identification method for IPv6 encrypted traffic based on machine learning to identify various types of encrypted traffic and their characteristics in IPv6 networks, including P2P (peer-to-peer) traffic, VoIP (network phone) traffic, video traffic, and other application traffic, to solve the problem of low accuracy and low efficiency of traditional port-based and deep packet inspection traffic identification methods caused by the widespread deployment and application of IPv6 protocol, the significant increase in complexity and diversity of network traffic, and the widespread use of traffic encryption algorithms. The engineer first cleans the acquired encrypted PCAP format raw traffic data, uses the editcap tool built into the Wireshark software to remove invalid SSDP, LLMNR, DNS, ARP, NetBIOS, ICMP, IGMP protocol packets in the PCAP file to avoid affecting the accuracy of traffic identification; secondly, the raw data packets after filtering the protocol packets are aggregated into flows according to the five-tuple information, which are used as target objects for feature extraction. The clear text information in the flow that can be used as features includes source IP address, destination IP address, source port, destination port, transmission layer protocol, packet length, window value, and flag bit; the frame length and window value of each packet in a flow are extracted to form a packet length sequence and a window sequence , and then and are combined into the final input data sequence; then the data is enhanced for encrypted traffic data, the first step is to use the k-nearest neighbor algorithm to randomly select K nearest neighbor samples, then synthesize new samples according to the synthesis formula and add them to the sample set, and repeat the above process until the number or class reaches balance; the data after data enhancement is mapped to a high-dimensional vector space, and is converted into a continuous vector through vector embedding, so that the neural network can discover more detailed features;

[0047] After the data preparation is completed, the engineer builds a neural network model TrCNN that combines convolutional neural networks and Transformers. The multi-head attention mechanism of the Transformer neural network is used as the first layer of the network, and the Encoder encoder is used as the second layer to obtain the global dependency of the data and the hidden dependency between the context of the continuous data stream; in order to avoid the gradient vanishing and gradient explosion phenomenon caused by deepening the model layers, a residual skip connection structure is introduced to maintain the features extracted by the neural network; in order to make up for the short board of the Transformer neural network in capturing local spatial features, a convolution feature extraction submodule is constructed, which includes two two-dimensional convolution layers, two max pooling layers, and two fully connected layers, which can effectively extract key local features of the data;

[0048] The engineer divides the previously prepared data into training set, test set and validation set according to different categories in the ratio of 7:2:1, the model input is the encrypted data, the label data is the original data before encryption, that is, the target output of the neural network; the optimizer selects Adam optimizer, the loss function selects cross entropy loss function (Cross Entropy Loss), and the model training is carried out; after the model training is completed, the performance test is completed on the test set, and the model with good test performance is selected for deployment.

[0049] Embodiment 5

[0050] In one embodiment, since the IPv6 network may be subject to abnormal traffic malicious attacks, the traditional algorithm often ignores the timing characteristics when detecting abnormal traffic, the detection accuracy is low and the traffic data processing is difficult, in order to ensure the safety of the traffic information in the network, the engineer combines big data analysis and deep learning technology to design an IPv6 traffic monitoring method based on big data analysis. First, a one-dimensional convolutional neural network (1D CNN) and a BERT neural network model are combined to construct an abnormal traffic identification neural network, which specifically includes two one-dimensional convolutional layers, two pooling layers, a BERT layer, a fully connected layer and a final output layer, which respectively learns the characteristics of abnormal traffic data from two dimensions of time and space, realizes efficient and accurate detection of abnormal traffic; at the same time, a normalization function is connected after each max pooling layer to accelerate the convergence of the neural network and improve the generalization ability of the model, and the calculation formula is: ; ; ; Among them, represents the input value in a certain training batch, m represents the total amount of data in a batch, is an arbitrarily small value in a mathematical sense, and are affine parameters;

[0051] After the model design is completed, data collection begins, engineers collect abnormal traffic data through public abnormal traffic detection data sets or crawler algorithms as the big data basis of this technology; the collected raw data is cleaned, mainly filtering the error information and missing values in it, and removing some useless information such as domain name service DNS for addressing, reducing the data sample set; then, the PCAP file used for training is converted into a time series session group through the pkt2flow tool to separate different flows into different files; after classifying the PCAP file, the corresponding features are extracted according to the type of different flows and labeled, and finally a CSV format file is formed; further, the data format conversion of the CSV file is carried out through one-hot encoding to ensure that all data in the CSV file have a unified format measurement; then the data is compressed and converted, so as to limit the data within a certain range to speed up the algorithm convergence speed, and the compression calculation formula is: ; wherein, represents the value of the i-th feature component, represents the value of the i-th feature component after compression, ranging from 0 to 1, represents the minimum value of the i-th feature component, represents the maximum value of the i-th feature component; Finally, the engineers divide the processed data into training set, validation set and test set according to the proportion and different types of abnormal traffic, and complete the model training on the training set and validation set with the neural network model constructed in steps S40 and S41, wherein the model input is the processed data, the label is whether it is abnormal traffic or the type of abnormal traffic, and the model performance test is completed on the test set after the model training on the training set and validation set.

[0052] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all changes and modifications falling within the scope of the present application.

[0053] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A big data analysis-based IPv6 traffic monitoring method, comprising a big data-based dynamic traffic acquisition method, an IPv6 secure traffic encryption method, an IPv6 encrypted traffic identification method based on traffic characteristics and behavior, and an abnormal traffic efficient detection method based on big data analysis, characterized in that, The dynamic traffic acquisition method based on big data is the data basis of the IPv6 traffic monitoring method based on big data analysis, and the IPv6 traffic data from different networks is acquired through big data technology to support subsequent intelligent traffic identification and prediction. The specific steps of the dynamic traffic acquisition method based on big data include: S10: Determine the data source, obtain the current network transmission architecture, including the layout and configuration of network equipment, security equipment and application servers, and determine the entry and exit points of IPv6 traffic; S101: Maintain and update the IP address database, including the allocation and use of IPv6 addresses, track and locate the source and target addresses of IPv6 traffic through the IP address management system; S102: Deploy traffic monitoring tools at key traffic nodes, use deep packet inspection technology to analyze the content of data packets, and analyze log files to obtain detailed information of IPv6 traffic; S103: Use protocol obfuscation identification and geographic location positioning technology to further analyze and determine the source of the traffic; S104: Correlation analysis of the collected data, and verification and testing to ensure the accuracy and reliability of the analysis results; S11: Deploy collection tools, including NetFlow, sFlow or IPFIX, configure the collection tools to capture IPv6 network traffic data in real time, convert it into standardized traffic records, and ensure the integrity and accuracy of the data; S12: Data preprocessing, preprocessing the collected raw traffic data, including data cleaning, format conversion and deduplication, to ensure data quality; extract key fields related to IPv6 traffic, including source IP address, destination IP address, protocol type, packet size, timestamp; S13: Select a storage solution, select an appropriate storage solution based on data volume and processing requirements to ensure that the storage solution can support large data volume storage and efficient data access; S14: Data import and storage, import the preprocessed IPv6 traffic data into the selected storage system, and establish indexing and partitioning strategies to improve data query, call and analysis efficiency; S15: Visualization processing, using data visualization tools through digital twin technology to more intuitively reflect the IPv6 traffic data acquisition situation, ensuring that the acquired data can support subsequent traffic identification. 2.The IPv6 traffic monitoring method based on big data analysis of claim 1, wherein, The IPv6 secure traffic encryption method is mainly used to solve the security problem of large-scale IPv6 traffic data obtained by big data during transmission in the network and storage in the node, and to reduce the probability of data leakage and network attack malicious events as much as possible. The specific steps include: S20: Key generation, generate a pair of keys using RSA encryption algorithm, including public key and private key, public key for symmetric key encryption, that is, triple data encryption key, private key for symmetric key decryption; S21: Selection of initialization vector, initialization vector for 3DES encryption CBC mode, initialization vector length is usually 8 bytes, and must be random to ensure the randomness and security of encryption; S22: plaintext data segmentation, the plaintext data to be encrypted is segmented according to the block size of the triple data encryption algorithm, and a plurality of data blocks are obtained; S23: 3DES encryption, using the generated 3DES key and IV, each data block is 3DES encrypted, the 3DES encryption process includes three times of DES encryption operation, and after encryption, a ciphertext data block is obtained; S24: RSA encryption key and IV, the 3DES key and IV are encrypted using the RSA public key, and after encryption, the ciphertext of the key and IV is obtained; S25: combination of ciphertext and encrypted key, the 3DES encrypted ciphertext data block and the RSA encrypted key and IV ciphertext are combined to form the final encrypted data packet. 3.The IPv6 traffic monitoring method based on big data analysis of claim 1, wherein, The IPv6 encrypted traffic identification method based on traffic characteristics and behaviors is mainly used to identify various encrypted traffic types and their characteristics in IPv6 networks; In the identification process, different application programs and services have specific characteristics when transmitting traffic. By analyzing the traffic characteristics, more information about the transmission data characteristics can be obtained, thereby realizing accurate identification of encrypted traffic. 4.The IPv6 traffic monitoring method based on big data analysis of claim 1, wherein, The IPv6 encrypted traffic identification method based on traffic characteristics and behaviors, the specific steps include: S30: data preprocessing, the collected data packets are parsed to extract key information, including source IP address, destination IP address, source port, destination port, protocol type, packet size, packet timestamp, and then the data packets are de-duplicated and filtered to remove irrelevant or redundant information; S31: traffic characteristic extraction, analyze the length, interval, and frequency statistical characteristics of the data packet, extract the flag bit, checksum field information of the data packet, and analyze the transmission layer protocol and application layer protocol information of the data packet; S32: behavior characteristic extraction, analyze the transmission mode of the data packet, including whether there is burst transmission or periodic transmission; measure the behavior of network connection, including connection establishment, disconnection, and retransmission behavior; analyze the directionality of network connection, including whether there is bidirectional communication or unidirectional communication; S33: construct a decision tree intelligent computing model, use known types of encrypted traffic data as a training set to train the model, and constantly adjust the parameters and feature selection of the model during the training process to improve the recognition accuracy of the model; S34: test the model using test set data to evaluate the recognition performance of the model, the test set data includes different types of encrypted traffic and normal traffic to verify the generalization ability of the model; S35: according to the test results, optimize the model, including adjusting the feature weight, adding new features, and improving the model structure, repeat the test and optimization process until the model reaches satisfactory recognition performance; S36: use the trained model to identify encrypted network traffic. 5.The IPv6 traffic monitoring method based on big data analysis of claim 1, wherein, The abnormal traffic efficient detection method based on big data analysis includes the following specific steps: S40: Construct an abnormal traffic identification neural network combining one-dimensional convolutional neural network and BERT neural network model, specifically including two one-dimensional convolutional layers, two pooling layers, a BERT layer, a fully connected layer and a final output layer, respectively learning abnormal traffic data features from time and space dimensions to realize efficient and accurate detection of abnormal traffic; S41: Connect a normalization function after each max pooling layer to speed up neural network convergence and improve model generalization ability, and the calculation formula is: wherein, represents the total amount of data in a batch for the input value in a certain training batch, is an arbitrarily small value in a mathematical sense, and is an affine parameter; S42: Data collection, collect abnormal traffic data through public abnormal traffic detection data set or crawler algorithm as the big data basis of this technology; S43: Data cleaning, the original data collected in step S42 is usually stored in the form of PCAP file, and data cleaning mainly filters the error information and missing values in it, and removes some useless information; S44: Data labeling, the PCAP file used for training is converted into a time series session group by the pkt2flow tool to separate different flows into different files; after classifying the PCAP file, the corresponding features of different flows are extracted according to the type and labeled, and finally a CSV format file is formed; S45: Data format conversion of CSV file through one-hot encoding to ensure that all data in CSV file have uniform format measurement; S46: Data normalization, compress and convert the data to limit it within a certain range to speed up the algorithm convergence speed, and the compression calculation formula is: wherein, represents the value of the i-th feature component, represents the value of the i-th feature component after compression, ranging from 0 to 1, represents the minimum value of the i-th feature component, represents the maximum value of the i-th feature component. S47: Divide the data processed in step S46 into training set, validation set and test set according to proportion and different abnormal traffic categories, and complete model training on training set and validation set using the neural network model constructed in steps S40 and S41, wherein the model input is the processed data, the label is whether it is abnormal traffic or abnormal traffic category, and after completing model training on training set and validation set, complete model performance test on test set. 6.A big data analysis based IPv6 traffic monitoring platform, characterized in that, The IPv6 flow monitoring platform based on big data analysis is realized by using the IPv6 flow monitoring method based on big data analysis in any one of claims 1-5, and the IPv6 flow monitoring platform based on big data analysis comprises: IPv6 flow acquisition subsystem: the flow acquisition subsystem is built-in with a dynamic flow acquisition method based on big data, which acquires dynamic flow in real time by deploying NetFlow, sFlow or IPFIX acquisition tools in the data transmission network, analyzes the content of data packets by deploying flow monitoring tools in key flow nodes, and analyzes log files to obtain detailed information of IPv6 flow. 7.The IPv6 traffic monitoring platform based on big data analysis of claim 6, wherein, Further comprising: Traffic encryption subsystem: the traffic encryption subsystem provides hardware support for IPv6 security encryption method, provides basic computing power required by security flow encryption algorithm, provides necessary data storage space for flow and encryption password, and realizes safe transmission of encrypted flow through various layer transmission protocols. 8.The IPv6 traffic monitoring platform based on big data analysis of claim 7, wherein, Further comprising: Intelligent algorithm processing platform: The intelligent algorithm processing platform is built-in with a large computing power GPU board card, which provides necessary computing power support for the training and testing of intelligent computing algorithms and neural network models in the encryption traffic identification and abnormal traffic detection methods, and supports the realization of basic data preprocessing; IPv6 traffic visualization monitoring platform: The IPv6 traffic visualization monitoring platform is built-in on the computer device display, and the front-end interface design and function module editing are realized through Qt, Pycharm, MATLAB software compilation platform, and the traffic monitoring results are displayed in real time through digital twin technology, including traffic information, traffic identification results and abnormal traffic detection results.

Images