A method, system and device for collecting high-speed network traffic based on a cloud environment

Abstract

The present invention discloses a network high-speed traffic collection method, system and device based on a cloud environment, which relate to the field of network communication technology. The present invention creates a cloud environment, utilizes an optimized FCN network high-speed traffic prediction model and real-time load indicators, and constructs a two-level cloud environment expansion and contraction strategy to achieve resource pre-allocation and precise adjustment. By introducing an optimization algorithm to dynamically search for the optimal solution of the model learning rate, a consistent hashing algorithm is adopted to ensure session continuity, and three-level dynamic filtering rules and lightweight DPI technology are combined to improve load balancing and processing throughput. The ring buffer management strategy and Kafka message queue synchronization mechanism reduce the key data loss rate and data synchronization delay, and meet the real-time analysis and long-term storage requirements in high-speed network scenarios. The present invention effectively solves the problem of insufficient resource elasticity in the cloud environment, improves the resource supply response speed, and reduces costs and prediction error rates.

Description

Technical Field

[0001] The present invention belongs to the field of network communication technology, and more specifically, relates to a method, system and device for collecting network high-speed traffic based on a cloud environment. Background Art

[0002] With the rapid development of cloud computing technology, cloud environments, with their flexible resource scheduling, virtualized isolation, and high availability, have gradually become the core infrastructure for high-speed network traffic collection and analysis. By dynamically allocating computing, storage, and network resources, cloud environments can provide flexible support for large-scale network traffic processing. However, in high-speed network scenarios (such as 5G, the Internet of Things, and the Industrial Internet), traffic volumes are growing exponentially. Traditional traffic collection methods based on physical devices face performance bottlenecks and struggle to meet the requirements of real-time performance, dynamic scalability, and resource utilization. Therefore, achieving efficient and intelligent high-speed network traffic collection and processing in cloud environments has become a key research challenge.

[0003] Chinese patent CN117294533B discloses a network traffic collection method and system based on a cloud environment. The present invention creates a mirrored virtual switch on a computing node and diverts the network traffic of the monitoring node to the switch through its patch port. Subsequently, the traffic is forwarded to a deployed virtualized instance agent node, where the network traffic data is converted into a file format and analyzed in real time. The analyzed data is further forwarded to a data security monitoring platform to achieve real-time monitoring of events. At the same time, the flow table of the virtual switch is dynamically adjusted according to monitoring strategies and event changes.

[0004] Existing technologies have problems such as insufficient resource elasticity in cloud environments, lack of traffic prediction mechanisms, limited traffic distribution and processing efficiency, and poor data aggregation and synchronization performance. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the problems in the related technologies, the present invention provides a network high-speed traffic collection method based on a cloud environment. The present invention solves the problems of insufficient resource elasticity in the cloud environment, lack of traffic prediction mechanism, limited traffic distribution and processing efficiency, and poor data aggregation and synchronization performance through a two-level dynamic expansion and contraction mechanism, deep learning technology, and a dynamic filtering mechanism.

[0007] (2) Technical solution

[0008] To solve the above technical problems, the present invention is achieved through the following technical solutions:

[0009] S1. Build a cloud environment and set its parameters.

[0010] S2. Mirror the real-time network high-speed traffic to the cloud environment and perform feature extraction to obtain real-time network high-speed traffic features;

[0011] S3. Build an FCN prediction model and use historical network high-speed traffic feature data and an optimization algorithm to improve the FCN prediction model. This results in an optimized FCN network high-speed traffic prediction model. Combined with real-time network high-speed traffic features, this model predicts network high-speed traffic. Based on the predicted network high-speed traffic, a decision is made on whether to perform the first cloud environment expansion or contraction, resulting in the cloud environment after the first expansion or contraction.

[0012] S4. Collect load indicator data of the cloud environment after the first expansion and contraction, obtain real-time load indicators, and determine whether to perform a second expansion and contraction of the cloud environment, thereby obtaining the cloud environment after the second expansion and contraction.

[0013] S5. Collect the network high-speed traffic packets of the cloud environment after the second expansion and contraction, and perform dynamic filtering processing to obtain a pre-processed network high-speed traffic data set;

[0014] S6. Aggregate and synchronize the pre-processed network high-speed traffic data set to obtain a network high-speed traffic data stream.

[0015] Preferably, the S1 comprises the following steps:

[0016] S11. Create a virtualized resource pool in the cloud platform and build a collection node cluster based on Kubernetes; configure an elastic scaling policy for the resource pool; the elastic scaling policy of the resource pool changes based on the predicted size of the network high-speed traffic and the real-time load rate;

[0017] Allocate a dedicated virtual network interface to each collection node in the resource pool and enable SR-IOV technology to improve network card performance;

[0018] S12. Deploy the SDN controller on the cloud gateway, configure the network high-speed traffic mirroring rules, and mirror the target network high-speed traffic to the collection system; set the bandwidth limit for the mirrored network high-speed traffic to avoid overload;

[0019] S13. Configuring an initial acquisition strategy in the control center. The initial acquisition strategy includes a network high-speed traffic distribution weight and a resource threshold for each node. The network high-speed traffic distribution weight is dynamically adjusted based on the node's latency. The resource threshold limits the maximum and minimum number of node clusters.

[0020] S14, obtain cloud environment through S11, S12, and S13;

[0021] The above steps build an efficient and flexible cloud environment by creating a virtualized resource pool in the cloud platform, building a collection node cluster based on Kubernetes, configuring elastic scaling policies, allocating dedicated virtual network interfaces and enabling SR-IOV technology, deploying an SDN controller and configuring network high-speed traffic mirroring rules, and configuring the initial collection policy in the control center.

[0022] Preferably, said S2 comprises the following steps:

[0023] S21. Mirror the real-time network high-speed traffic to the cloud environment to obtain the real-time mirrored network high-speed traffic;

[0024] S22. Extracting a five-tuple feature dataset and application layer content features of the real-time mirrored network high-speed traffic through lightweight DPI to obtain real-time network high-speed traffic features; the five-tuple feature dataset includes the source IP address, destination IP address, protocol type, source port, and destination port, and the application layer content features reflect the priority of the network high-speed traffic data; using a consistent hashing algorithm, mapping the real-time network high-speed traffic to a collection node cluster according to the hash value of the five-tuple feature dataset;

[0025] The above steps implement an efficient and orderly real-time network high-speed traffic processing mechanism by mirroring real-time network high-speed traffic to a cloud environment, using lightweight DPI technology to extract key feature data, and adopting a consistent hashing algorithm for traffic mapping. This effectively improves the processing efficiency of network high-speed traffic, ensures the orderliness of data processing and the continuity of sessions, and lays the foundation for subsequent real-time analysis and long-term storage.

[0026] Preferably, the step S3 includes the following steps:

[0027] S31. Build the FCN prediction model and set the initial learning rate of the FCN prediction model q ;

[0028] S32. Collect characteristic data of historical network high-speed traffic in the cloud environment to obtain historical network high-speed traffic characteristic data;

[0029] S33, setting a maximum number of training iterations and a training accuracy threshold; inputting the historical network high-speed traffic characteristic data into the FCN prediction model for training to obtain a training accuracy; adjusting a learning rate of the FCN prediction model according to the training accuracy; and obtaining an FCN network high-speed traffic prediction model when the training accuracy is greater than or equal to the training accuracy threshold or the maximum number of training iterations is reached;

[0030] S34, using an optimization algorithm to find a learning rate of the FCN network high-speed traffic prediction model to obtain an optimal solution; using the optimal solution as the learning rate of the FCN network high-speed traffic prediction model to obtain an optimized FCN network high-speed traffic prediction model;

[0031] S35, inputting the real-time network high-speed traffic characteristic data into the optimized FCN network high-speed traffic prediction model to obtain the predicted network high-speed traffic;

[0032] S36, set the predicted network high-speed traffic expansion and contraction rules, and perform the first cloud environment expansion and contraction or keep it unchanged according to the predicted network high-speed traffic expansion and contraction rules, the real-time network high-speed traffic and the predicted network high-speed traffic, to obtain the cloud environment after the first expansion and contraction; the predicted network high-speed traffic expansion and contraction rules are to set the predicted network high-speed traffic expansion threshold d , predict network high-speed traffic reduction threshold f ; When the ratio of predicted network high-speed traffic to current network high-speed traffic e ≥Predicted network high-speed traffic expansion threshold d , expand the collection node group in the cloud environment to the current collection node group times; when the ratio of predicted network high-speed traffic to current network high-speed traffic g ≤Predicted network high-speed traffic reduction threshold f , reducing the collection node group in the cloud environment to the current collection node group times;

[0033] The above steps achieved the first expansion and contraction of the cloud environment by building and optimizing the FCN prediction model, combining historical and real-time network high-speed traffic feature data for training and prediction, and then combining it with the set predicted network high-speed traffic expansion and contraction rules. This effectively improved resource utilization, ensured that the cloud environment can dynamically adjust resources based on real-time and predicted traffic, avoided resource waste and overload problems, and significantly improved the system's flexibility and efficiency in responding to high-speed network traffic.

[0034] Preferably, in S34, finding the learning rate of the FCN network high-speed traffic prediction model using an optimization algorithm to obtain the optimal solution includes the following steps:

[0035] S341. Construct a sunflower population and set the size of the sunflower population to n ;

[0036] S342. Randomly set the initial height of the sunflower population according to the learning rate of the FCN network high-speed traffic prediction model to obtain an initial height set of the sunflower population;

[0037] S343. The height growth of sunflowers is limited by resources. Too high a height will lead to excessive energy consumption or structural instability. Set the height gain coefficient and height penalty coefficient and define the fitness function.

[0038] S344, iterating the initial height set of the sunflower population, updating the height of each sunflower in the initial height set of the sunflower population according to the fitness function during each iteration, and obtaining the best individual sunflower in the sunflower population and the global best sunflower during each iteration;

[0039] S345, repeat S344, when the maximum number of optimization iterations is reached, stop the iteration, and take the global best sunflower as the optimal solution;

[0040] The above steps use the Sunflower optimization algorithm to find the optimal solution for the learning rate of the FCN network high-speed traffic prediction model. By constructing a sunflower population, setting the initial height, defining the fitness function, and performing iterative operations, the global best sunflower is ultimately determined as the optimal solution for the learning rate. This significantly improves the accuracy and efficiency of the FCN prediction model, provides a more accurate prediction basis for dynamic scaling of cloud environments, and improves system resource utilization and response speed.

[0041] Preferably, said S4 comprises the following steps:

[0042] S41. Collect the real-time CPU utilization and real-time memory occupancy of the collection node group in the cloud environment after the first expansion and contraction to obtain a real-time load indicator.

[0043] S42, set load indicator expansion and contraction rules, and perform a second expansion and contraction of the cloud environment or keep it unchanged according to the load indicator expansion and contraction rules and the real-time load indicator, to obtain the cloud environment after the second expansion and contraction; the load indicator expansion and contraction rules are to set the maximum CPU utilization threshold h 1. Maximum memory usage threshold h 2. Minimum CPU utilization threshold h 3 and the minimum memory usage threshold h 4; When the real-time CPU utilization k 1 ≥ Maximum CPU utilization threshold h 1 or real-time memory usage k 2≥maximum load memory usage threshold h 2, expand the collection node group in the cloud environment to the current collection node group times; when the real-time CPU utilization k 1≤Minimum CPU utilization threshold h 3 or real-time load rate k 2≤Maximum load memory usage threshold h4, reduce the collection node group in the cloud environment to the current collection node group times;

[0044] The above steps achieve secondary precise scaling of the cloud environment by collecting the real-time CPU utilization and memory usage of the cloud environment collection node group after the first scaling as load indicators and setting scaling rules based on these indicators. This ensures that the cloud environment can efficiently and flexibly adapt to actual work needs, further optimizes resource allocation, improves system performance and stability, and reduces operating costs.

[0045] Preferably, the S5 comprises the following steps:

[0046] S51. In the node cluster of the cloud environment after the second expansion and contraction, batches of network high-speed traffic data are acquired using the DPDK kernel bypass technology to obtain network high-speed traffic packets.

[0047] S52, setting the dynamic filtering rule engine to discard ICMP protocol, block blacklist IP and match attack signature library; combining the dynamic filtering rule engine to perform three-level filtering on the network high-speed traffic packet to obtain the filtered network high-speed traffic packet;

[0048] S53, extracting metadata fields from the filtered network high-speed traffic packets, and adding threat level labels to the filtered network high-speed traffic packets based on the metadata fields to generate a preprocessed network high-speed traffic dataset;

[0049] The above steps utilize DPDK kernel bypass technology to efficiently acquire high-speed network traffic data in batches, and perform multi-level deep filtering on traffic packets through the set dynamic filtering rule engine, effectively discarding irrelevant protocol data, blocking malicious IP addresses, and identifying potential attack characteristics. Metadata fields of filtered traffic packets are extracted and threat level labels are added to generate a pre-processed network high-speed traffic data set. This significantly improves data collection and processing efficiency and enhances network security protection capabilities.

[0050] Preferably, the S6 comprises the following steps:

[0051] S61, setting a ring buffer threshold; when the ring buffer of the cloud environment reaches the ring buffer threshold, discarding the low-priority data in the pre-processed network high-speed traffic data set and compressing and encrypting it into Avro format to obtain a processed network high-speed traffic data set;

[0052] S62. Writing the processed network high-speed traffic data set into a Kafka message queue, partitioning the data in the processed network high-speed traffic data set by source IP hash values ​​to obtain a network high-speed traffic data stream;

[0053] The above steps set a ring buffer threshold and implement an intelligent data management strategy. When the buffer data accumulates to the threshold, the low-priority data in the pre-processed network high-speed traffic data set is selectively discarded, compressed and encrypted into Avro format, and the processed data set is written to the Kafka message queue and partitioned by the source IP hash value, generating a standardized network high-speed traffic data stream. This optimizes data storage and transmission efficiency and ensures high data availability and orderliness.

[0054] A cloud-based high-speed network traffic collection system, used to implement the above-mentioned cloud-based high-speed network traffic collection method, includes a cloud-based environment construction module, a traffic mirroring and feature processing module, an optimized FCN network high-speed traffic prediction model module, a first cloud-based environment scaling decision module, a second cloud-based environment scaling decision module, a dynamic filtering and threat tagging module, and a data aggregation and synchronization module.

[0055] The cloud environment building module is used to create an elastic resource pool, configure the collection node cluster and network parameters, and implement the deployment of infrastructure in the cloud environment;

[0056] The traffic mirroring and feature processing module is used to obtain real-time mirrored network high-speed traffic and extract key features to obtain real-time network high-speed traffic features;

[0057] The optimized FCN network high-speed traffic prediction model module improves the FCN network prediction model by using historical network high-speed traffic characteristics and optimization algorithms to obtain an optimized FCN network high-speed traffic prediction model;

[0058] The first cloud environment expansion / contraction decision module is used to obtain predicted network high-speed traffic based on the optimized FCN network high-speed traffic prediction model and real-time network high-speed traffic characteristics; determine whether to perform the first cloud environment expansion / contraction based on the predicted network high-speed traffic and the predicted network high-speed traffic expansion / contraction rules, and obtain the cloud environment after the first expansion / contraction;

[0059] The second cloud environment expansion and contraction decision module is used to determine whether to perform a second cloud environment expansion and contraction based on the load indicator expansion and contraction rules and the real-time load indicator, and obtain the cloud environment after the second expansion and contraction;

[0060] The dynamic filtering and threat marking module is used to perform multi-level cleaning on network high-speed traffic packets and mark the threat level;

[0061] The data aggregation and synchronization module is used to efficiently aggregate multi-node data and achieve low-latency, high-throughput standardized data stream output.

[0062] A network high-speed traffic collection device based on a cloud environment has a program stored thereon. When the program is executed by a processor, it is used to implement the above-mentioned network high-speed traffic collection method based on a cloud environment.

[0063] (3) Beneficial effects

[0064] The present invention has the following beneficial effects:

[0065] The present invention effectively solves the problem of insufficient resource elasticity in traditional cloud environments through a two-level dynamic scaling mechanism that combines prediction-driven scaling with real-time load feedback scaling. The first stage uses traffic prediction based on an optimized FCN network high-speed traffic prediction model to achieve resource pre-allocation and avoid processing delays caused by burst traffic. The second stage is based on precise adjustment of real-time CPU / memory indicators to further optimize resource utilization. Compared with a single threshold trigger mechanism, this improves the response speed of resource supply, reduces resource redundancy, and significantly reduces costs.

[0066] The present invention introduces an optimization algorithm to dynamically search for the optimal solution of the learning rate of the FCN network high-speed traffic prediction model, thereby obtaining an optimized FCN network high-speed traffic prediction model. This method overcomes the limitation of the traditional gradient descent method that is prone to falling into local optimality. By defining a fitness function, the bio-inspired adaptive adjustment of the model parameters is realized, which improves the training convergence speed of the model, reduces the prediction error rate, and provides a reliable basis for dynamic resource scheduling.

[0067] The present invention ensures fixed node processing of the same session traffic by adopting a consistent hashing algorithm, avoids disorder problems, and improves session continuity; combines three-level dynamic filtering rules to achieve fine-grained traffic cleaning and reduce the amount of invalid data processing; extracts application layer content features through lightweight DPI, dynamically adjusts node distribution weights, improves node cluster load balancing, and improves overall processing throughput.

[0068] The present invention designs a priority-aware ring buffer management strategy to prioritize high-priority traffic when the buffer reaches the threshold, thereby reducing the loss rate of critical data. It uses Kafka message queues to partition and synchronize data according to the source IP hash value, ensuring data orderliness. Combined with Avro format compression and encryption technology, it reduces data synchronization latency and storage space usage, meeting real-time analysis and long-term storage requirements in high-speed network scenarios.

[0069] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, they can also obtain drawings based on these drawings without paying any creative work.

[0071] Figure 1 This is a flow chart of a method for collecting high-speed network traffic based on a cloud environment according to the present invention;

[0072] Figure 2 This is a module diagram of a network high-speed traffic collection system based on a cloud environment in the present invention. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only part of the embodiments of the invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0074] In the description of the present invention, it should be understood that the terms "opening", "upper", "lower", "top", "middle", "inside" and the like indicating orientation or positional relationship are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the invention.

[0075] Example 1

[0076] See also Figure 1 The present invention discloses a method for collecting high-speed network traffic based on a cloud environment, comprising the following steps:

[0077] S1. Build a cloud environment and set its parameters.

[0078] Said S1 comprises the following steps:

[0079] S11. Create a virtualized resource pool in the cloud platform and build a collection node cluster based on Kubernetes; configure an elastic scaling policy for the resource pool; the elastic scaling policy of the resource pool changes based on the predicted size of the network high-speed traffic and the real-time load rate;

[0080] Allocate a dedicated virtual network interface to each collection node in the resource pool and enable SR-IOV technology to improve network card performance;

[0081] S12. Deploy the SDN controller on the cloud gateway, configure the network high-speed traffic mirroring rules, and mirror the target network high-speed traffic to the collection system. Set the bandwidth limit for the mirrored network high-speed traffic to avoid overload; for example, only mirror 50% of the network high-speed traffic for sampling and analysis.

[0082] S13. Configure an initial collection strategy in the control center. The strategy includes a network high-speed traffic distribution weight and a resource threshold for each node. The network high-speed traffic distribution weight is dynamically adjusted based on the node's latency. The resource threshold limits the maximum and minimum number of node clusters. For example, a collection node cluster retains a minimum of two collection nodes and can be expanded to a maximum of 50 collection nodes. Expansion is triggered when CPU usage is ≥70%. Filtering rules are defined, including basic rule templates, such as discarding ICMP protocols and marking abnormal IP network high-speed traffic.

[0083] S14, obtain cloud environment through S11, S12, and S13;

[0084] S2. Mirror the real-time network high-speed traffic to the cloud environment and perform feature extraction to obtain real-time network high-speed traffic features;

[0085] The S2 comprises the following steps:

[0086] S21. Mirror the real-time network high-speed traffic to the cloud environment to obtain the real-time mirrored network high-speed traffic;

[0087] S22. Extracting a five-tuple feature dataset and application layer content features of the real-time mirrored network high-speed traffic through lightweight DPI to obtain real-time network high-speed traffic features; the five-tuple feature dataset includes the source IP address, destination IP address, protocol type, source port, and destination port, and the application layer content features reflect the priority of the network high-speed traffic data; using a consistent hashing algorithm, mapping the real-time network high-speed traffic to a collection node cluster according to the hash value of the five-tuple feature dataset;

[0088] S3. Build an FCN prediction model and use historical network high-speed traffic feature data and an optimization algorithm to improve the FCN prediction model. This results in an optimized FCN network high-speed traffic prediction model. Combined with real-time network high-speed traffic features, this model predicts network high-speed traffic. Based on the predicted network high-speed traffic, a decision is made on whether to perform the first cloud environment expansion or contraction, resulting in the cloud environment after the first expansion or contraction.

[0089] The S3 includes the following steps:

[0090] S31. Build the FCN prediction model and set the initial learning rate of the FCN prediction model q ;

[0091] S32. Collect characteristic data of historical network high-speed traffic in the cloud environment to obtain historical network high-speed traffic characteristic data;

[0092] S33, setting a maximum number of training iterations and a training accuracy threshold; inputting the historical network high-speed traffic characteristic data into the FCN prediction model for training to obtain a training accuracy; adjusting a learning rate of the FCN prediction model according to the training accuracy; and obtaining an FCN network high-speed traffic prediction model when the training accuracy is greater than or equal to the training accuracy threshold or the maximum number of training iterations is reached;

[0093] S34, using an optimization algorithm to find a learning rate of the FCN network high-speed traffic prediction model to obtain an optimal solution; using the optimal solution as the learning rate of the FCN network high-speed traffic prediction model to obtain an optimized FCN network high-speed traffic prediction model;

[0094] In S34, the optimization algorithm is used to find the learning rate of the FCN network high-speed traffic prediction model, and obtaining the optimal solution includes the following steps:

[0095] S341. Construct a sunflower population and set the size of the sunflower population to n , then the sunflower population is represented by r={r1,r2,…,r i ,…,r n},in, r i Indicates the sunflower population i sunflowers; set the maximum number of optimization iterations;

[0096] S342, randomly setting the initial height of the sunflower population according to the learning rate of the FCN network high-speed traffic prediction model, and obtaining the initial height set of the sunflower population as m={m1,m2,…,m i ,…,m n},in m i Indicates the sunflower population i Only the height of the sunflower;

[0097] S343. The height growth of sunflowers is limited by resources. Too high will lead to excessive energy consumption or structural instability. The height gain coefficient is set to k 1. The height penalty coefficient is k 2. Define the fitness function. The fitness function formula is as follows:

[0098] ;

[0099] in, f ( m ) represents the fitness function, which is used to evaluate the sunflower's current heightm The fitness of m Indicates the current height of the sunflower;

[0100] S344, iterating the initial height set of the sunflower population, updating the height of each sunflower in the initial height set of the sunflower population according to the fitness function during each iteration, and obtaining the best individual sunflower in the sunflower population and the global best sunflower during each iteration;

[0101] S345, repeat S344, when the maximum number of optimization iterations is reached, stop the iteration, and take the global best sunflower as the optimal solution;

[0102] S35, inputting the real-time network high-speed traffic characteristic data into the optimized FCN network high-speed traffic prediction model to obtain the predicted network high-speed traffic;

[0103] S36, set the predicted network high-speed traffic expansion and contraction rules; according to the predicted network high-speed traffic expansion and contraction rules, the real-time mirror network high-speed traffic and the predicted mirror network high-speed traffic, the collection node group in the cloud environment is expanded, contracted or kept unchanged for the first time to obtain the cloud environment after the first expansion and contraction; the predicted network high-speed traffic expansion and contraction rules are to set the predicted network high-speed traffic expansion threshold d , predict network high-speed traffic reduction threshold f ; When the ratio of predicted network high-speed traffic to current network high-speed traffic e ≥Predicted network high-speed traffic expansion threshold d , expand the collection node group in the cloud environment to the current collection node group times; when the ratio of predicted network high-speed traffic to current network high-speed traffic g ≤Predicted network high-speed traffic reduction threshold f , reducing the collection node group in the cloud environment to the current collection node group times; such as when the predicted mirror network high-speed traffic data is 140% of the real-time mirror network high-speed traffic data, the number of collection node groups in the cloud environment is expanded to 1.2 times the current number of collection node groups;

[0104] S4. Collect load indicator data of the cloud environment after the first expansion and contraction, obtain real-time load indicators, and determine whether to perform a second expansion and contraction of the cloud environment, thereby obtaining the cloud environment after the second expansion and contraction.

[0105] The S4 comprises the following steps:

[0106] S41. Collect the real-time CPU utilization and real-time memory occupancy of the collection node group in the cloud environment after the first expansion and contraction to obtain a real-time load indicator.

[0107] S42, set load indicator expansion and contraction rules, and perform a second expansion and contraction of the cloud environment or keep it unchanged according to the load indicator expansion and contraction rules and the real-time load indicator, to obtain the cloud environment after the second expansion and contraction; the load indicator expansion and contraction rules are to set the maximum CPU utilization threshold h 1. Maximum memory usage threshold h 2. Minimum CPU utilization threshold h 3 and the minimum memory usage threshold h 4; When the real-time CPU utilization k 1 ≥ Maximum CPU utilization threshold h 1 or real-time memory usage k 2≥maximum load memory usage threshold h 2, expand the collection node group in the cloud environment to the current collection node group times; when the real-time CPU utilization k 1≤Minimum CPU utilization threshold h 3 or real-time load rate k 2≤Maximum load memory usage threshold h 4, reduce the collection node group in the cloud environment to the current collection node group times; such as the maximum CPU utilization threshold h 1. Maximum load memory usage threshold h 2 are both 75%, real-time CPU utilization k 1 is 85%, memory usage k 2 is 70%, then the number of nodes is expanded to 1.133 times the current number of nodes;

[0108] S5. Collect the network high-speed traffic packets of the cloud environment after the second expansion and contraction, and perform dynamic filtering processing to obtain a pre-processed network high-speed traffic data set;

[0109] The S5 comprises the following steps:

[0110] S51. In the node cluster of the cloud environment after the second expansion and contraction, batches of network high-speed traffic data are acquired using the DPDK kernel bypass technology to obtain network high-speed traffic packets.

[0111] S52, setting the dynamic filtering rule engine to discard ICMP protocol, block blacklist IP and match attack signature library; combining the dynamic filtering rule engine to perform three-level filtering on the network high-speed traffic packet to obtain the filtered network high-speed traffic packet;

[0112] S53, extracting metadata fields from the filtered network high-speed traffic packets, and adding threat level labels to the filtered network high-speed traffic packets based on the metadata fields to generate a preprocessed network high-speed traffic dataset;

[0113] S6. Aggregate and synchronize the pre-processed network high-speed traffic data set to obtain a network high-speed traffic data stream;

[0114] The S6 comprises the following steps:

[0115] S61, setting a ring buffer threshold; when the ring buffer of the cloud environment reaches the ring buffer threshold, discarding the low-priority data in the pre-processed network high-speed traffic data set and compressing and encrypting it into Avro format to obtain a processed network high-speed traffic data set;

[0116] S62. Write the processed network high-speed traffic data set into a Kafka message queue, partition the data in the processed network high-speed traffic data set according to the source IP hash value, and obtain a network high-speed traffic data stream.

[0117] Example 2

[0118] See also Figure 2 A cloud-based high-speed network traffic collection system is used to implement the above-mentioned cloud-based high-speed network traffic collection method, including a cloud-based environment construction module, a traffic mirroring and feature processing module, an optimized FCN network high-speed traffic prediction model module, a first cloud-based environment scaling decision module, a second cloud-based environment scaling decision module, a dynamic filtering and threat marking module, and a data aggregation and synchronization module.

[0119] The cloud environment building module is used to create an elastic resource pool, configure the collection node cluster and network parameters, and implement the deployment of infrastructure in the cloud environment;

[0120] The traffic mirroring and feature processing module is used to obtain real-time mirrored network high-speed traffic and extract key features to obtain real-time network high-speed traffic features;

[0121] The optimized FCN network high-speed traffic prediction model module improves the FCN network prediction model by using historical network high-speed traffic characteristics and optimization algorithms to obtain an optimized FCN network high-speed traffic prediction model;

[0122] The first cloud environment expansion / contraction decision module is used to obtain predicted network high-speed traffic based on the optimized FCN network high-speed traffic prediction model and real-time network high-speed traffic characteristics; determine whether to perform the first cloud environment expansion / contraction based on the predicted network high-speed traffic and the predicted network high-speed traffic expansion / contraction rules, and obtain the cloud environment after the first expansion / contraction;

[0123] The second cloud environment expansion and contraction decision module is used to determine whether to perform a second cloud environment expansion and contraction based on the load indicator expansion and contraction rules and the real-time load indicator, and obtain the cloud environment after the second expansion and contraction;

[0124] The dynamic filtering and threat marking module is used to perform multi-level cleaning on network high-speed traffic packets and mark the threat level;

[0125] The data aggregation and synchronization module is used to efficiently aggregate multi-node data and achieve low-latency, high-throughput standardized data stream output.

[0126] Example 3

[0127] A network high-speed traffic collection device based on a cloud environment has a program stored thereon. When the program is executed by a processor, it is used to implement the above-mentioned network high-speed traffic collection method based on a cloud environment.

[0128] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0129] The preferred embodiments of the invention disclosed above are intended only to help illustrate the invention. These preferred embodiments do not exhaust all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations are possible based on the content of this specification. These embodiments are selected and described in detail in this specification to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention.

Claims

1. A method for collecting high-speed network traffic based on a cloud environment, characterized in that: The following steps are involved: S1. Build a cloud environment and set its parameters. S2. Mirror the real-time network high-speed traffic to the cloud environment and perform feature extraction to obtain real-time network high-speed traffic features; S3. Build an FCN prediction model and use historical network high-speed traffic feature data and an optimization algorithm to improve the FCN prediction model. This results in an optimized FCN network high-speed traffic prediction model. Combined with real-time network high-speed traffic features, this model predicts network high-speed traffic. Based on the predicted network high-speed traffic, a decision is made on whether to perform the first cloud environment expansion or contraction, resulting in the cloud environment after the first expansion or contraction. The S3 includes the following steps: S31. Build the FCN prediction model and set the initial learning rate of the FCN prediction model q ; S32. Collect characteristic data of historical network high-speed traffic in the cloud environment to obtain historical network high-speed traffic characteristic data; S33, setting a maximum number of training iterations and a training accuracy threshold; inputting the historical network high-speed traffic characteristic data into the FCN prediction model for training to obtain a training accuracy; adjusting a learning rate of the FCN prediction model according to the training accuracy; and obtaining an FCN network high-speed traffic prediction model when the training accuracy is greater than or equal to the training accuracy threshold or the maximum number of training iterations is reached; S34, using an optimization algorithm to find a learning rate of the FCN network high-speed traffic prediction model to obtain an optimal solution; using the optimal solution as the learning rate of the FCN network high-speed traffic prediction model to obtain an optimized FCN network high-speed traffic prediction model; S35, inputting the real-time network high-speed traffic characteristic data into the optimized FCN network high-speed traffic prediction model to obtain the predicted network high-speed traffic; S36, set the predicted network high-speed traffic expansion and contraction rules, and perform the first cloud environment expansion and contraction or keep it unchanged according to the predicted network high-speed traffic expansion and contraction rules, the real-time network high-speed traffic and the predicted network high-speed traffic, to obtain the cloud environment after the first expansion and contraction; the predicted network high-speed traffic expansion and contraction rules are to set the predicted network high-speed traffic expansion threshold d , predict network high-speed traffic reduction threshold f ; When the ratio of predicted network high-speed traffic to current network high-speed traffic e ≥Predicted network high-speed traffic expansion threshold d , expand the collection node group in the cloud environment to the current collection node group times; when the ratio of predicted network high-speed traffic to current network high-speed traffic g ≤Predicted network high-speed traffic reduction threshold f , reducing the collection node group in the cloud environment to the current collection node group times; S4. Collect load indicator data of the cloud environment after the first expansion and contraction, obtain real-time load indicators, and determine whether to perform a second expansion and contraction of the cloud environment, thereby obtaining the cloud environment after the second expansion and contraction. S5. Collect the network high-speed traffic packets of the cloud environment after the second expansion and contraction, and perform dynamic filtering processing to obtain a pre-processed network high-speed traffic data set; S6. Aggregate and synchronize the pre-processed network high-speed traffic data set to obtain a network high-speed traffic data stream.

2. The method for collecting high-speed network traffic based on a cloud environment according to claim 1, characterized in that: Said S1 comprises the following steps: S11. Create a virtualized resource pool in the cloud platform and build a collection node cluster based on Kubernetes; configure an elastic scaling policy for the resource pool; the elastic scaling policy of the resource pool changes based on the predicted size of the network high-speed traffic and the real-time load rate; Allocate a dedicated virtual network interface to each collection node in the resource pool and enable SR-IOV technology to improve network card performance; S12. Deploy the SDN controller on the cloud gateway, configure the network high-speed traffic mirroring rules, mirror the target network high-speed traffic to the collection system, and set the bandwidth limit for the mirrored network high-speed traffic; S13. Configuring an initial acquisition strategy in the control center. The initial acquisition strategy includes a network high-speed traffic distribution weight and a resource threshold for each node. The network high-speed traffic distribution weight is dynamically adjusted based on the node's latency. The resource threshold limits the maximum and minimum number of node clusters. S14. Obtain a cloud environment through S11, S12, and S13.

3. The method for collecting high-speed network traffic based on a cloud environment according to claim 1, characterized in that: The S2 comprises the following steps: S21. Mirror the real-time network high-speed traffic to the cloud environment to obtain the real-time mirrored network high-speed traffic; S22. Extract the five-tuple feature data set and application layer content features of the real-time mirrored network high-speed traffic through lightweight DPI to obtain real-time network high-speed traffic features; the five-tuple feature data set includes the source IP address, destination IP address, protocol type, source port and destination port, and the application layer content features reflect the priority of the network high-speed traffic data; use the consistent hashing algorithm to map the real-time network high-speed traffic to the collection node cluster according to the hash value of the five-tuple feature data set.

4. The method for collecting high-speed network traffic based on a cloud environment according to claim 1, characterized in that: In S34, the optimization algorithm is used to find the learning rate of the FCN network high-speed traffic prediction model, and obtaining the optimal solution includes the following steps: S341, construct sunflower population; S342. Randomly set the initial height of the sunflower population according to the learning rate of the FCN network high-speed traffic prediction model to obtain an initial height set of the sunflower population; S343. The height growth of sunflowers is limited by resources. Set the height gain coefficient and height penalty coefficient and define the fitness function. S344, iterating the initial height set of the sunflower population, updating the height of each sunflower in the initial height set of the sunflower population according to the fitness function during each iteration, and obtaining the best individual sunflower in the sunflower population and the global best sunflower during each iteration; S345. Repeat S344. When the maximum number of optimization iterations is reached, stop the iteration and take the global best sunflower as the optimal solution.

5. The method for collecting high-speed network traffic based on a cloud environment according to claim 1, characterized in that: The S4 comprises the following steps: S41. Collect the real-time CPU utilization and real-time memory occupancy of the collection node group in the cloud environment after the first expansion and contraction to obtain a real-time load indicator. S42, set load indicator expansion and contraction rules, and perform a second expansion and contraction of the cloud environment or keep it unchanged according to the load indicator expansion and contraction rules and the real-time load indicator, to obtain the cloud environment after the second expansion and contraction; the load indicator expansion and contraction rules are to set the maximum CPU utilization threshold h 1. Maximum memory usage threshold h 2. Minimum CPU utilization threshold h 3 and the minimum memory usage threshold h 4; When the real-time CPU utilization k 1 ≥ Maximum CPU utilization threshold h 1 or real-time memory usage k 2≥maximum load memory usage threshold h 2, the collection node group in the cloud environment is expanded to the current collection node group times; when the real-time CPU utilization k 1≤Minimum CPU utilization threshold h 3 or real-time load rate k 2≤Maximum load memory usage threshold h 4, reduce the number of collection nodes in the cloud environment to the number of current collection nodes. times.

6. The method for collecting high-speed network traffic based on a cloud environment according to claim 1, characterized in that: The S5 comprises the following steps: S51. In the node cluster of the cloud environment after the second expansion and contraction, batches of network high-speed traffic data are acquired using the DPDK kernel bypass technology to obtain network high-speed traffic packets. S52, setting the dynamic filtering rule engine to discard ICMP protocol, block blacklist IP and match attack signature library; combining the dynamic filtering rule engine to perform three-level filtering on the network high-speed traffic packet to obtain the filtered network high-speed traffic packet; S53 , extracting metadata fields of the filtered network high-speed traffic packets, and adding threat level labels to the filtered network high-speed traffic packets according to the metadata fields, to generate a pre-processed network high-speed traffic data set.

7. The method for collecting high-speed network traffic based on a cloud environment according to claim 1, characterized in that: The S6 comprises the following steps: S61, setting a ring buffer threshold; when the ring buffer of the cloud environment reaches the ring buffer threshold, discarding the low-priority data in the pre-processed network high-speed traffic data set and compressing and encrypting it into Avro format to obtain a processed network high-speed traffic data set; S62. Write the processed network high-speed traffic data set into a Kafka message queue. Partition the data in the processed network high-speed traffic data set according to the source IP hash value to obtain a network high-speed traffic data stream.

8. A cloud-based high-speed network traffic collection system, configured to implement a cloud-based high-speed network traffic collection method according to any one of claims 1 to 7, the system comprising a cloud-based environment construction module, a traffic mirroring and feature processing module, an optimized FCN network high-speed traffic prediction model module, a first cloud-based environment scaling decision module, a second cloud-based environment scaling decision module, a dynamic filtering and threat tagging module, and a data aggregation and synchronization module; The cloud environment building module is used to create an elastic resource pool, configure the collection node cluster and network parameters, and implement the deployment of infrastructure in the cloud environment; The traffic mirroring and feature processing module is used to obtain real-time mirrored network high-speed traffic and extract key features to obtain real-time network high-speed traffic features; The optimized FCN network high-speed traffic prediction model module improves the FCN prediction model by using historical network high-speed traffic characteristics and optimization algorithms to obtain an optimized FCN network high-speed traffic prediction model; The first cloud environment expansion / contraction decision module is used to obtain predicted network high-speed traffic based on the optimized FCN network high-speed traffic prediction model and real-time network high-speed traffic characteristics; determine whether to perform the first cloud environment expansion / contraction based on the predicted network high-speed traffic and the predicted network high-speed traffic expansion / contraction rules, and obtain the cloud environment after the first expansion / contraction; The second cloud environment expansion and contraction decision module is used to determine whether to perform a second cloud environment expansion and contraction based on the load indicator expansion and contraction rules and the real-time load indicator, and obtain the cloud environment after the second expansion and contraction; The dynamic filtering and threat marking module is used to perform multi-level cleaning on network high-speed traffic packets and mark the threat level; The data aggregation and synchronization module is used to efficiently aggregate multi-node data and achieve low-latency, high-throughput standardized data stream output.

9. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, a network high-speed traffic collection method based on a cloud environment as described in any one of claims 1 to 7 is implemented.

Images