Lightweight network flow storage and management method and system based on flow clustering

Abstract

The invention relates to a lightweight network flow storage and management method and system based on flow clustering, and belongs to the field of computer network security, and the method comprises the steps: deploying a Zeek monitoring component and P4 equipment at a network boundary and a core node, and carrying out the structural analysis of a multi-source weblog; based on data source and flow feature clustering and grading processing, dividing into a key flow, a common flow and a secondary flow; hierarchical forwarding and priority scheduling of the logs are realized through Filebeat and Kafka; a hierarchical storage architecture of a core flow layer, a main service flow layer and an archiving flow layer is constructed based on ClickHouse, and efficient retrieval and traceability analysis are realized through a materialized view and a multi-dimensional index. According to the invention, flexible scheduling and grading processing of traffic can be realized in a high-concurrency network environment, the overall throughput efficiency and security situation awareness capability of the system are improved, real-time performance and storage cost control are both considered, and the method is suitable for threat monitoring and event tracing scenes in a large-scale network environment.

Description

TECHNICAL FIELD

[0001] The application relates to a lightweight network traffic storage and management method and system based on flow clustering, and belongs to the technical field of computer network security. BACKGROUND

[0002] With the continuous expansion of network scale and the increasing richness of business types, network traffic monitoring and analysis systems need to process and store massive data in real time in a complex environment with high throughput and high concurrency. Traditional traffic collection and storage methods mostly adopt a "centralized collection + unified storage" architecture, such as writing traffic collected by network probes into a database. Although this approach can retain traffic information completely, it also has obvious technical bottlenecks: High import delay: During traffic peaks, the database write pressure rises sharply, causing log backlog, serious import delay, and difficulty in guaranteeing real-time performance; Waste of storage space: A large amount of redundant or low-risk traffic occupies storage space, causing resource waste and cost increase; Poor retrieval performance: When performing threat backtracking and security posture analysis on TB-level or even PB-level log data, the query delay is high, making it difficult to meet the needs of real analysis; In response to multi-service, sudden and dynamic traffic scenarios, processing bottlenecks, storage pressure surges and real-time performance deficiencies are likely to occur. Existing log collection and import strategies cannot effectively distinguish and process different types of traffic, resulting in high-value business flows and low-value redundant flows being treated the same in the storage and analysis process. This not only causes a large waste of storage resources, but also lowers the overall efficiency of log processing and threat detection. At the same time, with the continuous evolution of network attack technology, short-cycle, high-concurrency malicious traffic has a greater impact on traditional traffic collection and storage systems. In the absence of dynamic scheduling and hierarchical processing mechanisms, the system is difficult to identify and track abnormal traffic in a timely manner, resulting in low efficiency in analyzing and locating malicious traffic and affecting the timeliness of network security monitoring and threat tracing.

[0003] In recent years, although log collection and storage schemes based on streaming processing architecture have emerged, such as using Apache Kafka and ClickHouse to build a streaming import system, most implementations still remain at the level of simple mapping of Topic and database tables, lacking clustering, compression and hierarchical import mechanisms based on flow characteristics. As a result, the proportion of redundant data is high, the utilization efficiency of storage resources is low, and the query performance improvement is limited.

[0004] Therefore, there is an urgent need for a new network traffic storage and management architecture that is oriented to large-scale network environments, can perform clustering analysis based on flow characteristics, implement hierarchical processing according to flow importance and attributes, and improve overall performance through an efficient hierarchical storage and scheduling mechanism, in order to solve the bottleneck problems of real-time performance, storage efficiency and retrieval performance of the prior art. SUMMARY

[0005] In view of the deficiencies of the prior art, the present application provides a lightweight network traffic storage and management method and system based on flow clustering, which realizes efficient collection, distribution and hierarchical storage of traffic by real-time clustering and hierarchical processing of traffic data, and improves the overall performance and scalability of the system.

[0006] Terminology: 1. Key flow: usually related to high-value assets, sensitive communication or potential attack behavior, with high security risk or analysis value. Such traffic needs to be prioritized and stored to ensure traceability and detection capability.

[0007] 2. Ordinary flow: corresponding to regular business communication, with certain analysis and statistical value, used to support basic situation awareness and traffic baseline modeling.

[0008] 3. Secondary flow: generally low-risk, low-value regular access traffic, which can be delayed or archived, used for auxiliary analysis or historical backtracking.

[0009] The present application adopts the following technical solutions: A lightweight network traffic storage and management method based on flow clustering, comprising the following steps: (1) deploying Zeek monitoring components at network boundaries or core switching positions to collect incoming and outgoing network traffic in real time, and performing structured parsing on TCP, UDP, DNS, HTTP and other protocols to generate standardized JSON logs, and deploying P4 devices to collect network traffic and parse to generate standardized JSON logs; During the collection process, only key information and necessary features of each flow are retained for lightweight storage, thereby reducing storage pressure and ensuring efficient processing of the system. Through multi-source collection and parsing of Zeek and P4, the system can be compatible with different network environments and data sources, providing reliable data support for subsequent flow clustering and hierarchical processing.

[0010] The present application proposes a joint collection mechanism of Zeek monitoring components and P4 devices, which supports real-time collection of multi-source traffic at multiple positions such as network boundaries and core switches, and generates unified standardized JSON logs through protocol parsing, only retaining key feature fields, realizing lightweight and standardization of data, and significantly improving the real-time performance, compatibility and storage efficiency of collection.

[0011] (2) Cluster and classify the multi-source standardized JSON logs generated in step (1), and divide them into key flows, ordinary flows and secondary flows; (3) Deploy Filebeat to monitor the Zeek monitoring component and the standardized JSON logs generated by P4 in real time, collect the standardized JSON logs, perform field standardization and buffer processing, and push the standardized JSON logs with the classification labels added in step (2) to the Kafka cluster. The Kafka cluster distributes the standardized JSON logs to high-priority, medium-priority and low-priority topics according to the classification labels. The key flows are distributed to the high-priority topic, the ordinary flows are distributed to the medium-priority topic, and the secondary flows are distributed to the low-priority topic, realizing hierarchical transmission according to key flows, ordinary flows and secondary flows, and ensuring low delay and high reliability of data in the transmission process; Kafka is a high-performance distributed message queue system, mainly used for: message transmission (such as logs, events, monitoring data); real-time data stream processing (such as traffic analysis, user behavior tracking, IoT data, etc.); Topic in Kafka is like a "message category" or "channel", Topic is a data channel in Kafka for classifying and storing messages.

[0012] (4) Kafka receives the logs with traffic classification and clustering labels forwarded in step (3), and distributes the logs to the corresponding priority topic according to the priority of key flows, ordinary flows and secondary flows, realizing shunting and buffer scheduling.

[0013] Key flows enter the high-priority topic to ensure priority processing and rapid warehousing. Ordinary flows enter the medium-priority topic for regular business situation awareness; secondary flows enter the low-priority topic, which can use delayed writing or batch archiving strategy to reduce system pressure. At the same time, the system can dynamically adjust the buffer queue length and distribution strategy according to the traffic load, realize peak clipping and valley filling, elastic scheduling, and ensure that all types of traffic can be stably processed under high concurrency, balancing real-time performance and system throughput efficiency.

[0014] In the log transmission and scheduling mechanism, the present application adopts a distributed message system of Filebeat+Kafka to realize multi-level Topic scheduling. Through hierarchical transmission and asynchronous buffer mechanism, the system can dynamically schedule traffic warehousing priority according to risk level, which not only ensures low delay transmission of high-risk traffic, but also realizes system load balancing and elastic expansion through hierarchical buffer strategy, with real-time performance and scalability advantages in high-concurrency scenarios.

[0015] (5) ClickHouse storage layer receives log streams of different priority topics in real time through Kafka table. The materialized view normalizes and indexes the fields, and the main table is partitioned and stored in compressed form according to traffic priority and business type, realizing efficient hierarchical management.

[0016] Based on ClickHouse, three different storage areas of core flow layer, main business flow layer and archive flow layer are constructed, and traffic logs are stored in layers according to data importance and risk level, including: Core flow layer (Core Layer): Corresponding to key flow, storing high-risk, high-value communication and attack behavior logs to ensure fast retrieval and situational awareness capability; Primary flow layer (Primary Flow Layer): Corresponding to ordinary flow, storing regular business traffic and basic network situation data, balancing query efficiency and storage performance; Archive flow layer (Archive Layer): Corresponding to secondary flow, storing low-priority or delayed archived logs for historical backtracking or sparse analysis to reduce system pressure.

[0017] At the same time, a multi-dimensional index structure is established to realize fast retrieval, abnormal behavior tracking and traceability analysis of various traffic. Through the combination of hierarchical storage and multi-dimensional indexing, the system balances real-time performance, query efficiency and storage optimization, ensuring that key flows are prioritized, ordinary flows are efficiently managed, and secondary flows are delayed archived, thereby improving overall network threat detection and forensics capabilities.

[0018] The application proposes a ClickHouse hierarchical storage system in data storage and indexing mechanism, and stores data in core flow layer, main business flow layer and archive flow layer according to traffic risk level and access frequency, and constructs multi-dimensional index (time, clustering ID, source / desination IP, etc.), realizing fast retrieval and traceability analysis of high-risk data. At the same time, hierarchical compression strategy is used to optimize storage utilization, improve query performance and system response speed.

[0019] Preferably, in step (1), the standardized JSON structured log includes IP address, port, protocol type, timestamp and key traffic features.

[0020] Preferably, the implementation process of step (2) is: (2.1) First, the standardized JSON structured log is preliminarily distinguished according to data source: Traffic from P4 devices is usually a key object that has been filtered or matched by hardware rules, containing attack behavior features, key asset access records or high-risk communication information, and is directly marked as a key flow by the system; Traffic from Zeek monitoring components covers regular business communication in the overlay network, as an important data source for basic network situation awareness, into the clustering and grading process of step (2.2); (2.2) Clustering and grading process: After completing the source-level preliminary distinction, the system further extracts multi-dimensional behavior characteristics for the traffic of the Zeek monitoring component, including session duration, communication direction, packet number, byte volume, protocol type, request frequency, domain name characteristics, and uses existing clustering algorithms based on KMeans to automatically aggregate and group the multi-dimensional behavior characteristics of the traffic. The traffic is divided into key flow, ordinary flow and secondary flow, and is labeled with a grading label respectively, so as to realize automatic grading management and high-value priority processing of multi-source heterogeneous traffic.

[0021] Preferably, in step (2.2), according to the feature distribution of the cluster center and the risk weight, the traffic of the Zeek monitoring component is divided into key flow, ordinary flow and secondary flow, and the risk weight is referred to the following factors: Consider the proportion of abnormal features of the cluster center (such as connection failure rate, port scanning frequency, burst access behavior proportion), the proportion of abnormal features of the cluster center (such as short period high frequency communication or abnormal persistent connection), protocol type and service sensitivity (such as high risk ports involving management protocol, DNS, database access, etc.), and the trust score of the target domain name or IP belonging to the region; The risk weight is divided into three intervals, risk weight > threshold T1, high risk weight cluster, marked as key flow, indicating that it may involve core business or potential abnormal behavior; threshold T2 ≤ risk weight ≤ threshold T1, indicating that it may involve core business or potential abnormal behavior, marked as ordinary flow, indicating normal but need to be monitored communication; risk weight < threshold T2, low risk weight cluster, marked as secondary flow, indicating that the business relevance is low or the background traffic is stable; the priority of key flow is the highest, the priority of ordinary flow is slightly lower, and the priority of secondary flow is the lowest.

[0022] Preferably, threshold T1 = 0.8, threshold T2 = 0.5, which can be dynamically adjusted according to the current system environment.

[0023] For example: (1) 70% of the sessions in a cluster attempt to connect to non-existent ports or IPs → high proportion of abnormal features → high weight.

[0024] (2) A session maintains 24-hour continuous communication → high weight.

[0025] (3) Most of the traffic in the cluster accesses the company's database port 3306 → high weight; (4) A domain name in the cluster is accessed, which has a large number of malicious activities in history → high weight; Through the above hierarchical mechanism, the system can introduce a feature weight evaluation model oriented to security analysis while maintaining the generality of the clustering algorithm, and realize automatic hierarchical management and high-value priority processing of multi-source heterogeneous Zeek traffic.

[0026] In terms of traffic analysis and classification mechanism, the system performs clustering calculation on multi-source traffic and automatically calculates risk weight to divide the traffic into key flow, ordinary flow and secondary flow. This method realizes risk-oriented automatic hierarchical management, completes preliminary risk screening before traffic storage, effectively reduces subsequent analysis load and improves response efficiency of high-risk traffic.

[0027] Preferably, in step (5), the multi-dimensional index includes time, cluster ID, IP, etc.

[0028] A lightweight network traffic storage and management system based on flow clustering is used to implement the above lightweight network traffic storage and management method based on flow clustering, comprising: The acquisition module is used to acquire the network traffic in and out in real time through the Zeek monitoring component and the P4 device, and parse and generate standardized JSON logs; The traffic feature clustering module is used for clustering and hierarchical processing of the standardized JSON logs; The log acquisition and forwarding module is used to monitor the standardized JSON logs generated by the Zeek monitoring component and the P4 in real time, and push the standardized JSON logs with hierarchical labels to the Kafka cluster, and distribute the standardized JSON logs to high priority, medium priority and low priority topics respectively; The hierarchical shunting and scheduling module is used for differential processing of traffic from different priorities; The hierarchical storage and index optimization module is used to construct three different storage areas of core flow layer, main business flow layer and archive flow layer, and different priority traffic enters the corresponding storage hierarchy according to the classification label; at the same time, a multi-dimensional index structure is established to realize fast retrieval, abnormal behavior tracking and traceability analysis of various traffic.

[0029] The overall architecture of the present application is composed of "multi-source acquisition-flow clustering-Kafka scheduling-ClickHouse storage", which omits the complex model training, visualization and strategy control module, has clear structure, flexible deployment, and is suitable for edge nodes and high-performance real-time processing scenarios. The present application can realize elastic scheduling and hierarchical processing of traffic in a high-concurrency network environment, improve the overall throughput efficiency and security situation awareness capability of the system, balance real-time performance and storage cost control, and is suitable for threat monitoring and event tracing scenarios in large-scale network environments.

[0030] The details of the present application can be referred to the prior art.

[0031] The beneficial effects of the present application are: 1. Strengthening flow monitoring and network situation awareness: through real-time collection, protocol analysis and lightweight storage of P4 and Zeek multi-source traffic, key communication information is efficiently extracted, priority attention is given to high-risk traffic, and situation awareness of key asset access and potential attack behavior is enhanced.

[0032] 2. Realize traffic clustering and hierarchical processing: automatically aggregate and group network traffic from different sources, support multi-level traffic classification and priority processing, and improve the detection and analysis efficiency of key traffic.

[0033] 3. Optimize log transmission and system throughput performance: combined with the hierarchical forwarding and buffering mechanism of Filebeat and Kafka, realize traffic category priority shunting and dynamic scheduling, and guarantee system stability and throughput efficiency in high concurrency environment.

[0034] 4. Improve storage management and traceability: realize priority storage of key flows, efficient management of ordinary flows and delayed archiving of secondary flows, improve log retrieval efficiency, abnormal flow tracking capability and network security traceability and evidence collection capability. BRIEF DESCRIPTION OF DRAWINGS

[0035] The drawings accompanying the specification of this application are used to provide a further understanding of the application, the illustrative embodiments of the application and their descriptions serve to explain the application, and do not constitute an improper limitation on the application.

[0036] Figure 1 The implementation process diagram of the lightweight network traffic storage and management method based on flow clustering of the embodiments of the present application. DETAILED DESCRIPTION

[0037] In order to make the technical personnel in the art better understand the technical solutions in the specification, the technical solutions in the embodiments of the specification will be described clearly and completely below with reference to the drawings in the implementation of the specification, but not limited to this, the present application is not described in detail, and it is according to the conventional technology in the art.

[0038] Embodiment 1 A lightweight network traffic storage and management method based on flow clustering, which adopts a five-level processing architecture of "collection-clustering-forwarding-scheduling-storage", is used to realize efficient collection, automatic clustering and hierarchical processing, hierarchical scheduling and hierarchical storage of multi-source heterogeneous network traffic, so as to improve the processing efficiency of key traffic and the throughput performance of the whole system.

[0039] Comprising the following steps: S100 collects the traffic in and out of the network boundary and core node in real time through a P4 device. The P4 device has high-speed hardware processing capability, can realize efficient traffic forwarding and key object screening, and is suitable for traffic collection scenarios in a high-performance environment.

[0040] After the traffic collected in S101 is processed by a traffic analysis module (the traffic analysis module is mainly responsible for protocol analysis, feature extraction and format processing of the original data stream delivered by the P4 device, specifically, the P4 device and the master node interact through a high-speed bus, and the master node starts a UDP listening program to receive real-time data stream from the P4. The traffic analysis module analyzes the data packet layer by layer at the receiving end, and combines the IP / DNS analysis engine to extract the source / destination address, port, protocol type, domain name resolution result, session duration, byte number and other key features of the session), key information and necessary features are extracted and formatted into standardized JSON structured logs, providing high-quality data input for subsequent feature clustering and hierarchical analysis, and then entering step S200 traffic feature clustering module.

[0041] S110 collects the traffic in and out of the network boundary and core node in real time through a Zeek listening component. The Zeek listening component can deeply analyze multiple protocols such as TCP, UDP, DNS and HTTP, has a wide coverage, and is suitable for regular business traffic and network situation awareness scenarios.

[0042] The traffic collected by S111 is processed by the built-in protocol analysis architecture of Zeek, which extracts the source IP, destination IP, port, protocol type, timestamp and basic communication features, and converts the analysis results into standardized JSON structured logs, providing stable data support for step S200 traffic feature clustering.

[0043] The basic communication features refer to the primary quantitative features of traffic behavior at the session level, including session duration, number of bidirectional data packets, transmission byte amount, connection frequency and other indicators reflecting communication behavior patterns. As the bottom layer feature description of traffic, the basic communication features provide input support for the traffic feature clustering in step S200. On this basis, the system further extracts higher layer key traffic features and generates traffic classification labels through clustering and hierarchical analysis, thereby realizing fine-grained classification and risk identification of traffic at the semantic level.

[0044] S200 enters the traffic feature clustering module, and multi-dimensional feature extraction and classification aggregation are performed on the standardized JSON logs collected in steps S100-S110. The clustering module mainly distinguishes the logs according to the data sources and traffic features: the traffic collected by P4 is directly labeled as critical flow; for the traffic from Zeek, the system further extracts multi-dimensional behavior information such as session duration, communication direction, packet quantity, byte amount, protocol type, request frequency, and domain name features, and uses clustering algorithms to automatically aggregate and group the traffic features. According to the feature distribution of the clustering center and the corresponding risk weight, the system further divides the Zeek traffic into three categories: critical flow, normal flow, and minor flow, and automatically adds a hierarchical label (cluster_label) to each traffic record.

[0045] The hierarchical traffic is shown in Table 1 (only the core fields are shown), including timestamp (timestamp), id.orig_h (source IP), id.orig_p (source port), id.resp_h (destination IP), id.resp_p (destination port), proto (protocol type), and cluster_label (traffic classification label). This table directly reflects the hierarchical situation of different sources and types of network traffic in the system, including critical flow (critical), normal flow (normal), and minor flow (minor).

[0046] Table 1 Traffic information table after hierarchical labeling

[0047] S300 enters the log collection and forwarding link module, and the system uses Filebeat to monitor the output directory of P4 and Zeek in real time, and pushes the standardized logs with classification labels added in step S200 to the Kafka cluster. Kafka distributes the logs to high-priority, medium-priority, and low-priority topics according to the hierarchical labels of the traffic, achieving shunting and buffer scheduling of different types of traffic.

[0048] S400 enters the hierarchical shunting and scheduling module, and the system adopts differentiated processing strategies for traffic from different priority levels in step S300. Critical flow realizes priority storage, ensuring rapid response to high-risk communication; normal flow is stored according to the regular time sequence, supporting business situation analysis and security monitoring; minor flow is processed through delayed writing or batch archiving to reduce system instantaneous load, realizing resource elastic scheduling and peak load regulation.

[0049] S500 enters the hierarchical storage and index optimization module, and the system constructs three different storage areas of a core flow layer, a main business flow layer and an archive flow layer based on ClickHouse. Traffic of different priorities enters the corresponding storage level according to the label. Through the materialized view and multi-dimensional index (including time, clustering ID, IP, etc.), efficient query and correlation analysis are realized, so as to improve the log retrieval speed and abnormal traceability.

[0050] Through the above steps, the embodiment realizes a complete closed-loop processing flow from network traffic collection, clustering and grading, distribution scheduling to hierarchical storage, which not only can preferentially guarantee the fast processing and safe response of key traffic, but also can balance the processing performance and storage cost in a large-scale network environment.

[0051] Embodiment 2 A lightweight network traffic storage and management system based on flow clustering is used to implement the lightweight network traffic storage and management method based on flow clustering of embodiment 1, comprising: A collection module is used to collect the network traffic in and out in real time through the Zeek monitoring component and the P4 device, and to parse and generate standardized JSON logs; A traffic feature clustering module is used to cluster and grade the standardized JSON logs; A log collection and forwarding module is used to monitor the standardized JSON logs generated by the Zeek monitoring component and the P4 in real time, and to push the standardized JSON logs with added grading labels to the Kafka cluster, and to distribute the standardized JSON logs to high-priority, medium-priority and low-priority topics respectively; A hierarchical shunting and scheduling module is used to adopt differential processing for traffic from different priorities; A hierarchical storage and index optimization module is used to construct three different storage areas of a core flow layer, a main business flow layer and an archive flow layer, and different priorities of traffic enter the corresponding storage level according to the classification label; at the same time, a multi-dimensional index structure is established to realize fast retrieval, abnormal behavior tracking and traceability analysis of various traffic.

[0052] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should also be considered within the scope of protection of the present application.

Claims

1. A lightweight network traffic storage and management method based on flow clustering, characterized in that, Includes the following steps: (1) Deploy the Zeek monitoring component to collect incoming and outgoing network traffic in real time, perform structured parsing, and generate standardized JSON logs. At the same time, deploy P4 devices to collect network traffic and parse and generate standardized JSON logs. (2) Cluster and classify the multi-source standardized JSON logs generated in step (1) into key streams, ordinary streams and secondary streams; (3) Deploy Filebeat to monitor the Zeek listening component and the standardized JSON logs generated by P4 in real time, and push the standardized JSON logs with added hierarchical tags in step (2) to the Kafka cluster. The Kafka cluster distributes the standardized JSON logs to high-priority, medium-priority and low-priority topics according to the hierarchical tags, so as to realize hierarchical transmission according to key flow, normal flow and secondary flow, and ensure low latency and high reliability of data in the transmission process. (4) Differentiated processing is adopted for traffic from different priorities, with critical traffic being prioritized for storage; ordinary traffic is stored according to the normal time sequence. Secondary streams are handled through delayed writing or batch archiving. (5) Based on ClickHouse, three different storage areas are constructed: the core flow layer, the main business flow layer, and the archive flow layer. Traffic of different priorities enters the corresponding storage layer according to the classification tags. The critical flow corresponds to the core flow layer, the ordinary flow corresponds to the main business flow layer, and the secondary flow corresponds to the archive flow layer. At the same time, a multi-dimensional index structure is established to enable rapid retrieval of various types of traffic, tracking of abnormal behavior, and source analysis.

2. The lightweight network traffic storage and management method based on flow clustering according to claim 1, characterized in that, In step (1), the standardized JSON structured log includes IP address, port, protocol type, timestamp and key traffic characteristics.

3. The lightweight network traffic storage and management method based on flow clustering according to claim 2, characterized in that, The implementation process of step (2) is as follows: (2.1) First, the standardized JSON structured logs are initially distinguished based on their data sources: Traffic from P4 devices is directly labeled as critical flow by the system; Traffic from the Zeek monitoring component covers routine business communications in the network, serving as an important data source for basic network situational awareness, and enters the clustering and hierarchical process in step (2.2). (2.2) Clustering and hierarchical process: Further multi-dimensional behavioral features are extracted from the traffic of the Zeek monitoring component, including session duration, communication direction, number of data packets, byte size, protocol type, request frequency, and domain characteristics. The KMeans-based clustering algorithm is used to automatically aggregate and group the multi-dimensional behavioral features of the traffic, dividing the traffic into critical flow, normal flow, and secondary flow, and labeling them with hierarchical tags respectively.

4. The lightweight network traffic storage and management method based on flow clustering according to claim 3, characterized in that, In step (2.2), based on the characteristic distribution and risk weight of the cluster centers, the traffic of the Zeek monitoring component is divided into critical flow, normal flow and secondary flow. The risk weight takes into account the proportion of abnormal features of the cluster centers, the protocol type and service sensitivity, and the credibility score of the target domain name or IP region. The risk weight is divided into three levels: risk weight > threshold T1, marked as critical flow; threshold T2 ≤ risk weight ≤ threshold T1, marked as normal flow; risk weight < threshold T2, marked as secondary flow. Critical flows have the highest priority, normal flows have a slightly lower priority, and secondary flows have the lowest priority.

5. The lightweight network traffic storage and management method based on flow clustering according to claim 4, characterized in that, Threshold T1 = 0.8, threshold T2 = 0.

5.

6. The lightweight network traffic storage and management method based on flow clustering according to claim 5, characterized in that, In step (5), the multidimensional index includes time, cluster ID, and IP.

7. A lightweight network traffic storage and management system based on flow clustering, characterized in that, The lightweight network traffic storage and management method based on flow clustering as described in any one of claims 1-6 includes: The acquisition module is used to collect incoming and outgoing network traffic in real time through the Zeek monitoring component and P4 device, and parse it to generate standardized JSON logs; The traffic feature clustering module is used to cluster and classify standardized JSON logs. The log collection and forwarding module is used to monitor the standardized JSON logs generated by the Zeek listening component and P4 in real time, and push the standardized JSON logs with added hierarchical tags to the Kafka cluster, and distribute the standardized JSON logs to high-priority, medium-priority and low-priority topics respectively. The hierarchical traffic distribution and scheduling module is used to differentiate traffic from different priorities. The tiered storage and index optimization module is used to construct three different storage areas: the core flow layer, the main business flow layer, and the archive flow layer. Traffic of different priorities enters the corresponding storage layer according to the classification tags. At the same time, a multi-dimensional index structure is established to enable fast retrieval of various types of traffic, tracking of abnormal behavior, and source analysis.

Images