Pre-filtering method and device for network traffic storage

Abstract

The invention discloses a network traffic storage pre-filtering method and device. The method comprises the following steps: configuring a traffic filtering rule; acquiring an initial network flow; filtering the network traffic by using a traffic filtering rule to obtain filtered network traffic information; performing application identification filtering processing on the filtered network flow information to obtain effective network flow information; updating a traffic filtering rule by using the effective network traffic information; and performing classified storage processing on the effective network flow information based on the application type information of the effective network flow information. According to the method, the network traffic is directly filtered in the kernel mode by using the eBPF, so that data replication and context switching are avoided, and the filtering efficiency is improved. According to the method, the change of the network environment can be monitored in real time, and the filtering rule is dynamically adjusted according to the change, so that the accuracy and the flexibility of the filtering effect are ensured.

Description

Technical Field

[0001] The present invention relates to the field of network security and big data technology, and in particular to a network traffic storage pre-filtering method and device. Background Art

[0002] With the rapid development of the Internet and digital technology, network traffic has shown an explosive growth trend. Enterprises, organizations and individuals carry out various activities on the Internet, generating massive amounts of data and information. In order to effectively manage and utilize this data, network traffic storage is particularly important. Network traffic storage can not only ensure the integrity and reliability of data, but also provide an important data foundation for subsequent data analysis, business decision-making, etc.

[0003] However, in practical applications, network traffic storage faces many challenges. The most prominent problem is that large data traffic such as videos and games accounts for a large proportion of network traffic, and the value of this traffic for subsequent data analysis is often not high. This large data traffic not only takes up a lot of storage resources, but may also cause storage system overload, affecting overall performance and the efficiency of subsequent data analysis. Therefore, how to effectively filter these low-value traffic and alleviate storage pressure has become a problem that needs to be solved urgently.

[0004] In response to the challenges of network traffic filtering and storage, existing technologies have proposed some solutions, but there are still obvious shortcomings and defects:

[0005] (1) Filtering efficiency problem: The filtering methods in the prior art often rely on user-mode processing programs, which require frequent data copying and context switching between user mode and kernel mode. This not only increases processing time, but may also cause data loss or errors, reducing filtering efficiency. For large-scale network traffic processing, this inefficient filtering method obviously cannot meet the needs.

[0006] (2) Lack of dynamism and flexibility: Once the filtering rules of existing technologies are set, it is often difficult to adjust them in real time according to changes in the network environment and business needs. As network applications become more diverse and complex, traffic types and characteristics are constantly changing, and existing technologies cannot adapt to such changes, resulting in unsatisfactory filtering effects. Filtering methods that lack dynamism and flexibility cannot meet the needs of actual applications.

[0007] (3) Inaccurate traffic identification: The traffic identification methods in the prior art can only identify simple traffic features and cannot accurately identify complex traffic types and application protocols. This causes some low-value traffic to be misidentified as important traffic, occupying valuable storage resources. At the same time, some important traffic may also be misidentified as worthless traffic, resulting in data loss and misjudgment. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide a network traffic storage pre-filtering method and device, which is conducive to solving the problems of low filtering efficiency, lack of dynamics and flexibility, and inaccurate traffic identification existing in the existing network traffic filtering and storage technology.

[0009] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses a network traffic storage pre-filtering method, comprising:

[0010] S1, configure traffic filtering rules; the traffic filtering rules include IP address information, port information, protocol information and combination information of the network traffic to be filtered; the combination information is a combination of the set IP address information, port information and protocol information;

[0011] S2, obtaining initial network traffic; the network traffic includes data packets;

[0012] S3, filtering the network traffic using traffic filtering rules to obtain filtered network traffic information;

[0013] S4, performing application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information;

[0014] S5, using the valid network traffic information to update the traffic filtering rules;

[0015] S6, based on the application type information of the effective network traffic information, classify, store and process the effective network traffic information.

[0016] The filtering of the network traffic by using the traffic filtering rule to obtain filtered network traffic information includes:

[0017] S31, obtaining IP address information, port information, and protocol information of the network traffic;

[0018] S32, using the traffic filtering rule, matching the IP address information, port information and protocol information of the network traffic to obtain matching IP address information, port information and protocol information;

[0019] S33, deleting the network traffic corresponding to the matched IP address information, port information and protocol information from the initial network traffic to obtain filtered network traffic information.

[0020] The performing application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information includes:

[0021] S41, obtaining application type information of the filtered network traffic information;

[0022] S42, using the preset filtering application information, performing matching processing on the application type information to obtain matching application type information;

[0023] S43, determining network traffic information to be filtered based on the matching application type information;

[0024] S44, deleting the network traffic information to be filtered from the filtered network traffic information to obtain valid network traffic information.

[0025] The determining of the network traffic information to be filtered based on the matching application type information includes:

[0026] S431, performing quantization encoding processing on the matching application type information, the matching IP address information, the port information and the protocol information, respectively, to obtain an application digital sequence, an address digital sequence, a port digital sequence and a protocol digital sequence;

[0027] S432, constructing a matched filter matrix using the application digital sequence, the address digital sequence, the port digital sequence and the protocol digital sequence; the row vector of the matched filter matrix is ​​a digital sequence; the first to fourth row vectors of the matched filter matrix are the application digital sequence, the address digital sequence, the port digital sequence and the protocol digital sequence, respectively;

[0028] S433, performing decomposition processing on the matching filter matrix to obtain a left decomposition matrix, a feature matrix, and a right decomposition matrix of the matching filter matrix;

[0029] The decomposition process is calculated as follows:

[0030] Y=UQV,

[0031] Among them, Y is the matching filter matrix, U is the left decomposition matrix, Q is the feature matrix, V is the right decomposition matrix, and both U and V are orthogonal matrices;

[0032] S434, performing feature calculation processing on the feature matrix to obtain a first feature matrix H;

[0033] The expression of the feature calculation process is:

[0034]

[0035] Among them, q ki is the element of the kth row and ith column of the feature matrix Q, m and n are the row and column dimensions of the feature matrix Q respectively, h ki is the element of the kth row and ith column of the first characteristic matrix H;

[0036] S435, performing quantization decoding processing on the first row elements of the first feature matrix H to obtain corresponding application type information to be filtered and matched;

[0037] S436, determining that the network traffic information corresponding to the to-be-filtered matching application type information in the filtered network traffic information is the to-be-filtered network traffic information;

[0038] S437, performing quantization decoding processing on the elements of the second to fourth rows of the characteristic matrix H respectively to obtain updated IP address information, port information and protocol information.

[0039] The updating process of the traffic filtering rules by using the effective network traffic information includes:

[0040] The matched IP address information, port information and protocol information and the updated IP address information, port information and protocol information are added to the traffic filtering rules to complete the update process of the traffic filtering rules.

[0041] The classifying and storing the valid network traffic information based on the application type information of the valid network traffic information includes:

[0042] Using the preset locally stored application type information, matching the application type information of the valid network traffic information to obtain a first matching application type;

[0043] Locally storing the network traffic corresponding to the first matching application type;

[0044] Using the preset cloud storage application type information, matching the application type information of the valid network traffic information to obtain a second matching application type;

[0045] The network traffic corresponding to the second matching application type is stored in the cloud.

[0046] In a second aspect of the embodiment of the present invention, a network traffic storage pre-filtering device is disclosed, characterized in that it is used to implement the network traffic storage pre-filtering method, and comprises: a traffic filtering rule module, an application identification filtering module and a classification storage module;

[0047] The traffic filtering rule module is implemented based on the eBPF module deployed on the server, and is used to filter the network traffic using the traffic filtering rule to obtain filtered network traffic information; and update the traffic filtering rule using the valid network traffic information;

[0048] The application identification and filtering module is connected to the traffic filtering rule module and the classification storage module, and is used to perform application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information;

[0049] The classification storage module is used to classify and store the valid network traffic information based on the application type information of the valid network traffic information.

[0050] According to a third aspect of an embodiment of the present invention, a computer storable medium is disclosed, wherein the computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the network traffic storage pre-filtering method.

[0051] According to a fourth aspect of the embodiments of the present invention, an information data processing terminal is disclosed, and the information data processing terminal is used to implement the network traffic storage pre-filtering method.

[0052] The beneficial effects of the present invention are:

[0053] The present invention proposes a network traffic storage pre-filtering method and device based on eBPF and application identification. The method uses eBPF to directly filter network traffic in kernel state, avoids data duplication and context switching, and improves filtering efficiency. At the same time, in user state, the method uses application identification technology to identify the type of network traffic, and feeds back the identification results to kernel state in real time to update eBPF filtering rules. The collaborative work of user state and kernel state enables the filtering rules to be dynamically adjusted according to the network environment and business needs, further enhancing the efficiency and accuracy of filtering. By adopting this pre-filtering method and device, it is possible to effectively filter out low-value video, game and other traffic at the source of network traffic, that is, kernel state eBPF. This pre-filtering at the kernel state eBPF level not only prevents unnecessary data from entering the storage system, thereby alleviating storage pressure, but also significantly improves the overall performance of the storage system. At the same time, because the method of the present invention can monitor changes in the network environment in real time and dynamically adjust the filtering rules according to the changes, the accuracy and flexibility of the filtering effect are ensured. The advantages of the present invention include:

[0054] 1. Efficient filtering: Utilize the efficient filtering capability of eBPF to perform preliminary filtering of traffic at the operating system kernel level, reducing processing complexity and latency.

[0055] 2. Accurate identification: The application identification module can accurately identify the application type of the traffic, providing accurate data support for subsequent filtering and storage.

[0056] 3. Dynamic adaptation: Dynamically adjust eBPF filtering rules based on the identification results of the application type to maintain continuous optimization and adaptability of the filtering effect.

[0057] 4. Intelligent storage: Through intelligent storage management, only the traffic that meets user needs and filtering rules is stored, which improves storage efficiency. At the same time, differentiated storage is performed according to the application type and importance of the traffic, further optimizing the utilization of storage space.

[0058] 5. The device adopts modular design, which is easy to expand and maintain, and can be widely used in various network environments and application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 1 is a flow chart for implementing the method of the present invention;

[0060] Figure 2 It is a schematic diagram of the composition of the device of the present invention. DETAILED DESCRIPTION

[0061] In order to better understand the content of the present invention, two embodiments are given here.

[0062] Figure 1 1 is a flow chart for implementing the method of the present invention; Figure 2 It is a schematic diagram of the composition of the device of the present invention.

[0063] Embodiment 1:

[0064] A first aspect of an embodiment of the present invention discloses a network traffic storage pre-filtering method, comprising:

[0065] S1, configure traffic filtering rules; the traffic filtering rules include IP address information, port information, protocol information and combination information of the network traffic to be filtered; the combination information is a combination of the set IP address information, port information and protocol information;

[0066] S2, obtaining initial network traffic; the network traffic includes data packets;

[0067] S3, using the traffic filtering rule to filter the network traffic to obtain filtered network traffic information; the traffic filtering rule is implemented based on the eBPF module deployed on the server;

[0068] S4, performing application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information;

[0069] S5, using the valid network traffic information to update the traffic filtering rules;

[0070] S6, based on the application type information of the effective network traffic information, classify, store and process the effective network traffic information; the application type information includes video, game, audio, office, finance, instant communication, email and picture.

[0071] The filtering of the network traffic by using the traffic filtering rule to obtain filtered network traffic information includes:

[0072] S31, obtaining IP address information, port information, and protocol information of the network traffic;

[0073] S32, using the traffic filtering rule, matching the IP address information, port information and protocol information of the network traffic to obtain matching IP address information, port information and protocol information;

[0074] S33, deleting the network traffic corresponding to the matched IP address information, port information and protocol information from the initial network traffic to obtain filtered network traffic information.

[0075] The performing application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information includes:

[0076] S41, obtaining application type information of the filtered network traffic information;

[0077] The obtaining of the application type information of the filtered network traffic information is achieved by using an application identification algorithm, including a deep learning algorithm, etc.

[0078] S42, using the preset filtering application information, performing matching processing on the application type information to obtain matching application type information;

[0079] S43, determining network traffic information to be filtered based on the matching application type information;

[0080] S44, deleting the network traffic information to be filtered from the filtered network traffic information to obtain valid network traffic information.

[0081] The using the preset filtering application information to match the application type information to obtain the matching application type information includes:

[0082] The application type information identical to the preset filtering application information is searched and the identical application type information is determined to be the matching application type information.

[0083] Determining the network traffic information to be filtered based on the matching application type information is to determine that the network traffic information corresponding to the matching application type information is the network traffic information to be filtered.

[0084] The determining of the network traffic information to be filtered based on the matching application type information includes:

[0085] S431, performing quantization encoding processing on the matching application type information, the matching IP address information, the port information and the protocol information, respectively, to obtain an application digital sequence, an address digital sequence, a port digital sequence and a protocol digital sequence;

[0086] S432, constructing a matched filter matrix using the application digital sequence, the address digital sequence, the port digital sequence and the protocol digital sequence; the row vector of the matched filter matrix is ​​a digital sequence; the first to fourth row vectors of the matched filter matrix are the application digital sequence, the address digital sequence, the port digital sequence and the protocol digital sequence, respectively;

[0087] S433, performing decomposition processing on the matching filter matrix Y to obtain a left decomposition matrix, a feature matrix, and a right decomposition matrix of the matching filter matrix;

[0088] The decomposition process is calculated as follows:

[0089] Y=UQV,

[0090] Among them, U is the left decomposition matrix, Q is the characteristic matrix, V is the right decomposition matrix, and both U and V are orthogonal matrices; the decomposition process can be implemented by using a matrix singular value decomposition algorithm.

[0091] S434, performing feature calculation processing on the feature matrix to obtain a feature matrix H;

[0092] The expression of the feature calculation process is:

[0093]

[0094] Among them, q ki is the element in the kth row and ith column of the feature matrix Q, and m and n are the row dimension and column dimension of the feature matrix Q respectively;

[0095] S435, performing quantization decoding processing on the first row elements of the feature matrix H to obtain corresponding application type information to be filtered and matched;

[0096] S436, determining that the network traffic information corresponding to the to-be-filtered matching application type information in the filtered network traffic information is the to-be-filtered network traffic information;

[0097] S437, performing quantization decoding processing on the elements of the second to fourth rows of the characteristic matrix H respectively to obtain updated IP address information, port information and protocol information.

[0098] The step S431 includes: encoding various types of IP address information, port information, protocol information, and application type information to obtain corresponding relationships between various types of information and numbers;

[0099] By using the corresponding relationship, the matching application type information, the matching IP address information, the port information and the protocol information are mapped into corresponding digital sequences. When encoding the protocol information, the encoding process is performed according to the type of the protocol information.

[0100] The quantization decoding process utilizes the corresponding relationship to reversely map the digital sequence into corresponding matching application type information, matching IP address information, port information and protocol information.

[0101] The updating process of the traffic filtering rules by using the effective network traffic information includes:

[0102] The matched IP address information, port information and protocol information and the updated IP address information, port information and protocol information are added to the traffic filtering rules to complete the update process of the traffic filtering rules.

[0103] The classifying and storing the valid network traffic information based on the application type information of the valid network traffic information includes:

[0104] Using the preset locally stored application type information, matching the application type information of the valid network traffic information to obtain a first matching application type;

[0105] Locally storing the network traffic corresponding to the first matching application type;

[0106] Using the preset cloud storage application type information, matching the application type information of the valid network traffic information to obtain a second matching application type;

[0107] storing the network traffic corresponding to the second matching application type in the cloud;

[0108] The preset local storage application type information includes office, finance, and email;

[0109] The preset cloud storage application type information includes instant messaging and pictures.

[0110] The second aspect of the embodiment of the present invention discloses a network traffic storage pre-filtering device, which is used to implement the network traffic storage pre-filtering method, including: a traffic filtering rule module, an application identification filtering module and a classification storage module;

[0111] The traffic filtering rule module is implemented based on the eBPF module or IPtable module deployed on the server, and is used to filter the network traffic using the traffic filtering rule to obtain filtered network traffic information; and update the traffic filtering rule using the valid network traffic information;

[0112] The application identification and filtering module is connected to the traffic filtering rule module and the classification storage module, and is used to perform application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information;

[0113] The classification storage module is used to classify and store the valid network traffic information based on the application type information of the valid network traffic information.

[0114] Embodiment 2:

[0115] The present invention provides a network traffic storage pre-filtering method based on eBPF and application identification, wherein the eBPF is deployed on a server; the server adopts a Linux system and is connected to the Internet; the method comprises:

[0116] S1, initialization;

[0117] S11, eBPF rule configuration: complete the initial configuration of the eBPF module;

[0118] Configure BPF filtering rules; BPF filtering rules are used to filter network traffic based on IP addresses, ports, and protocols. This configuration uses a more efficient and flexible rule configuration method and supports custom rules, such as discarding traffic from specific IP addresses or allowing only traffic from specific ports to pass. At the same time, the rule expression method uses extended tcpdump and libpcap syntax to provide more powerful filtering capabilities.

[0119] S12, application identification module initialization; loading application identification algorithm (including machine learning algorithm) to realize the identification of application type of traffic, such as video stream, game data, etc., which can be accurately identified. The application types of traffic include video, game, audio, office, finance, instant communication, email, and picture;

[0120] S13, storage configuration: complete the initial configuration of the server's storage, including storage methods, storage strategies, etc., to ensure data security and efficiency.

[0121] S2, receiving and preliminary filtering of network traffic: the system seamlessly receives data streams from the network through the network interface. Once the data stream enters the system, the eBPF filtering module will quickly perform preliminary filtering on it, and this process is completed entirely in the kernel space.

[0122] S21, obtaining network traffic information; the network traffic information includes data packets;

[0123] S22, eBPF filtering process: The efficiency of the eBPF filtering mechanism is reflected in that as soon as a data packet arrives at the network interface, it is immediately processed by the eBPF module. The first step of processing is to check the header information of the data packet, which includes key elements such as source / destination IP address, port number, protocol type, etc. Next, eBPF will match according to the predefined filtering rules.

[0124] This matching process relies on a series of carefully designed hook functions, which are the starting points of eBPF programs and can be activated when the data packet traverses different stages of the network stack. For example, a hook function may focus on checking the destination port number of the data packet to determine whether it meets the conditions of a specific rule. Once a match is successful, the corresponding action will be performed, such as discarding the data packet or allowing it to continue to pass. This mechanism ensures that the eBPF filtering module can perform preliminary screening before the data packet reaches the user space. This not only greatly reduces the impact of unnecessary traffic on the user space, but also significantly improves the overall network performance and security.

[0125] S23, Application of rules: The application of eBPF rules is achieved by loading eBPF programs in kernel space. These programs consist of user-defined filtering logic and are executed when packets pass through the network stack. Since eBPF programs run in kernel space, they are able to perform filtering operations with extremely low latency, effectively reducing the impact on system performance. Through eBPF's efficient filtering capabilities and flexible configuration mechanism, the system is able to achieve preliminary filtering of network traffic, effectively reducing unnecessary traffic entering user space, and improving overall network performance and security.

[0126] S3, application identification; after preliminary filtering, the traffic is imported into the user space, and the application identification algorithm is used to deeply analyze the traffic and identify the application type of the traffic, such as video streams, audio streams, game data and other big data application types.

[0127] S4, dynamic rule update and real-time effectiveness: In the process of network traffic management, dynamic rule update and real-time effectiveness mechanism is crucial. This mechanism enables the system to quickly adapt to changes in the network environment and ensure that only qualified traffic enters the network.

[0128] S41, dynamic rule update: When the application identification module successfully identifies the application type of the traffic, the system will capture and record the five-tuple information of the traffic. If the application type is identified as a video, audio, or game application, the corresponding eBPF rule syntax will be generated based on the five-tuple information of these applications and automatically added to the eBPF filter rule configuration.

[0129] S42, real-time effect: Compared with the traditional filtering rule update method, the dynamic rule update mechanism in this patent has the characteristic of real-time effect. Once new filtering rules are added to the eBPF filtering rule list, they will take effect immediately without restarting the system or reloading the rules. This mechanism greatly reduces the time and complexity required for rule updates and improves the flexibility and responsiveness of the system.

[0130] Through dynamic rule updates and real-time effectiveness mechanisms, the system can adjust filtering rules in real time to adapt to changes in the network environment and meet business needs. This helps ensure efficient operation, security and stability of the network.

[0131] S5, storage of network traffic; after eBPF and application filtering, the qualified network traffic data will be stored for subsequent analysis and processing.

[0132] S51, Storage method selection: The system will select the appropriate storage method for saving according to the user's needs and configuration. Common storage methods include local storage and cloud storage. Local storage is suitable for scenarios (applications and quintuples) with high requirements for data security and privacy, while cloud storage provides larger storage space and better data backup and recovery capabilities.

[0133] S52, data storage strategy: formulate intelligent storage strategy according to the application type and importance of traffic. For example, for important traffic that occupies a large space, use compressed storage; for unimportant or temporary traffic, choose not to store or store it briefly.

[0134] S53, data storage: according to the formulated storage strategy, the traffic that meets the conditions after filtering is stored in the corresponding storage medium. At the same time, according to the usage of storage space, the storage strategy is dynamically adjusted to ensure storage efficiency and reasonable use of storage space.

[0135] In addition, the solution of the present invention also provides monitoring and display, feedback loop and continuous optimization functions, as follows:

[0136] S6, Monitoring and Display: In order to ensure the effective management of network traffic and the stable operation of the system, the monitoring and display function is also crucial. By real-time monitoring of total network traffic, filtering (such as five-tuple information, application type, number of packets, number of bytes, application proportion, etc.), storage (such as five-tuple information, application type, number of packets, number of bytes, application proportion, etc.), storage status, and system usage, the administrator can timely understand the system's operating status and adjust filtering and storage policies as needed.

[0137] S7, feedback loop and continuous optimization: In order to ensure the efficient operation of the network traffic management solution and flexibly adapt to various network environments and business needs, the present invention introduces a feedback loop and continuous optimization mechanism. The core of this mechanism is to ensure that the system always maintains the best working state through continuous data collection, performance evaluation and strategy adjustment.

[0138] S71, Data Collection and Performance Evaluation: Data collection, the system will continuously and comprehensively collect data related to network traffic, filtering rule efficiency, etc. These data not only cover basic information such as the source, destination, and protocol type of the traffic, but also include key information such as the execution of filtering rules and traffic identification results. Performance evaluation, based on the collected data, the system will conduct an in-depth performance evaluation of its current filtering and storage strategies. The evaluation focuses on the effectiveness of filtering rules, storage efficiency, etc.

[0139] S72, Monitoring and Risk Identification: The system also monitors network traffic in real time, paying special attention to abnormal behaviors, such as sudden traffic growth, abnormal application types, etc. This monitoring helps to timely identify potential security risks or performance issues, and provides a basis for subsequent policy adjustments.

[0140] S73, continuous optimization strategy: Based on the results of the feedback loop, the system will intelligently adjust and optimize the filtering rules. For example, for rules that are too strict, the system may appropriately relax the conditions or delete the rule; for rules that are too loose, the system will strengthen the conditions to improve the filtering effect. At the same time, according to the dynamic changes in network traffic and the characteristics of application types, the system will flexibly adjust the storage strategy to optimize storage efficiency and space utilization. For example, for less important or temporary traffic, the system may choose compressed storage or short-term storage. Through this feedback loop and continuous optimization mechanism, the system can continue to learn and improve to ensure that it always remains in the best state of efficiency and strong adaptability.

[0141] On the other hand, this embodiment discloses a network traffic storage pre-filtering device based on eBPF and application identification, such as Figure 2 As shown, it includes: initialization module, network traffic receiving and preliminary filtering module, application identification module, dynamic rule management module, traffic storage module, monitoring and display module, feedback loop and optimization module. The details are as follows:

[0142] 1. Initialize the module

[0143] This module contains the following three modules:

[0144] eBPF rule configuration module: defines eBPF filtering rules to ensure preliminary filtering of network traffic, which is efficient and flexible.

[0145] Application identification module: uses advanced identification algorithms to identify the application type of traffic, providing a basis for subsequent filtering and storage.

[0146] Storage configuration module: select appropriate storage methods and strategies based on user needs and system operation status to ensure data security and storage efficiency.

[0147] 2. Network traffic receiving and preliminary filtering module

[0148] This module is responsible for receiving data streams from the network and performing preliminary filtering in the kernel space according to the eBPF rule filtering configuration.

[0149] Specifically, it includes: (1) Receiving traffic: The module will monitor the network interface and capture the network traffic entering the system. (2) Preliminary filtering: Using the rules in the eBPF filtering configuration, deep packet inspection is performed on the traffic in the kernel space. Traffic that does not meet the requirements (such as video and game application traffic) will be discarded.

[0150] 3. Application identification module

[0151] Advanced recognition algorithms are used to identify the application type of traffic, providing a basis for subsequent filtering and storage.

[0152] 4. Dynamic rule management module

[0153] Dynamically adjust filtering rules based on application identification results to ensure that the rules take effect in real time and improve the adaptability and security of the system.

[0154] 5. Traffic storage module

[0155] According to the storage configuration, the traffic data that meets the conditions after filtering is stored in the corresponding media to ensure the integrity and analyzability of the data.

[0156] 6. Monitoring and display module

[0157] This module is responsible for real-time monitoring of various indicators of network traffic and system status, and displays them through an intuitive user interface, allowing administrators to understand the operating status of the system in a timely manner and adjust filtering and storage policies as needed.

[0158] 7. Feedback loop and optimization module

[0159] This module continuously collects and analyzes data, evaluates the performance of filtering and storage policies, monitors network traffic in real time to identify potential risks, and intelligently adjusts and optimizes rules based on feedback results, thereby maintaining the system in optimal working condition, improving network security, performance, and stability, and meeting growing business needs.

[0160] Through the detailed description of these modules, it can be seen in the present invention that each module plays a specific role and works closely with other modules to jointly achieve efficient management and optimization of network traffic.

[0161] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A network traffic storage pre-filtering method, It is characterized in that include: S1, configure traffic filtering rules; the traffic filtering rules include IP address information, port information, protocol information and combination information of the network traffic to be filtered; the combination information is a combination of the set IP address information, port information and protocol information; S2, obtain the initial network traffic; S3, filtering the network traffic using traffic filtering rules to obtain filtered network traffic information; S4, performing application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information; S5, using the valid network traffic information to update the traffic filtering rules; S6, based on the application type information of the effective network traffic information, classify, store and process the effective network traffic information.

2. The network traffic storage pre-filtering method according to claim 1, It is characterized in that The filtering of the network traffic by using the traffic filtering rule to obtain filtered network traffic information includes: S31, obtaining IP address information, port information, and protocol information of the network traffic; S32, using the traffic filtering rule, matching the IP address information, port information and protocol information of the network traffic to obtain matching IP address information, port information and protocol information; S33, deleting the network traffic corresponding to the matched IP address information, port information and protocol information from the initial network traffic to obtain filtered network traffic information.

3. The network traffic storage pre-filtering method according to claim 2, It is characterized in that The performing application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information includes: S41, obtaining application type information of the filtered network traffic information; S42, using the preset filtering application information, performing matching processing on the application type information to obtain matching application type information; S43, determining network traffic information to be filtered based on the matching application type information; S44, deleting the network traffic information to be filtered from the filtered network traffic information to obtain valid network traffic information.

4. The network traffic storage pre-filtering method according to claim 3, It is characterized in that The determining of the network traffic information to be filtered based on the matching application type information includes: S431, performing quantization encoding processing on the matching application type information, the matching IP address information, the port information and the protocol information, respectively, to obtain an application digital sequence, an address digital sequence, a port digital sequence and a protocol digital sequence; S432, constructing a matched filter matrix using the application digital sequence, the address digital sequence, the port digital sequence and the protocol digital sequence; the row vector of the matched filter matrix is ​​a digital sequence; the first to fourth row vectors of the matched filter matrix are the application digital sequence, the address digital sequence, the port digital sequence and the protocol digital sequence, respectively; S433, performing decomposition processing on the matching filter matrix to obtain a left decomposition matrix, a feature matrix, and a right decomposition matrix of the matching filter matrix; The decomposition process is calculated as follows: Y=UQV, Among them, Y is the matching filter matrix, U is the left decomposition matrix, Q is the feature matrix, V is the right decomposition matrix, and both U and V are orthogonal matrices; S434, performing feature calculation processing on the feature matrix to obtain a first feature matrix H; The expression of the feature calculation process is: Among them, q ki is the element of the kth row and ith column of the feature matrix Q, m and n are the row and column dimensions of the feature matrix Q respectively, h ki is the element of the kth row and ith column of the first characteristic matrix H; S435, performing quantization decoding processing on the first row elements of the first feature matrix H to obtain corresponding application type information to be filtered and matched; S436, determining that the network traffic information corresponding to the to-be-filtered matching application type information in the filtered network traffic information is the to-be-filtered network traffic information; S437, performing quantization decoding processing on the elements of the second to fourth rows of the characteristic matrix H respectively to obtain updated IP address information, port information and protocol information.

5. The network traffic storage pre-filtering method according to claim 4, It is characterized in that The updating process of the traffic filtering rules by using the effective network traffic information includes: The matched IP address information, port information and protocol information and the updated IP address information, port information and protocol information are added to the traffic filtering rules to complete the update process of the traffic filtering rules.

6. The network traffic storage pre-filtering method according to claim 4, It is characterized in that The classifying and storing the valid network traffic information based on the application type information of the valid network traffic information includes: Using the preset locally stored application type information, matching the application type information of the valid network traffic information to obtain a first matching application type; Locally storing the network traffic corresponding to the first matching application type; Using the preset cloud storage application type information, matching the application type information of the valid network traffic information to obtain a second matching application type; The network traffic corresponding to the second matching application type is stored in the cloud.

7. A network traffic storage pre-filtering device, It is characterized in that Used to implement the network traffic storage pre-filtering method as claimed in any one of claims 1 to 6, comprising: a traffic filtering rule module, an application identification filtering module and a classification storage module; The traffic filtering rule module is implemented based on the eBPF module deployed on the server, and is used to filter the network traffic using the traffic filtering rule to obtain filtered network traffic information; and update the traffic filtering rule using the valid network traffic information; The application identification and filtering module is connected to the traffic filtering rule module and the classification storage module, and is used to perform application identification and filtering processing on the filtered network traffic information to obtain valid network traffic information; The classification storage module is used to classify and store the valid network traffic information based on the application type information of the valid network traffic information.

8. A network traffic storage pre-filtering device, It is characterized in that The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the network traffic storage pre-filtering method according to any one of claims 1-6.

9. A computer storable medium, It is characterized in that The computer storable medium stores computer instructions, and when the computer instructions are called, they are used to execute the network traffic storage pre-filtering method according to any one of claims 1 to 6.

10. An information data processing terminal, It is characterized in that The information data processing terminal is used to implement the network traffic storage pre-filtering method described in any one of claims 1-6.

Images