NetFlow traffic processing method and device

Through multi-level and multi-dimensional processing and load balancing technology, NetFlow data processing speed lag and resource utilization unbalanced problems are solved, and efficient traffic analysis services are realized.

CN118984336APending Publication Date: 2024-11-19CHINA UNITECHS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410970663.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The existing NetFlow data processing method lags in the processing speed when facing massive traffic, resulting in a large number of packet loss and the unbalanced resource utilization of proxy servers, affecting the accuracy and timeliness of traffic analysis.

Method used

Through multi-level and multi-dimensional processing of NetFlow traffic, the LZ4 compression algorithm is used to classify and store the data, and the load is automatically adjusted using preset thresholds and HTB algorithms to ensure load balancing of the proxy server, and the mapping relationship between routers, port indexes and acquisition directions is used for accurate results Filter and statistics.

Benefits of technology

It improves the processing efficiency of NetFlow traffic, balances the resource utilization rate of proxy servers, and provides operators with more accurate and timely traffic analysis services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984336A_ABST
    Figure CN118984336A_ABST
Patent Text Reader

Abstract

The invention discloses a NetFlow traffic processing method and device, and the method comprises the steps: writing a mapping relation among routers, an acquisition port index and an acquisition direction into a configuration file, and pre-configuring a default router number associated with a proxy server and a corresponding router IP; compressing the collected NetFlow traffic by using an LZ4 compression algorithm, and storing the compressed NetFlow traffic into different hdfs directories according to router IPs and dates in a classified manner; the proxy server reads the corresponding NetFlow flow in the hdfs according to the associated router; the resource utilization rate of the proxy server is monitored in real time, and the load is automatically adjusted through a preset threshold value and an HTB algorithm; according to the method, the pre-established mapping relation among the router, the port index and the acquisition direction is utilized to accurately filter the NetFlow flow, and then the filtered NetFlow flow is counted and displayed. According to the method and the device, the NetFlow flow is processed, so that the NetFlow flow processing efficiency is improved, the resource utilization rate of the proxy server is balanced, and accurate and timely flow analysis service is provided for an operator.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of communications, and in particular to a NetFlow traffic processing method and device. Background Art

[0002] As an effective tool for measuring and counting IP traffic flowing through routers, NetFlow can help operators gain in-depth insights into traffic trends. However, in actual applications, routers generate a huge amount of NetFlow data. If it is not processed in time, it is very easy to cause a large number of NetFlow packet loss, which seriously affects the accuracy of subsequent analysis and processing.

[0003] At present, the common way to process NetFlow data is to store the collected data locally before marking and analyzing it. However, when faced with massive NetFlow traffic, this method often cannot keep up with the speed at which NetFlow traffic is generated, resulting in processing delays. In addition, due to the large number of routers of operators and the uneven scale of NetFlow traffic processed by each router, the resource utilization of the proxy server is extremely uneven when processing NetFlow traffic from different routers. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a NetFlow traffic processing method and device, which processes NetFlow traffic at multiple levels and dimensions, improves the processing efficiency of NetFlow traffic, balances the resource utilization rate of the proxy server, and provides operators with more accurate and timely traffic analysis services.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] In one embodiment of the present invention, a NetFlow traffic processing method is proposed, the method comprising:

[0007] S01, write the mapping relationship between routers, collection port indexes and collection directions into the configuration file, and pre-configure the default number of routers associated with the proxy server and the corresponding router IP;

[0008] Further, the S01 includes:

[0009] S011. Collect all router IPs, collection port indexes and collection directions, and write the mapping relationship into the configuration file;

[0010] Furthermore, the configuration file format is: router IP: collection port index-collection direction.

[0011] S012. Setting proxy server configuration information according to the information collected in S011, wherein the proxy server configuration information includes: server name, server IP, number of routers associated with the server, IP of routers associated with the server, resource status of the server, resource usage limit threshold and resource usage maximum threshold.

[0012] Furthermore, the resource status of the server includes: sufficient resources, moderate resources, and insufficient resources.

[0013] Furthermore, when the source usage rate is less than a specified threshold, the resource status is sufficient resources; when the resource usage rate is greater than the specified threshold and less than the maximum threshold, the resource status is moderate resources; when the resource status is greater than the maximum threshold, the resource status is insufficient resources.

[0014] S02. Use the LZ4 compression algorithm to compress the collected NetFlow traffic, and store it in different HDFS directories according to the router IP and date.

[0015] NetFlow traffic is stored by router IP and date to facilitate subsequent data retrieval and processing.

[0016] S03, the proxy server reads the corresponding NetFlow traffic in hdfs according to the associated router;

[0017] S04. Monitor the resource usage of the proxy server in real time, and automatically adjust the load through the preset threshold and HTB algorithm to ensure that the load of each proxy server remains balanced;

[0018] Further, the S04 includes:

[0019] S041. If the resource usage rate of the proxy server does not exceed the specified threshold, the resource status is set to sufficient resources, and the system does not need to actively adjust the router associated with the proxy server and continues real-time monitoring;

[0020] S042. If the resource usage of the proxy server exceeds a specified threshold, set the resource status to moderate resources;

[0021] When the resource usage of other proxy servers exceeds the maximum threshold, their associated routers will not be transferred to proxy servers with moderate resources.

[0022] S043. If the resource usage of the proxy server exceeds the maximum threshold, the resource status is set to insufficient resources, and the system automatically searches for other proxy servers with low usage;

[0023] Further, the S043 includes:

[0024] S0431. If there are no other proxy servers with low usage, the system will adopt a queuing mechanism to temporarily store the NetFlow traffic to be processed by the proxy server;

[0025] S0432. If there are other proxy servers with low utilization, the system automatically adjusts the load, reduces the number of routers associated with the proxy server, and transfers the associated routers to other proxy servers with lower resource utilization.

[0026] S044. If the number of routers associated with the proxy server has been reduced and the resource usage of the proxy server is still increasing, the system will send an alarm message to the administrator;

[0027] Furthermore, in S044, the system sends the warning information via short line, email or other means.

[0028] S045. When the resource usage rate of the proxy server is lower than a defined threshold, the resource state is set to sufficient resources. At this time, the number of routers associated with the proxy server can be increased to ensure that the load of each proxy server remains balanced.

[0029] S05. Utilize the pre-established mapping relationship between routers, port indexes, and collection directions to accurately filter NetFlow traffic, and then count and display the filtered NetFlow traffic.

[0030] Further, the S05 includes:

[0031] S051, the system uses mapping relationships to accurately filter NetFlow traffic;

[0032] S052. Count and display the filtered NetFlow traffic.

[0033] In one embodiment of the present invention, a NetFlow traffic processing device is also proposed, the device comprising:

[0034] The mapping relationship writing module writes the mapping relationship between routers, collection port indexes and collection directions into the configuration file, and pre-configures the default number of routers associated with the proxy server and the corresponding router IPs;

[0035] The traffic compression module uses the LZ4 compression algorithm to compress the collected NetFlow traffic and stores it in different HDFS directories according to the router IP and date.

[0036] Traffic reading module, the proxy server reads the corresponding NetFlow traffic in hdfs according to the associated router;

[0037] The load balancing module monitors the resource usage of the proxy server in real time, and automatically adjusts the load through the preset threshold and HTB algorithm to ensure that the load of each proxy server remains balanced;

[0038] The traffic filtering module uses the pre-established mapping relationship between routers, port indexes, and collection directions to accurately filter NetFlow traffic, and then counts and displays the filtered NetFlow traffic.

[0039] In one embodiment of the present invention, a computer device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the aforementioned NetFlow traffic processing method is implemented.

[0040] In one embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program for executing the NetFlow traffic processing method.

[0041] Beneficial effects:

[0042] The present invention provides a NetFlow traffic processing method and device, which processes NetFlow traffic in multiple levels and dimensions through grayscale publishing services, improves the processing efficiency of NetFlow traffic, balances the resource utilization rate of the proxy server, and provides operators with more accurate and timely traffic analysis services. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the NetFlow traffic processing method of the present invention;

[0044] Figure 2 It is a schematic diagram of the structure of the NetFlow traffic processing device of the present invention;

[0045] Figure 3 It is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The principles and spirit of the present invention will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present invention, and are not intended to limit the scope of the present invention in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0047] Those skilled in the art will appreciate that the embodiments of the present invention may be implemented as a system, device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0048] According to the implementation mode of the present invention, a NetFlow traffic processing method and device are proposed, which improve the processing efficiency of NetFlow traffic by processing NetFlow traffic at multiple levels and dimensions, balance the resource utilization rate of the proxy server, and provide operators with more accurate and timely traffic analysis services.

[0049] The principle and spirit of the present invention are explained in detail below with reference to several representative embodiments of the present invention.

[0050] like Figure 1 , Figure 2 As shown, the present invention relates to a NetFlow traffic processing method, the method comprising:

[0051] S01, write the mapping relationship between routers, collection port indexes and collection directions into the configuration file, and pre-configure the default number of routers associated with the proxy server and the corresponding router IP;

[0052] The S01 includes:

[0053] S011. Collect all router IPs, collection port indexes and collection directions, and write the mapping relationship into the configuration file;

[0054] The configuration file format is: router IP: collection port index-collection direction.

[0055] S012. Setting proxy server configuration information according to the information collected in S011, wherein the proxy server configuration information includes: server name, server IP, number of routers associated with the server, IP of routers associated with the server, resource status of the server, resource usage limit threshold and resource usage maximum threshold.

[0056] The resource status of the server includes: sufficient resources, moderate resources, and insufficient resources.

[0057] When the source usage rate is less than the specified threshold, the resource status is sufficient resources; when the resource usage rate is greater than the specified threshold and less than the maximum threshold, the resource status is moderate resources; when the resource status is greater than the maximum threshold, the resource status is insufficient resources.

[0058] S02. Use the LZ4 compression algorithm to compress the collected NetFlow traffic, and store it in different HDFS directories according to the router IP and date.

[0059] NetFlow traffic is stored by router IP and date to facilitate subsequent data retrieval and processing.

[0060] S03, the proxy server reads the corresponding NetFlow traffic in hdfs according to the associated router;

[0061] S04. Monitor the resource usage of the proxy server in real time, and automatically adjust the load through the preset threshold and HTB algorithm to ensure that the load of each proxy server remains balanced;

[0062] The S04 includes:

[0063] S041. If the resource usage rate of the proxy server does not exceed the specified threshold, the resource status is set to sufficient resources, and the system does not need to actively adjust the router associated with the proxy server and continues real-time monitoring;

[0064] S042. If the resource usage of the proxy server exceeds a specified threshold, set the resource status to moderate resources;

[0065] When the resource usage of other proxy servers exceeds the maximum threshold, their associated routers will not be transferred to proxy servers with moderate resources.

[0066] S043. If the resource usage of the proxy server exceeds the maximum threshold, the resource status is set to insufficient resources, and the system automatically searches for other proxy servers with low usage;

[0067] The S043 includes:

[0068] S0431. If there are no other proxy servers with low usage, the system will adopt a queuing mechanism to temporarily store the NetFlow traffic to be processed by the proxy server;

[0069] S0432. If there are other proxy servers with low utilization, the system automatically adjusts the load, reduces the number of routers associated with the proxy server, and transfers the associated routers to other proxy servers with lower resource utilization.

[0070] S044. If the number of routers associated with the proxy server has been reduced and the resource usage of the proxy server is still increasing, the system will send an alarm message to the administrator;

[0071] In the S044, the system sends the warning information via short line, email or other means.

[0072] S045. When the resource usage rate of the proxy server is lower than a defined threshold, the resource state is set to sufficient resources. At this time, the number of routers associated with the proxy server can be increased to ensure that the load of each proxy server remains balanced.

[0073] S05. Utilize the pre-established mapping relationship between routers, port indexes, and collection directions to accurately filter NetFlow traffic, and then count and display the filtered NetFlow traffic.

[0074] The S05 includes:

[0075] S051, the system uses mapping relationships to accurately filter NetFlow traffic;

[0076] S052. Count and display the filtered NetFlow traffic.

[0077] It should be noted that, although the operations of the method of the present invention are described in a specific order in the above embodiments and the accompanying drawings, this does not require or imply that the operations must be performed in the specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0078] In order to explain the above NetFlow traffic processing method more clearly, it is described below in conjunction with a specific embodiment. However, it should be noted that this embodiment is only for better illustrating the present invention and does not constitute an improper limitation to the present invention.

[0079] S01, write the mapping relationship between routers, collection port indexes and collection directions into the configuration file, and pre-configure the default number of routers associated with the proxy server and the corresponding router IP;

[0080] S011. The operation and maintenance personnel collect all router IPs, collection port indexes and collection directions, and write the mapping relationship into the configuration file;

[0081] The configuration file format is: router IP: collection port index-collection direction.

[0082] In specific implementation, if the router IP is 192.168.6.1, the collection port index is 749, and the collection direction is inbound, then the mapping relationship is 192.168.6.1:749-in.

[0083] S012. The operation and maintenance personnel set the proxy server configuration information according to the collected information, wherein the proxy server configuration information includes: server name, server IP, number of routers associated with the server, IP of routers associated with the server, resource status of the server, resource usage limit threshold and resource usage maximum threshold.

[0084] The resource status of the server includes: sufficient resources, moderate resources, and insufficient resources.

[0085] When the source usage rate is less than the specified threshold, the resource status is sufficient resources; when the resource usage rate is greater than the specified threshold and less than the maximum threshold, the resource status is moderate resources; when the resource status is greater than the maximum threshold, the resource status is insufficient resources.

[0086] The router information associated with the proxy server is stored in the database. The field format is shown in Table 1 below:

[0087]

[0088] Table 1

[0089] If the default number of proxy servers is 5 and the number of routers is 100, then the default number of routers handled by each proxy server is 20. Write the proxy servers and the corresponding number of routers and router IP list into the database.

[0090] S02. Use the LZ4 compression algorithm to compress the collected NetFlow traffic, and store it in different HDFS directories according to the router IP and date.

[0091] NetFlow traffic is stored by router IP and date to facilitate subsequent data retrieval and processing.

[0092] The router IP is 192.168.2.10, and the current date is 20240601, so the hdfs storage directory is: / data / 192.168.2.10 / 20240601.

[0093] S03, the proxy server reads the corresponding NetFlow traffic in hdfs according to the associated router;

[0094] The proxy server serviceA is associated with routers 192.168.2.10 and 192.168.2.11, so serviceA only needs to read the NetFlow traffic of these two routers.

[0095] S04. Monitor the resource usage of the proxy server in real time, and automatically adjust the load through the preset threshold and HTB algorithm to ensure that the load of each proxy server remains balanced;

[0096] The S04 includes:

[0097] S041. If the resource usage rate of the proxy server does not exceed the specified threshold, the resource status is set to sufficient resources, and the system does not need to actively adjust the router associated with the proxy server and continues real-time monitoring;

[0098] S042. If the resource usage of the proxy server exceeds a specified threshold, set the resource status to moderate resources;

[0099] When the resource usage of other proxy servers exceeds the maximum threshold, their associated routers will not be transferred to proxy servers with moderate resources.

[0100] S043. If the resource usage of the proxy server exceeds the maximum threshold, the resource status is set to insufficient resources, and the system automatically searches for other proxy servers with low usage;

[0101] The S043 includes:

[0102] S0431. If there are no other proxy servers with low usage, the system will adopt a queuing mechanism to temporarily store the NetFlow traffic to be processed by the proxy server;

[0103] S0432. If there are other proxy servers with low utilization, the system automatically adjusts the load, reduces the number of routers associated with the proxy server, and transfers the associated routers to other proxy servers with lower resource utilization.

[0104] S044. If the number of routers associated with the proxy server has been reduced and the resource usage of the proxy server is still increasing, the system will send an alarm message to the administrator;

[0105] In the S044, the system sends the warning information via short line, email or other means.

[0106] S045. When the resource usage rate of the proxy server is lower than a defined threshold, the resource state is set to sufficient resources. At this time, the number of routers associated with the proxy server can be increased to ensure that the load of each proxy server remains balanced.

[0107] In specific implementation, the threshold of resource utilization is configurable, the limit threshold is set to 60% of the resources, the maximum threshold is set to 80% of the resources, and the threshold settings of CPU utilization, memory utilization, and disk I / O utilization are all the same.

[0108] In specific implementation, if any one of the CPU usage, memory usage, and disk I / O usage exceeds the threshold, the system will be triggered to automatically adjust the load. Excessive usage will cause performance problems for the proxy server;

[0109] In the specific implementation, assume that there are two proxy servers serviceA and serviceB that need to process NetFlow traffic. The resource utilization rate of proxy server serviceA is high, and the resource utilization rate of proxy server serviceB is low. The router information associated with the proxy servers is shown in Table 2 below:

[0110]

[0111] Table 2

[0112] During the specific implementation, the system will monitor the CPU usage, memory usage and disk I / O usage. Here, we will take the CPU usage as an example to explain the system's automatic adjustment process of the proxy server load.

[0113] The system detects that the NetFlow traffic processed by serviceA has been increasing, and the resource status has changed to insufficient resources. The CPU usage has reached 80%, and the system starts to look for other proxy servers with low usage.

[0114] The system detects that the resource status of serviceB is moderate, the CPU usage is 60%, and the number of associated routers cannot be increased;

[0115] As shown in Table 3 below:

[0116]

[0117] Table 3

[0118] After the system continues to monitor for a period of time, the CPU usage of serviceA is still over 80%, while the CPU usage of serviceB drops below 60%, and the system starts to automatically adjust the load;

[0119] The system associates router 192.168.2.10 with serviceB. At this time, the CPU usage of serviceA drops below 80%, and the resource status changes to moderate resources. The system continues to monitor the proxy server.

[0120] As shown in Table 4:

[0121]

[0122] Table 4

[0123] S05. Utilize the pre-established mapping relationship between routers, port indexes, and collection directions to accurately filter NetFlow traffic, and then count and display the filtered NetFlow traffic.

[0124] The S05 includes:

[0125] S051, the system uses mapping relationships to accurately filter NetFlow traffic;

[0126] In specific implementation, if the mapping relationship of router 192.168.6.1 is 192.168.6.1:749-in and 192.168.6.1:750-in, it means that only the inbound traffic of port index 749 and 750 in 192.168.6.1 needs to be counted this time, and if the traffic from other ports and directions is received, it can be discarded.

[0127] S052. Count and display the filtered NetFlow traffic.

[0128] During specific implementation, the administrator can display the required NetFlow traffic statistics on the interface.

[0129] Based on the same inventive concept, the present invention also proposes a NetFlow traffic processing device. The implementation of the device can refer to the implementation of the above method, and the repeated parts will not be repeated. The term "module" used below can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0130] Figure 2 Schematic diagram of the structure of the NetFlow flow processing device of the present invention. Figure 2 As shown, the device comprises:

[0131] The mapping relationship writing module 110 writes the mapping relationship between the router, the collection port index and the collection direction into the configuration file, and pre-configures the default number of routers associated with the proxy server and the corresponding router IP;

[0132] The traffic compression module 120 compresses the collected NetFlow traffic using the LZ4 compression algorithm and stores it in different HDFS directories according to the router IP and date classification;

[0133] The flow reading module 130, the proxy server reads the corresponding NetFlow flow in the hdfs according to the associated router;

[0134] The load balancing module 140 monitors the resource usage of the proxy server in real time, and automatically adjusts the load through a preset threshold and HTB algorithm to ensure that the load of each proxy server remains balanced;

[0135] The traffic filtering module 150 uses the pre-established mapping relationship between routers, port indexes and collection directions to accurately filter the NetFlow traffic, and then counts and displays the filtered NetFlow traffic.

[0136] It should be noted that although several modules of the NetFlow traffic processing device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment.

[0137] Based on the above invention concept, Figure 3 As shown, the present invention further proposes a computer device 200, including a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220. When the processor 220 executes the computer program 230, the aforementioned NetFlow traffic processing method is implemented.

[0138] Based on the aforementioned inventive concept, the present invention further proposes a computer-readable storage medium, which stores a computer program for executing the aforementioned NetFlow traffic processing method.

[0139] The present invention provides a NetFlow traffic processing method and device, which processes NetFlow traffic in multiple levels and dimensions through grayscale publishing services, improves the processing efficiency of NetFlow traffic, balances the resource utilization rate of the proxy server, and provides operators with more accurate and timely traffic analysis services.

[0140] Although the spirit and principle of the present invention have been described with reference to several specific embodiments, it should be understood that the present invention is not limited to the disclosed specific embodiments, and the division of various aspects does not mean that the features in these aspects cannot be combined to benefit, and such division is only for the convenience of expression. The present invention is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the attached claims.

[0141] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0142] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0143] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0145] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0146] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0147] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0148] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

[0149] Regarding the limitation of the protection scope of the present invention, those skilled in the art should understand that, based on the technical solution of the present invention, various modifications or variations that can be made by those skilled in the art without creative work are still within the protection scope of the present invention.

Claims

1. A NetFlow traffic processing method, characterized in that: The method includes: S01, write the mapping relationship between routers, collection port indexes and collection directions into the configuration file, and pre-configure the default number of routers associated with the proxy server and the corresponding router IP; S02. Use the LZ4 compression algorithm to compress the collected NetFlow traffic, and store it in different HDFS directories according to the router IP and date. S03, the proxy server reads the corresponding NetFlow traffic in hdfs according to the associated router; S04. Monitor the resource usage of the proxy server in real time, and automatically adjust the load through the preset threshold and HTB algorithm to ensure that the load of each proxy server remains balanced; S05. Utilize the pre-established mapping relationship between routers, port indexes, and collection directions to accurately filter NetFlow traffic, and then count and display the filtered NetFlow traffic.

2. The NetFlow traffic processing method according to claim 1, characterized in that: The S01 includes: S011. Collect all router IPs, collection port indexes and collection directions, and write the mapping relationship into the configuration file; S012. Setting proxy server configuration information according to the information collected in S011, wherein the proxy server configuration information includes: server name, server IP, number of routers associated with the server, IP of routers associated with the server, resource status of the server, resource usage limit threshold and resource usage maximum threshold.

3. The NetFlow traffic processing method according to claim 2, characterized in that: The configuration file format is: router IP: collection port index-collection direction.

4. The NetFlow traffic processing method according to claim 2, characterized in that: The resource status of the server in S012 includes: sufficient resources, moderate resources, and insufficient resources.

5. The NetFlow traffic processing method according to claim 4, characterized in that: When the source usage rate is less than the specified threshold, the resource status is sufficient resources; when the resource usage rate is greater than the specified threshold and less than the maximum threshold, the resource status is moderate resources; when the resource status is greater than the maximum threshold, the resource status is insufficient resources.

6. The NetFlow traffic processing method according to claim 1, characterized in that: The S04 includes: S041. If the resource usage rate of the proxy server does not exceed the specified threshold, the resource status is set to sufficient resources, and the system does not need to actively adjust the router associated with the proxy server and continues real-time monitoring; S042. If the resource usage of the proxy server exceeds a specified threshold, set the resource status to moderate resources; When the resource usage of other proxy servers exceeds the maximum threshold, their associated routers will not be transferred to proxy servers with moderate resources. S043. If the resource usage of the proxy server exceeds the maximum threshold, the resource status is set to insufficient resources, and the system automatically searches for other proxy servers with low usage; S044. If the number of routers associated with the proxy server has been reduced and the resource usage of the proxy server is still increasing, the system will send an alarm message to the administrator; S045. When the resource usage rate of the proxy server is lower than a defined threshold, the resource state is set to sufficient resources. At this time, the number of routers associated with the proxy server can be increased to ensure that the load of each proxy server remains balanced.

7. The NetFlow traffic processing method according to claim 6, characterized in that: The S043 includes: S0431. If there are no other proxy servers with low usage, the system will adopt a queuing mechanism to temporarily store the NetFlow traffic to be processed by the proxy server; S0432. If there are other proxy servers with low utilization, the system automatically adjusts the load, reduces the number of routers associated with the proxy server, and transfers the associated routers to other proxy servers with lower resource utilization.

8. The NetFlow traffic processing method according to claim 6, characterized in that: In the S044, the system sends the warning information via short line, email or other means.

9. The NetFlow traffic processing method according to claim 1, characterized in that: The S05 includes: S051, the system uses mapping relationships to accurately filter NetFlow traffic; S052. Count and display the filtered NetFlow traffic.

10. A NetFlow traffic processing device, characterized in that: The device includes: The mapping relationship writing module writes the mapping relationship between routers, collection port indexes and collection directions into the configuration file, and pre-configures the default number of routers associated with the proxy server and the corresponding router IPs; The traffic compression module uses the LZ4 compression algorithm to compress the collected NetFlow traffic and stores it in different HDFS directories according to the router IP and date. Traffic reading module, the proxy server reads the corresponding NetFlow traffic in hdfs according to the associated router; The load balancing module monitors the resource usage of the proxy server in real time, and automatically adjusts the load through the preset threshold and HTB algorithm to ensure that the load of each proxy server remains balanced; The traffic filtering module uses the pre-established mapping relationship between routers, port indexes, and collection directions to accurately filter NetFlow traffic, and then counts and displays the filtered NetFlow traffic.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the method according to any one of claims 1 to 9.