A process-based network traffic statistics method

Abstract

The present invention provides a process-based network traffic statistics method, comprising the following steps: a coroutine captures the front-end data packet from the packet queue, performs transport layer filtering, network layer filtering, and application layer filtering, calculates the effective data volume of the data packet, pushes the effective data volume value to a data channel, calculates the effective data volume value within the data channel, and continuously obtains the traffic value of the process. The method has the following advantages: 1. It is simple to use out of the box, using a compiled binary executable program, and can be used quickly and cheaply in Linux x86 and ARM environments; 2. Starting from the application layer, it filters traffic based on process and service names, thereby better aligning with business objectives, reducing interference factors, and reducing analysis difficulty. This method effectively reduces the difficulty of observation in scenarios such as port negotiation, where there are many ports and they change dynamically.

Description

Technical Field

[0001] The present invention belongs to the technical field of network traffic statistics, and in particular relates to a process-based network traffic statistics method. Background Art

[0002] Network traffic analysis is widely used in various scenarios. For example, in internet application operation and maintenance, bandwidth usage and traffic analysis can improve QoS. During software development and testing, streaming media traffic analysis can identify environmental or software issues. Tracing back traffic changes can significantly reduce the cost of problem location.

[0003] Current network traffic statistical analysis methods mainly focus on coarse-grained traffic analysis and are unable to effectively perform fine-grained traffic analysis. Therefore, when relying on traffic to locate problems, it is impossible to precisely locate the cause of the problem, which has significant limitations. Summary of the Invention

[0004] In view of the defects of the existing technology, the present invention provides a process-based network traffic statistics method, which can effectively solve the above problems.

[0005] The technical solution adopted in the present invention is as follows:

[0006] The present invention provides a process-based network traffic statistics method, comprising the following steps:

[0007] Step 1: Each process started by the network device continuously captures data packets and places the data packets into a data packet queue; the file system records the timestamp, data packet ID, receiving port number, process number, source IP address and transport layer protocol of each captured data packet;

[0008] Step 2: Determine the process ID of the network device for which traffic statistics need to be collected.

[0009] Step 3: The network traffic statistics module starts one or more coroutines; each of the coroutines runs in parallel; and each of the coroutines runs in the following manner:

[0010] Step 3.1, the coroutine captures the data packet at the front of the queue from the data packet queue;

[0011] Step 3.2, the coroutine uses the preset transport layer filtering rules to determine whether the captured data packet meets the transport layer filtering rules. If not, the data packet is discarded and the process returns to step 3.1; if it meets, step 3.3 is executed;

[0012] Step 3.3, the coroutine uses the preset network layer filtering rules to determine whether the data packet meets the network layer filtering rules. If not, the data packet is discarded and the process returns to step 3.1; if it meets, step 3.4 is executed;

[0013] Step 3.4, the coroutine uses the preset application layer filtering rules to determine whether the data packet meets the application layer filtering rules. If not, the data packet is discarded and the process returns to step 3.1. If it meets the rules, step 3.5 is executed.

[0014] Step 3.5, the coroutine calculates the value of the effective data volume of the data packet, and determines whether the value of the effective data volume exceeds the length of the data channel Channel. If it exceeds, the data packet is discarded and the process returns to step 3.1; if it does not exceed, step 3.6 is executed;

[0015] Step 3.6: The coroutine pushes the value of the effective data volume of the data packet to the data channel Channel; return to step 3.1;

[0016] Step 4: The data channel Channel has a fixed length and arranges the valid data values ​​pushed by each coroutine in order; if the data channel Channel is full, when a new valid data value is received, the valid data value at the head of the data channel Channel queue is removed and the new valid data value is pushed to the end of the data channel Channel queue;

[0017] In step 5, the traffic statistics tool calculates the value of the effective data volume in the data channel Channel in real time and continuously obtains the traffic value of the process.

[0018] Preferably, the transport layer filtering rule is: specifying a transport layer protocol, if the captured data packet complies with the specified transport layer protocol, it complies with the transport layer filtering rule; otherwise, it does not comply with the transport layer filtering rule.

[0019] Preferably, the specified transport layer protocol is TCP protocol or UDP protocol.

[0020] Preferably, the network layer filtering rule is: specifying a source IP address; if the source IP address of the captured data packet is the same as the specified source IP address, it complies with the network layer filtering rule; otherwise, it does not comply with the network layer filtering rule.

[0021] Preferably, the application layer filtering rule is: specifying a receiving port number; if the receiving port number of the captured data packet is the same as the specified receiving port number, it complies with the application layer filtering rule; otherwise, it does not comply with the application layer filtering rule.

[0022] Preferably, the value of the effective data volume of the data packet refers to the storage space occupied by the effective data of the data packet; wherein, the effective data of the data packet refers to the amount of net payload data after removing the header of the data packet.

[0023] Preferably, the method further includes: isolating and protecting the coroutine and traffic statistics tool for performing traffic statistics so that the memory resources consumed by them are limited.

[0024] Preferably, it also includes:

[0025] Step 6: Render the continuously obtained process flow values ​​into a table or graph.

[0026] The process-based network traffic statistics method provided by the present invention has the following advantages:

[0027] ① Easy out-of-the-box use: Utilizing a compiled binary executable, it can be quickly and cost-effectively deployed in Linux x86 and ARM environments. ② Starting from the application layer, traffic is filtered based on process and service names, thereby better aligning with business objectives, reducing interference factors, and simplifying analysis. This effectively reduces the difficulty of observation in scenarios such as port negotiation, where there are numerous and dynamically changing ports. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a process-based network traffic statistics method provided by the present invention;

[0029] Figure 2 This is the traffic data update strategy diagram provided by the present invention. DETAILED DESCRIPTION

[0030] In order to make the technical problems, technical solutions and beneficial effects solved by the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0031] This invention relates to the technical field of software network testing and monitoring, and more specifically to a real-time, process-based network traffic statistics method. This method can improve bandwidth usage and QoS during Internet application operation and maintenance, as well as conduct security backdoor traffic analysis. Furthermore, it can analyze streaming media traffic during software development and testing, significantly reducing the cost of problem location by tracing back traffic changes.

[0032] Combine Figure 1 and Figure 2 The present invention provides a process-based network traffic statistics method, comprising the following steps:

[0033] Step 1: Each process started by the network device continuously captures data packets and places the data packets into a data packet queue; the file system records the timestamp, data packet ID, receiving port number, process number, source IP address and transport layer protocol of each captured data packet;

[0034] Step 2: Determine the process ID of the network device for which traffic statistics need to be collected.

[0035] Step 3: The network traffic statistics module starts one or more coroutines; each of the coroutines runs in parallel; and each of the coroutines runs in the following manner:

[0036] Step 3.1, the coroutine captures the data packet at the front of the queue from the data packet queue;

[0037] Step 3.2, the coroutine uses the preset transport layer filtering rules to determine whether the captured data packet meets the transport layer filtering rules. If not, the data packet is discarded and the process returns to step 3.1; if it meets, step 3.3 is executed;

[0038] In this step, the transport layer filtering rule is: specifying a transport layer protocol, if the captured data packet complies with the specified transport layer protocol, then it complies with the transport layer filtering rule; otherwise, it does not comply with the transport layer filtering rule.

[0039] Step 3.3, the coroutine uses the preset network layer filtering rules to determine whether the data packet meets the network layer filtering rules. If not, the data packet is discarded and the process returns to step 3.1; if it meets, step 3.4 is executed;

[0040] In this step, the network layer filtering rule is: specify a source IP address; if the source IP address of the captured data packet is the same as the specified source IP address, it complies with the network layer filtering rule; otherwise, it does not comply with the network layer filtering rule.

[0041] Step 3.4, the coroutine uses the preset application layer filtering rules to determine whether the data packet meets the application layer filtering rules. If not, the data packet is discarded and the process returns to step 3.1. If it meets the rules, step 3.5 is executed.

[0042] In this step, the application layer filtering rule is: specifying a receiving port number; if the receiving port number of the captured data packet is the same as the specified receiving port number, it complies with the application layer filtering rule; otherwise, it does not comply with the application layer filtering rule.

[0043] Step 3.5, the coroutine calculates the value of the effective data volume of the data packet, and determines whether the value of the effective data volume exceeds the length of the data channel Channel. If it exceeds, the data packet is discarded and the process returns to step 3.1; if it does not exceed, step 3.6 is executed;

[0044] The value of the effective data volume of the data packet refers to the storage space occupied by the effective data of the data packet; and the effective data of the data packet refers to the amount of net payload data after removing the header of the data packet.

[0045] Step 3.6: The coroutine pushes the value of the effective data volume of the data packet to the data channel Channel; return to step 3.1;

[0046] Step 4: The data channel Channel has a fixed length and arranges the valid data values ​​pushed by each coroutine in order; if the data channel Channel is full, when a new valid data value is received, the valid data value at the head of the data channel Channel queue is removed and the new valid data value is pushed to the end of the data channel Channel queue;

[0047] In step 5, the traffic statistics tool calculates the value of the effective data volume in the data channel Channel in real time and continuously obtains the traffic value of the process.

[0048] Step 6: Render the continuously obtained process flow values ​​into a table or graph.

[0049] The present invention also includes isolating and protecting the coroutine and traffic statistics tool for performing traffic statistics so that the memory resources consumed by them are limited.

[0050] A specific embodiment is described below:

[0051] The first step is to specify the source and target IP and the process corresponding to the port. This process is the process that needs to be observed.

[0052] The correspondence between the source and target IP addresses, ports, and processes can be obtained through the net and fd tables of the proc file system and the application relationship between the inodes of the two.

[0053] The second step is to start one or a group of coroutines to continuously capture data packets. According to the filtering rules of the application layer, transport layer, and network layer, the qualified data packets are retained and included in the statistics. See here for details. Figure 2 For example, for SIP data of video surveillance, select UDP protocol, and all TCP packets will be discarded;

[0054] The third step is to accumulate the effective data payload size of the data packets that meet the current timestamp of the statistics. The accumulated data will also be adapted to the rate unit according to the traffic volume, and the statistical data will be pushed to a channel;

[0055] In step 4, while processing in step 2, CGroups (Control Groups) and channels isolate and protect consumed system resources. Because CGroups can limit the CPU and memory resources of a Linux process or process group, packets exceeding the channel length are discarded to avoid excessive memory usage when counting extremely large traffic flows.

[0056] Step 5: Based on the traffic volume pushed in step 3, continuously loop to obtain the data in the channel and render it into a table or other form.

[0057] In practical applications, the following points should be noted:

[0058] Step 1: Enter the process name, preferably the command line, to exclude services with the same process name but different parameters.

[0059] Step 2: Flexibly monitor the target through CLI or configuration file.

[0060] Step 3: Combining the Golang development language with the Prometheus technology stack can make traffic data persistent and form historical data. This data will be more valuable after being collected for a long time.

[0061] The present invention provides a set of monitoring methods based on process services and an overall solution that does not interfere with the use of business resources. It has the following characteristics:

[0062] 1. From the perspective of the interface behind the traffic exposed by Linux, currently only the data in / proc / net / tcp can be used to obtain overall operating system data. There is no intuitive interface at the process level. To solve this problem, the inode corresponding to the socket corresponding to / proc / [pid] / fd is matched with the inode in / proc / net / tcp, and traffic data is further obtained using Google's packet capture library gopacket.

[0063] 2. Solve the problem of excessive resource consumption when traffic is large:

[0064] Since it is traffic monitoring, it is absolutely impossible to preempt business resources. The existence of monitoring is zero-intrusion and low-perception. Therefore, the present invention makes the following design:

[0065] 1. The packet capture data queue is limited; packets exceeding the limit will be discarded. 2. CGROUP technology is used, which limits resources. 3. Golang's semantic-level coroutine technology does not use threads, reducing CPU resource overhead. 4. The range of network card devices used for packet capture is limited; not all network card devices are captured. For example, business data does not go through the LO loop, and only external network devices are captured. 5. Loopbacks and multicasts are filtered out at the IP level. 6. Further filtering is performed on the transport layer of the monitored object. 7. Only packets in the established and listening (ESTABLISHED and LISTEN) states are processed.

[0066] 3. Of course, there are many mature solutions for traffic analysis in the industry, such as traffic mirroring and proxy forwarding at the switch or gateway level. However, the present invention is different in that it does not require excessive hardware equipment and only relies on the packet capture library to achieve low-cost and efficient traffic analysis.

[0067] Fourth, UDP traffic analysis is difficult because UDP is a stateless protocol and inherently supports broadcast and multicast. Therefore, the proc file system cannot fully display the UDP connection information including the remote end. The remote IP address is often displayed as 00000000:0000, making it impossible to correlate traffic information with the socket information used in the file description of the process. This invention solves this problem.

[0068] 5. The proc information is relatively easy to process. The difficulty lies in matching the captured packet data with the proc information. Because the data flows in according to business needs, the packet capture rules only retain the packets whose destination address is the business IP, and then construct the IP address information corresponding to the proc.

[0069] The technical points involved in the present invention are: 1. Process-based traffic statistics; 2. Operating system resource isolation; 3. Compatibility with multiple operating systems.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] ①Easy to use out of the box, using compiled binary executable programs, you can quickly and cheaply use it in Linux x86 and ARM environments;

[0072] ② Starting from the application layer, traffic is filtered based on process and service names to better align with business objectives, reduce interference factors, and reduce analysis difficulty. This effectively reduces the difficulty of observation in scenarios such as port negotiation, where there are many ports and they change dynamically.

[0073] ③ Pay attention to the reasonable allocation of resource usage to ensure that the normal operation of the business is not affected. This will make monitoring a normal practice and help to perceive the status in advance.

[0074] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A process-based network traffic statistics method, characterized in that: The following steps are involved: Step 1: Each process started by the network device continuously captures data packets and places the data packets into a data packet queue; the file system records the timestamp, data packet ID, receiving port number, process number, source IP address and transport layer protocol of each captured data packet; Step 2: Determine the process ID of the network device for which traffic statistics need to be collected. Step 3: The network traffic statistics module starts one or more coroutines; each of the coroutines runs in parallel; and each of the coroutines runs in the following manner: Step 3.1, the coroutine captures the data packet at the front of the queue from the data packet queue; Step 3.2, the coroutine uses the preset transport layer filtering rules to determine whether the captured data packet meets the transport layer filtering rules. If not, the data packet is discarded and the process returns to step 3.1; if it meets, step 3.3 is executed; Step 3.3, the coroutine uses the preset network layer filtering rules to determine whether the data packet meets the network layer filtering rules. If not, the data packet is discarded and the process returns to step 3.1; if it meets, step 3.4 is executed; Step 3.4, the coroutine uses the preset application layer filtering rules to determine whether the data packet meets the application layer filtering rules. If not, the data packet is discarded and the process returns to step 3.

1. If it meets the rules, step 3.5 is executed. Step 3.5, the coroutine calculates the value of the effective data volume of the data packet, and determines whether the value of the effective data volume exceeds the length of the data channel Channel. If it exceeds, the data packet is discarded and the process returns to step 3.1; if it does not exceed, step 3.6 is executed; Step 3.6, the coroutine pushes the value of the effective data volume of the data packet to the data channel Channel; Return to step 3.1; Step 4: The data channel Channel has a fixed length and arranges the valid data values ​​pushed by each coroutine in order; if the data channel Channel is full, when a new valid data value is received, the valid data value at the head of the data channel Channel queue is removed and the new valid data value is pushed to the end of the data channel Channel queue; In step 5, the traffic statistics tool calculates the value of the effective data volume in the data channel Channel in real time to continuously obtain the traffic value of the process.

2. The process-based network traffic statistics method according to claim 1, characterized in that: The transport layer filtering rule is: specifying a transport layer protocol, if the captured data packet complies with the specified transport layer protocol, it complies with the transport layer filtering rule; otherwise, it does not comply with the transport layer filtering rule.

3. The process-based network traffic statistics method according to claim 2, characterized in that: The specified transport layer protocol is the TCP protocol or the UDP protocol.

4. The process-based network traffic statistics method according to claim 1, characterized in that: The network layer filtering rule is: specify a source IP address; if the source IP address of the captured data packet is the same as the specified source IP address, it complies with the network layer filtering rule; otherwise, it does not comply with the network layer filtering rule.

5. The process-based network traffic statistics method according to claim 1, characterized in that: The application layer filtering rule is: specifying a receiving port number; if the receiving port number of the captured data packet is the same as the specified receiving port number, it complies with the application layer filtering rule; otherwise, it does not comply with the application layer filtering rule.

6. The process-based network traffic statistics method according to claim 1, characterized in that: The value of the effective data volume of the data packet refers to the storage space occupied by the effective data of the data packet; wherein, the effective data of the data packet refers to the amount of the net load data after removing the packet header of the data packet.

7. The process-based network traffic statistics method according to claim 1, characterized in that: Also includes: Isolate and protect the coroutines and traffic statistics tools that perform traffic statistics, limiting their memory consumption.

8. The process-based network traffic statistics method according to claim 1, characterized in that: Also includes: Step 6: Render the continuously obtained process flow values ​​into a table or graph.

Images