Topology sniffing based inter-process network traffic trace control method and system

Abstract

The application provides a process inter-network traffic tracking control method and system based on topology sniffing. The application adopts a traffic packet monitoring mode to track the traffic of different processes on different machines. The process inter-network topology in a distributed environment is outlined. Because the network technology at the bottom layer of the operating system is adopted, the application is independent of the language or framework of service development and the service life cycle management. The application also considers the challenges brought by the container technology, and through monitoring the port forwarding information specific to the container, the traffic topology of the processes in the container is also included in the observation range of the method. The application can quickly generate topology information, and is independent of the service deployment process, and can be closed at any time after the inter-service network traffic topology is outlined.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of flow control, in particular to a process inter-network flow tracking control method and system based on topology sniffing. BACKGROUND

[0002] In an unfamiliar deployment environment, although there may be deployment instructions, the inter-service dependency relationship of the deployed services is chaotic and unobservable for the operation and maintenance personnel, which is not conducive to service operation and maintenance management. The existing flow packet-based method can capture the flow information of a certain process, but is used as process flow statistics and other functions, and cannot intuitively display the source and destination of the flow packet. If related information needs to be investigated, the operation and maintenance personnel need to have considerable knowledge of the deployed services in the environment, which is often lacking in operation and maintenance personnel. The existing flow packet-based method often analyzes data on a single node machine. It cannot effectively handle the flow analysis of processes distributed among different machines in a cluster environment. The current popular distributed service development method urgently needs a method that can outline the process flow topology between distributed clusters. Using link tracking to outline the topology between processes is a common means of service governance in microservices. However, this technology is related to the development language used by the service, and is not a language-independent general method. Moreover, related code needs to be introduced into the service. This is invasive to program development, occupies necessary resources, and increases security risks. For the current container technology, there is a lack of corresponding technical means to track the flow topology information within the container.

[0003] Most current flow packet-based tools focus on the flow information statistics of a single process, such as total flow consumption, number of packets received per unit time, and other information. The source and destination of the flow packet are not tracked and analyzed; the focus is on the flow information on a single machine, and the flow topology information between different machines is not displayed; although the flow information on the machine is captured and outlined, the display information is single, and the user needs to have sufficient background knowledge.

[0004] The nat operation of the current virtual container technology on the network in the container makes it impossible to effectively outline the inter-process network traffic topology by traffic capture alone. The current distributed tracking technology can outline the inter-service traffic information, but it is not a general way and depends on the specific support of the SDK in different languages to outline the inter-service traffic information, but it needs to add related code when developing programming. It is invasive to the service, the code developed permanently resides in the service package, has security risks, can outline the inter-service traffic information, but the related logic of tracking consumes additional resources, and some need to deploy additional services. In addition, the related logic of tracking and the host service of listening have the same life cycle and are not independent of the host service, and cannot achieve the effect of "plug and play". SUMMARY

[0005] To overcome the problems in the prior art, the present application provides an inter-process network traffic tracking control method and system based on topology sniffing. The present application uses traffic packet listening to track the traffic of different processes on different machines. The inter-process network topology in a distributed environment is outlined. Because it uses the network technology at the bottom of the operating system, it is independent of the language or framework of service development and independent of the life cycle management of the service. This technology also considers the challenges brought by container technology, and by listening to the port forwarding information specific to the container, the traffic topology of the processes in the container is also included in the observation range of the method. The present application can quickly generate topology information, and is independent of the service deployment process and can be turned off at any time after outlining the inter-service network traffic topology.

[0006] The present application is realized by the following technical solutions:

[0007] An inter-process network traffic tracking control method based on topology sniffing, the method comprising the following steps:

[0008] Step S100, the collector acquires the access network connection state and its related information in the kernel, and filters and cleans the data acquired at regular intervals to store them in the database in the form of key-value pairs;

[0009] Step S200, the collector acquires the operating system process list information;

[0010] Step S300, the collector captures the traffic packet by adding a bypass to the data link layer;

[0011] Step S400, the collector acquires the network information of the operating system, and views the files under the operating system directory to acquire the local traffic packet record information;

[0012] Step S500, the collector acquires the IP information packet filtering system information;

[0013] Step S600, the collector acquires the connection record item information of the network stack traffic packet;

[0014] Step S700, the key-value pair recorded in step S300 is found in the database through format change to find the corresponding link relationship;

[0015] Step S800, for the network traffic passing through the network card, the network communication relationship between the two processes is obtained by analyzing the key-value pair in step S400 and the corresponding link relationship obtained in step S700, and for the network traffic passing through the virtual network, the network communication relationship between the two processes is obtained through step S500 and step S600;

[0016] Step S900, through the collector, the process list of each node is obtained, and the network communication corresponding relationship between the processes is obtained, forming a directed graph, which constitutes the network traffic topology relationship of the current distributed environment;

[0017] Step S1000, the collector continuously performs the traffic packet monitoring and collection work, and the collected traffic packet is parsed and corresponded to the sending process or receiving process according to the above steps, and after a period of collection, the traffic consumption quantity and sending content information of the process are tracked;

[0018] Step S1100, through the visualization page, the network traffic topology relationship between the processes is displayed, and the query operation of the traffic tracking information is provided.

[0019] Further, in step S200, the process number array of the process is obtained by using the method of calling the operating system console command at a fixed time, and then the process number array is traversed to obtain the detailed information of the process.

[0020] Further, in step S300, when a data packet arrives at the network interface, first, the copy of the data packet is obtained from the link layer driver, and then the data packet is sent to the filter, the filter matches the data packet one by one according to the defined filtering rule, and after the matching is successful, the data packet is put into the kernel buffer and is transmitted to the user buffer, and if the matching fails, the data packet is directly discarded.

[0021] Further, in step S500, the mapping information of the internal and external ports of the container is obtained by using the method of calling the operating system console command at a fixed time.

[0022] Further, in step S600, the link record item information is obtained by using the method of calling the operating system console command at a fixed time.

[0023] The application further provides a process inter-network traffic tracking control system based on topology sniffing, which comprises a process inter-network traffic collection module and a process inter-network data analysis and display module.

[0024] The process inter-network traffic collection module is a network traffic collector, an operating system level provides an interface for observing processes, determines exposed port numbers of the processes through port usage of the operating system, and obtains network traffic packets through a network card and analyzes the packets.

[0025] The process inter-network data analysis and display module analyzes network access relationships between different processes through data provided by the collector, outlines an overall process inter-network topology, and displays the topology through a visual page.

[0026] The application further relates to an electronic device, which comprises:

[0027] at least one processor; and

[0028] a memory connected with the at least one processor in communication; wherein

[0029] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method.

[0030] The application further relates to a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the method.

[0031] The application further relates to an electronic device, which comprises:

[0032] at least one processor; and

[0033] a memory connected with the at least one processor in communication; wherein

[0034] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method.

[0035] The application further relates to a non-transitory computer readable storage medium storing computer instructions for causing a computer to execute the method.

[0036] The technical scheme of the application can achieve the following beneficial technical effects:

[0037] (1) Distributed link tracking technology can outline the topology relationship of inter-process traffic in a distributed environment in a fine-grained manner. The application invades relevant tracking services into programs and accompanies the entire declaration cycle of services.

[0038] (2) Traditional process network analysis tools focus on the instantaneous state of a single network, such as whether the network is normal, instantaneous traffic packets, traffic packet statistical information in a certain period of time, and the like. However, the application is different from the mapping relationship of inter-process network traffic. By analyzing network traffic, process information and the like at the operating system level, the network mapping relationship between different processes can be quickly outlined.

[0039] (3) Virtual container technology is a technology that has just emerged in recent years. Containers usually perform network and port forwarding during runtime, which brings challenges to traditional traffic packet capture tools. The application includes the network traffic of containers in the monitoring range from the network level of the operating system, and fuses the traffic information outside the containers. While analyzing the traffic information outside the containers, the network traffic topology information of the containers can also be analyzed. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 a process inter-network traffic tracking control system structure diagram based on topology sniffing of the application;

[0041] Figure 2 a process inter-network traffic tracking control method principle diagram based on topology sniffing of the application. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions and advantages of the application clearer and more explicit, the application will be further described in detail below in combination with specific embodiments and with reference to the drawings. It should be understood that these descriptions are only exemplary and are not intended to limit the scope of the application. In addition, in the following description, the description of well-known structures and technologies is omitted to avoid unnecessary confusion of the concept of the application.

[0043] The application will be described in detail below in combination with the drawings and embodiments.

[0044] Term explanation:

[0045] 1. Five-tuple

[0046] A set composed of five quantities of source IP address, source port, destination IP address, destination port and transport layer protocol.

[0047] 2. Transport layer

[0048] The transport layer provides end-to-end services between host application processes, and the basic functions are as follows: segmentation and reassembly of data, addressing by port numbers, connection management, error control and flow control, and error correction functions. The transport layer provides the reliability of communication services to the session layer, avoiding errors such as message errors, loss, delay time disorder, repetition, disorder, etc.

[0049] 3. Port classification

[0050] a) Hardware port: CPU transmits data to peripherals through interface registers or special circuits, which are called ports. Among them, the port in the hardware field is also called interface, such as parallel port, serial port, etc.

[0051] b) Network port: In network technology, port (Port) has several meanings. The port of a hub, switch, or router refers to the interface that connects other network devices, such as an RJ-45 port, a serial port, etc. The port we refer to here is not a physical port, but specifically refers to the port in the TCP / IP protocol, which is a logical port.

[0052] c) Software port: buffer.

[0053] 4. Libpcap

[0054] Libpcap is the English abbreviation of Packet Capture Libray, which means packet capture function library. The C function interface provided by the library is used to capture packets passing through the specified network interface, which should be set to promiscuous mode. The famous software TCPDUMP is developed on the basis of Libpcap. The interface function provided by Libpcap implements and encapsulates the process related to packet interception. Libpcap provides a user-level network packet capture interface and fully considers the portability of the application. Libpcap can run on most Linux platforms. Its work is between the upper application program and the network interface.

[0055] 5. Distributed tracking

[0056] (1) Trace: a transaction supported by distributed microservices collaboration. A trace contains various service requests that provide services for the transaction.

[0057] (2) Span: Span is a workflow in a transaction. A span contains timestamp, log and label information. Spans contain parent-child relationships or master-slave (Followup) relationships.

[0058] (3) Span Context: Span Context is the key to support distributed tracing, which can be passed between the services of the call, the content of the context includes, such as: from one service to another service time, the ID of the trace, the ID of the Span and other information that needs to be passed from the upstream service to the downstream service.

[0059] 6、OpenTracing standard

[0060] Based on the concept proposed by Google, OpenTracing defines an open standard for distributed tracing. Span is the basic unit of distributed tracing, which represents a separate work unit in a distributed system. Each Span can contain references to other Spans. Multiple Spans together constitute a Trace.

[0061] The first aspect of the present application provides a process inter-network traffic tracking control method based on topology sniffing, comprising the following steps:

[0062] Step S100, the collector acquires the access network connection state and its related information in the kernel by timing the execution of the operating system interface, and the collector filters and cleans the data obtained by timing and stores them in the database in the form of key-value pairs.

[0063] Specifically, the list of process port protocols, local ports, process numbers, program names, etc. is obtained by using the method of timing the operating system console command, and the key-value pairs are stored in the in-memory database.

[0064] Step S200, the collector acquires the operating system process list information by timing the execution of the operating system interface, such as obtaining the process number array of the process by timing the operating system console command. Then traverse the process number array to obtain the detailed information of the process, such as obtaining the inode information corresponding to the process by the operating system console command. By mapping the process number and the inode information of the process, the collector filters and cleans the mapped key-value pairs and stores them in the database, such as storing the key-value pairs in the in-memory database redis.

[0065] Step S300, the collector captures the traffic packet by adding a bypass in the data link layer. When a data packet arrives at the network interface, first get a copy of the data packet from the link layer driver, and then send the data packet to the BPF filter. The BPF filter matches the data packet one by one according to the defined filtering rule, and puts it into the kernel buffer and passes it to the user buffer after successful matching, and discards it directly after unsuccessful matching. The collector parses the captured data packet, such as using the general traffic packet parsing tool libcap to get the five-tuple information of the traffic packet: source IP address, source port, destination IP address, destination port, and transport layer protocol. The collector filters, cleans and converts the parsed data into key-value pairs and stores them in the database, such as storing the key-value pairs in the in-memory database redis.

[0066] Step S400, the collector obtains the network information of the operating system in a timed manner, and views the files under the operating system directory to obtain the local traffic packet record information. Get the source IP address, source port, destination IP address, destination port, and inode information of the traffic packet. By mapping the obtained information, the collector filters, cleans and stores the mapped key-value pairs in the database, such as storing the key-value pairs in the in-memory database redis.

[0067] Step S500, the collector obtains the IP information packet filtering system information by executing the operating system interface in a timed manner, such as obtaining the mapping information of the internal and external ports of the container by calling the operating system console command in a timed manner. The collector filters, cleans and stores the data obtained in a timed manner in the database in the form of key-value pairs, such as storing the key-value pairs in the in-memory database redis, and the form of key-value pairs has multiple forms.

[0068] Step S600, the collector obtains the connection record item information of the traffic packet through the network stack by executing the operating system interface in a timed manner, such as obtaining the link record item information by calling the operating system console command in a timed manner. The collector filters, cleans and stores the data obtained in a timed manner in the database in the form of key-value pairs, such as storing the key-value pairs in the in-memory database redis.

[0069] Step S700, from the perspective of traffic packets, a communication process has a sender and a receiver. The source IP address and source port in the traffic packet sent by the sender will be processed as the destination IP address and destination port in the receiver. Therefore, the key-value pairs recorded in step S300 can actually find the corresponding link relationship in the database through format changes.

[0070] Step S800, for network traffic through the network card, by analyzing the key-value pairs in step S400 and the corresponding link relationship obtained in step S700, the corresponding relationship between a certain process pid and another process pid can be obtained, that is, the network communication relationship between the two processes. For network traffic through the virtual network, because of the forwarding through the virtual network card, through steps S500 and S600, the corresponding relationship between the port number of a certain process and the port number of another process can be obtained, that is, the network communication relationship between the two processes.

[0071] Step S900, through the collector, the process list of each node is obtained, and the network communication corresponding relationship between the processes is obtained, logically forming a directed graph, which constitutes the network traffic topology relationship of the current distributed environment.

[0072] Step S1000, the collector continuously performs the listening and collecting work of the traffic packet, and the collected traffic packet is parsed and corresponded to the sending process or the receiving process according to the above steps, so that through a period of collection, the traffic consumption quantity, sending content and other information of the process can be tracked.

[0073] Step S1100, through the visualization page, the network traffic topology relationship between the processes is displayed, and the query operation of the traffic tracking information and the like is provided.

[0074] The present application describes the network traffic topology of the process, which realizes the independence from the development language or framework by listening to the network, process and other information at the operating system level, and has universality. The traditional related method focuses on the network information of a single node, and the present application describes the network traffic condition under the distributed environment. The present application describes the challenge brought by the traditional network traffic topology to the container technology, and the container network is included in the listening range.

[0075] Specifically, the second aspect of the present application provides an inter-process network traffic tracking control system based on topology sniffing, as shown in Figure 1 The system of the present application comprises an inter-process network traffic collection module and an inter-process network data analysis and display module.

[0076] Firstly, the inter-process network traffic collection module is a network traffic collector. The operating system level provides interfaces for observing processes, such as determining the process number of a process through the process list of the operating system, determining the port number exposed by a process through the port usage of the operating system, obtaining network traffic packets through a network card and performing packet analysis, and the like. These operating system-provided interfaces provide conditions for inter-process network traffic collection. The collector utilizes programs to realize the automation of process observation on the operating system, records process network communication at a finer time granularity, and provides more detailed data to support the analysis of the inter-process network topology. As shown in Figure 1 The collector is deployed on a node that needs to collect the inter-process network topology, collects the operating system level information and inter-process network communication of the target node, and filters, cleans, aggregates and persistently stores the monitored network traffic packet information, operating system information and the like into a database.

[0077] Secondly, the inter-process network data analysis and display module analyzes the data provided by the collector to analyze the network access relationship between different processes, outlines the overall inter-process network topology, and displays the same through a visual page. Inter-process communication is intermittent, and the information obtained by monitoring the network has high timeliness and needs to be processed and analyzed in a timely manner. This module includes a database, inter-process topology analysis, DashboardUI and the like.

[0078] The present application also relates to an electronic device, which comprises:

[0079] at least one processor; and,

[0080] a memory in communication with the at least one processor; wherein,

[0081] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method.

[0082] The present application also relates to a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method.

[0083] In summary, the present application provides a specific embodiment of the present application is understood to be merely exemplary or explanatory of the principles of the present application, and does not constitute a limitation on the present application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the present application shall be included in the protection scope of the present application. In addition, the appended claims of the present application are intended to cover all variations and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

Claims

1. A method for inter-process network traffic tracing and control based on topology sniffing, characterized in that, The method includes the following steps: Step S100: The collector obtains the network connection status and related information in the kernel, and filters and cleans the data obtained periodically and stores it in the database in the form of key-value pairs. Step S200: The collector acquires the operating system process list information; In step S300, the collector captures traffic packets by adding a bypass at the data link layer; Step S400: The collector obtains the network information of the operating system and checks the files in the operating system directory to obtain the local traffic packet record information. Step S500: The collector acquires IP packet filtering system information; Step S600: The collector acquires the connection record information of the network stack traffic packets; Step S700: Find the corresponding connection relationship in the database by changing the format of the key-value pairs recorded in step S300; In step S800, for network traffic passing through the network card, the network communication relationship between the two processes is obtained by analyzing the key-value pairs in step S400 and the corresponding connection relationship obtained in step S700. For network traffic passing through the virtual network, the network communication relationship between the two processes is obtained through steps S500 and S600. Step S900: The collector obtains the process list of each node and the network communication correspondence between the processes, forming a directed graph. This directed graph constitutes the network traffic topology of the current distributed environment. In step S1000, the collector continuously monitors and collects traffic packets, and parses the collected traffic packets to match the sending or receiving process according to the above steps. After a period of collection, the traffic consumption and content information of the process are tracked. Step S1100: Display the network traffic topology relationship between processes through a visualization page, and provide query operations for traffic tracing information.

2. The inter-process network traffic tracing and control method based on topology sniffing according to claim 1, characterized in that, In step S200, the process ID array of the process is obtained by periodically calling the operating system console command, and then the process ID array is traversed to obtain detailed information about the process.

3. The inter-process network traffic tracing and control method based on topology sniffing according to claim 1, characterized in that, In step S300, when a data packet arrives at the network interface, a copy of the data packet is first obtained from the link layer driver, and then the data packet is sent to the filter. The filter matches the data packets one by one according to the defined filtering rules. If a match is successful, the data packet is placed in the kernel buffer and then passed to the user buffer. If a match fails, the data packet is discarded directly.

4. The inter-process network traffic tracing and control method based on topology sniffing according to claim 1, characterized in that, In step S500, the mapping information of the internal and external ports of the container is obtained by periodically calling the operating system console command.

5. The inter-process network traffic tracing and control method based on topology sniffing according to claim 1, characterized in that, In step S600, connection record information is obtained by periodically calling operating system console commands.

6. A topology-sniffing-based inter-process network traffic tracing control system based on the topology-sniffing-based inter-process network traffic tracing control method according to any one of claims 1-5, characterized in that, The system includes an inter-process network traffic acquisition module and an inter-process network data analysis and display module; The inter-process network traffic acquisition module is a network traffic collector. It provides an interface at the operating system level to observe processes, determines the port numbers exposed by processes by checking the port usage of the operating system, and obtains network traffic packets through the network card and parses the packets. The inter-process network data analysis and display module analyzes the data provided by the collector to identify the network access relationships between different processes, outlines the overall inter-process network topology, and displays it through a visualization page.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1 to 5.

Images