Whole stack path tracking method and system based on network analysis and electronic equipment

Abstract

The invention relates to the field of network flow collection and analysis, in particular to a full-stack path tracking method and system based on network analysis and electronic device.The method comprises the steps that data packets are collected from key network nodes, and key indexes of the data packets are extracted through TCP analysis; constructing a session record which takes a composite key as a unique identifier and aggregates the key indexes; inputting identification information of a target session, and querying all related sessions of bidirectional traffic of the target session in the session record; dividing the queried sessions into a session set of a request path from the client to the server and a session set of a response path from the server to the client, and sorting the session sets of the request path and the response path by using a non-insertion association algorithm; and determining a final path according to a sorting result. According to the method, a communication path can be completely restored without implanting codes, and low-intrusion and high-reliability full-stack traffic tracking and visual analysis are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of network traffic collection and analysis, and in particular to a full-stack path tracking method and system based on network analysis and an electronic device. BACKGROUND

[0002] In the field of network traffic collection and analysis, the traditional technical solution mainly relies on data collection and monitoring of a single network device (such as a gateway, router or switch). This solution analyzes the data packets flowing through a specific network node to achieve partial insight into the network status and traffic. However, in actual business request processing, a complete user request or transaction processing often needs to span multiple heterogeneous systems, service nodes and different network domains. In this distributed, cross-network interaction scenario, it is difficult to completely and accurately restore and present the end-to-end execution path and life cycle of the request based on data collected from a single point device, and it is difficult to effectively implement the behavior concatenation and global tracking of the request between different systems and network zones.

[0003] To address the above challenges, a solution for traffic collection and link association through plug-ins or agents has appeared in the prior art. Such a solution typically requires embedding specific monitoring code or functional modules in application systems or network nodes to achieve more fine-grained traffic interception and context transfer. However, this plug-in, embedded implementation inevitably has some invasiveness to the application runtime environment, not only introducing additional performance overhead and resource consumption, but also increasing the complexity of system deployment and maintenance. Especially in the current increasingly popular virtualized network environment and cloud network platform architecture, network interactions between service components are becoming more frequent and complex, and network traffic often needs to pass through Overlay tunnels, virtual switches, container networks and other virtual network boundaries, making it more difficult to deploy observation means based on plug-ins, and the reliability and coverage of data collection are also reduced.

[0004] Therefore, based on the prior art solution, the following technical defects mainly exist: (1) Invasiveness: The collection method relying on integrated SDK or deployed agent has invasiveness to the application system or runtime environment, which may affect the application performance and increase the complexity of operation and maintenance.

[0005] (2) Incomplete tracking: Limited by single-point collection or local deployment, it is difficult to cover the complete data path from the client-initiated request, through the intermediate network and service nodes, to the final server response, resulting in missing link information.

[0006] (3) Difficulty in connecting: In complex distributed systems and heterogeneous network environments, there is a lack of efficient and unified mechanisms to accurately associate and connect the traffic and events generated by a request on all the nodes it passes through, making it difficult to build a complete panoramic view of the request.

[0007] In summary, existing network traffic analysis technologies have significant shortcomings in handling distributed request tracing scenarios across systems and networks, particularly in terms of non-intrusive deployment, full-link coverage, and end-to-end connectivity. A new technical solution is urgently needed to address the challenges of complex network traffic analysis in cloud environments. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a full-stack path tracing method, system and electronic device based on network analysis.

[0009] In a first aspect, the present invention provides a full-stack path tracing method based on network analysis, comprising the following steps: S1: Collect data packets from each key network node, extract key indicators of the data packets through TCP analysis, and construct a session record with a composite key as the unique identifier and aggregating the key indicators; S2: Input the identification information of the target session, and query all related sessions of the bidirectional traffic of the target session in the session record; S3: Divide the queried sessions into a set of sessions for request paths from the client to the server and a set of sessions for response paths from the server to the client. Use a non-insertionable association algorithm to sort the set of sessions for request paths and response paths respectively. Determine the final path based on the sorting results.

[0010] Preferably, in step S1, the composite key includes: service layer source IP address and source port, service layer destination IP address and destination port, tunnel layer source IP address and source MAC address, and tunnel layer destination IP address and destination MAC address; The key metrics include: client's first packet TCP sequence number, client's first packet TTL in the tunnel layer, client's first packet TTL in the service layer, client's first packet timestamp, server's first packet TCP sequence number, server's first packet TTL in the tunnel layer, server's first packet TTL in the service layer, and server's first packet timestamp.

[0011] Furthermore, in step S2, the query includes at least three levels of queries: First query: Based on the service layer source IP address and source port, as well as the service layer destination IP address and destination port of the target session, query the first packet IP identifier and the first packet TCP sequence number of the target session; Second query: Based on the first packet IP identifier and the first packet TCP sequence number, query the service layer source IP address and source port, as well as the service layer destination IP address and destination port of the relevant session; The third query: Based on the source IP address and source port of the service layer and the destination IP address and destination port of the service layer, query all related sessions of the bidirectional traffic of the target session.

[0012] Furthermore, in step S3, request path sessions are filtered out based on the condition that the timestamp of the first packet from the client is greater than 0; response path sessions are filtered out based on the condition that the timestamp of the first packet from the server is greater than 0.

[0013] Further, in step S3, the method for sorting using the non-insertionable association algorithm includes: If the service layer IPs of two sessions are exactly the same or opposite to each other, or if the tunnel layer IPs of two sessions are exactly the same, they are sorted first by the service layer TTL value from largest to smallest; if the service layer TTLs are the same, they are sorted by the tunnel layer TTL value from largest to smallest; if both the service layer TTL and the tunnel layer TTL are the same, they are sorted by the first packet timestamp value from smallest to largest. If two sessions do not meet the conditions of having identical or opposite service layer IPs and identical tunnel layer IPs, they are directly sorted by the timestamp value of the first packet from smallest to largest.

[0014] Furthermore, to address path discontinuities caused by missing data collection, a path completion algorithm is used to insert virtual paths at the beginning and end of paths or between breakpoints within paths; the path completion algorithm includes path beginning and end completion and path breakpoint completion. The path completion steps include: sequentially checking the start and end sessions of the request path and the response path; if the source or destination IP address of the service layer does not match the IP address of the path endpoint, then completing a virtual path connecting the endpoint and the service layer IP address at the corresponding endpoint. The path breakpoint completion step includes: sequentially checking the session sequence within the request path and response path; when the tunnel layer destination IP address of the previous session and the tunnel layer source IP address of the next session do not match, a virtual path with the tunnel layer destination IP address as the source and the tunnel layer source IP address as the destination is inserted between the two adjacent sessions.

[0015] Furthermore, if a path has multiple possibilities, the score of each adjacent session pair is calculated based on preset rules and accumulated to form the total path score. Finally, the path with the highest cumulative score is selected as the final path. If a path has only one possibility, it is directly determined as the final path.

[0016] Furthermore, it also includes the visualization of the final path, and the method steps include: Sessions with the same source IP address and destination IP address at the service layer and the same source IP address and destination IP address at the tunnel layer will be merged into one session; Grouping sessions based on the network segment of the tunnel layer IP address or the vendor of the tunnel layer MAC address, and grouping sessions with the same network segment or vendor into the same group; Combine the established request and response paths into a complete communication link, and create attributes for each element in the path for interface display.

[0017] In a second aspect, the present invention provides a full-stack path tracing system based on network analysis, for implementing a full-stack path tracing method based on network analysis as described in any one of the preceding claims, the system comprising: The database module is used to store data packets collected at each key network node and provide query services; The path tracing module is used to trace the business path of a session through sorting, completion, and scoring processes. The display module is used to display path tracing results and the graphical user interface.

[0018] In a third aspect, the present invention provides an electronic device including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a full-stack path tracing method based on network analysis as described in any of the preceding claims.

[0019] Compared with existing technologies, the advantages of this invention are as follows: By constructing session records with composite keys as unique identifiers, this invention can efficiently aggregate and query all bidirectional associated sessions of a target session. By dividing the request and response path sets and sorting them using a non-insertionable association algorithm, the complete communication path from the client to the server can be accurately reconstructed. This method does not require the implantation of code or proxies in the target environment, significantly reducing the invasiveness and deployment complexity of the system, effectively solving the problems of path breakage and reconnection difficulties, realizing reliable tracking and visualization analysis of full-stack traffic, and providing an efficient observation method for complex network environments. Attached Figure Description

[0020] Figure 1 This is a flowchart of the full-stack path tracing method based on network analysis in Example 1. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to specific embodiments. However, this should not be construed as limiting the scope of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0022] Unless otherwise specified, the terms "upper," "lower," "left," "right," "center," "inner," and "outer," etc., used in the description of specific embodiments of the present invention to indicate orientation or positional relationships, are based on the orientation or positional relationships shown in the accompanying drawings, or the orientation or positional relationship in which the product / equipment / device is usually placed during use. These terms are merely for the purpose of facilitating the description of the present invention or simplifying the description in specific embodiments, and for enabling those skilled in the art to quickly understand the solution, and do not indicate or imply that a particular device / component / element must have a specific orientation, or be constructed and operated in a specific positional relationship. Therefore, they should not be construed as limitations on the present invention.

[0023] Furthermore, the use of terms such as "horizontal," "vertical," "suspended," "parallel," and "coaxial" does not imply that the corresponding device / component / element must be absolutely horizontal, vertical, suspended, parallel, or coaxial. Slight tilt or deviation is permissible, as long as it does not affect the normal function of the relevant component. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," not that the structure must be perfectly horizontal; a slight tilt is acceptable. "Coaxial" means that two components are arranged as coaxially as possible, allowing them to move coaxially or approximately coaxially when their relative positions change. Alternatively, it can be simplified to mean that the corresponding device / component / element, when arranged in "horizontal," "vertical," "suspended," "parallel," or "coaxial" directions, can have an error / deviation of ±10% relative to the corresponding direction, more preferably within ±8%, more preferably within ±6%, more preferably within ±5%, and more preferably within ±4%. For example, the deviation in the "coaxial" direction is controlled within 0.2-1mm, preferably within 0.2-0.5mm. As long as the corresponding device / component / element is within the error / deviation range, it can still achieve its function in the solution of the present invention.

[0024] Furthermore, the use of terms such as "first," "second," and "third" in terminology is merely for distinguishing descriptions of identical or similar components and should not be interpreted as emphasizing or implying the relative importance of a particular component.

[0025] Furthermore, in the description of the embodiments of the present invention, "several", "more than", and "a number of" represent at least two. The number can be any number, such as two, three, four, five, six, seven, eight, or nine, and can even exceed nine.

[0026] Furthermore, in the description of the technical solution of this invention, unless otherwise explicitly specified / limited / restricted, the terms "set up," "install," "connect," "link," "provided with," "laid out," and "arranged" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to common connection methods in the art, such as welding, riveting, bolting, and threaded connections. Such connections can be mechanical, electrical, or otherwise; they can be direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components.

[0027] Example 1 Figure 1 Here is a flowchart of a full-stack path tracing method based on network analysis, including the following steps: S1: Collect data packets from each key network node, extract key indicators of the data packets through TCP analysis, and construct a session record with a composite key as the unique identifier and aggregating the key indicators; S2: Input the identification information of the target session, and query all related sessions of the bidirectional traffic of the target session in the session record; S3: Divide the queried sessions into a set of sessions for request paths from the client to the server and a set of sessions for response paths from the server to the client. Use a non-insertionable association algorithm to sort the set of sessions for request paths and response paths respectively. Determine the final path based on the sorting results.

[0028] In an optional implementation, in step S1, the composite key includes: service layer source IP address and source port, service layer destination IP address and destination port, tunnel layer source IP address and source MAC address, and tunnel layer destination IP address and destination MAC address; The key metrics include: client first packet TCP sequence number (client_first_pkt_tcpseq), client tunnel layer first packet TTL (client_pa_first_pkt_ipttl), client service layer first packet TTL (client_svc_first_pkt_ipttl), client first packet timestamp (client_first_pkt_ns), server first packet TCP sequence number (server_first_pkt_tcpseq), server tunnel layer first packet TTL (server_pa_first_pkt_ipttl), server service layer first packet TTL (server_svc_first_pkt_ipttl), and server first packet timestamp (server_first_pkt_ns).

[0029] In an optional implementation, step S2 includes at least three levels of querying (query principle: in tunnel transmission, the original service data packet is completely encapsulated as the payload, and its first packet TCP sequence number and first packet IP identifier remain unchanged in end-to-end transmission; only the protocol header information of the outer tunnel connection changes). First query: Based on the source IP address and source port of the target session's service layer, as well as the destination IP address and destination port of the service layer, query the IP identifier and TCP sequence number of the first packet of the target session (e.g., enter "ip_addr_client=10.69.5.37 and ip_addr_server=10.100.0.37 and server_port = 37777 and client_port = 42476 and client_first_pkt_ns>0"). Second query: Based on the first packet IP identifier and the first packet TCP sequence number, query the service layer source IP address and source port, as well as the service layer destination IP address and destination port of the relevant session (for example: input "first_pkt_ipid = 43530and first_pkt_tcpseq = 1387548193"). The third query: Based on the source IP address and source port of the service layer and the destination IP address and destination port of the service layer, query all related sessions of the bidirectional traffic of the target session (for example: input "ip_addr_client=10.69.5.37and ip_addr_server=10.100.0.37 and server_port = 37777 and client_port =42476").

[0030] Because the path from the client to the server is different from the path from the server to the client (the path from the server to the client may not return according to the network node from the client to the server), it is necessary to perform separate statistics for the client-to-server and server-to-client paths when conducting session statistics.

[0031] In an optional implementation, in step S3, request path sessions are filtered out based on the condition that the timestamp of the first packet from the client is greater than 0; response path sessions are filtered out based on the condition that the timestamp of the first packet from the server is greater than 0.

[0032] In an optional implementation, step S3, the method for sorting using the non-insertionable association algorithm, includes: If the service layer IPs of two sessions are exactly the same or opposite to each other, or if the tunnel layer IPs of two sessions are exactly the same, they are sorted first by the service layer TTL value from largest to smallest; if the service layer TTLs are the same, they are sorted by the tunnel layer TTL value from largest to smallest; if both the service layer TTL and the tunnel layer TTL are the same, they are sorted by the first packet timestamp value from smallest to largest. If two sessions do not meet the conditions of having identical or opposite service layer IPs and identical tunnel layer IPs, they are directly sorted by the timestamp value of the first packet from smallest to largest.

[0033] In an optional implementation, to address path discontinuity caused by missing data collection, a path completion algorithm is used to insert virtual paths at the beginning and end of paths or between breakpoints within paths; the path completion algorithm includes path beginning and end completion and path breakpoint completion. The path completion steps include: sequentially checking the start and end sessions of the request path and the response path; if the source or destination IP address of the service layer does not match the IP address of the path endpoint, then completing a virtual path connecting the endpoint and the service layer IP address at the corresponding endpoint. The path breakpoint completion step includes: sequentially checking the session sequence within the request path and response path; when the tunnel layer destination IP address of the previous session and the tunnel layer source IP address of the next session do not match, a virtual path with the tunnel layer destination IP address as the source and the tunnel layer source IP address as the destination is inserted between the two adjacent sessions.

[0034] In an optional implementation, if a path has multiple possibilities, the score of each adjacent session pair is calculated based on a preset rule and accumulated to form the total path score. Finally, the path with the highest accumulated score is selected as the final path. If a path has only one possibility, it is directly determined as the final path.

[0035] The preset rule is as follows: 50 points are awarded when the tunnel layer destination IP address of the current session is the same as the tunnel layer source IP address of the next session. 40 points are awarded when the tunnel layer source IP address of the current session is the same as the tunnel layer source IP address of the next session. 30 points are awarded when the tunnel layer destination IP address of the previous session is in the same network segment as the tunnel layer source IP address of the next session, or when the tunnel layer destination IP address of the previous session is the same as the tunnel layer destination IP address of the next session. 10 points are awarded if the tunnel layer destination MAC address of the previous session and the tunnel layer destination MAC address of the next session belong to the same vendor, or if the service layer destination IP address of the previous session and the service layer destination IP address of the next session are the same, and the service layer source IP address of the previous session and the service layer source IP address of the next session are the same.

[0036] In an optional implementation, the method also includes drawing a visualization chart of the final path, with the following steps: Sessions with the same service layer source IP address and destination IP address, as well as tunnel layer source IP address and destination IP address, are merged into one session (first sorted using a non-insertionable association algorithm, then the tunnel layer source IP address of the first session and the tunnel layer destination IP address of the last session in the sorted result are concatenated as the merged result). Grouping sessions based on the network segment of the tunnel layer IP address or the vendor of the tunnel layer MAC address, and grouping sessions with the same network segment or vendor into the same group; The established request and response paths are combined into a complete communication link, and attributes are created for each element in the path for interface display (the attributes include corresponding display parameters such as ID, name, IP, MAC, and latency).

[0037] Based on the same inventive concept, this invention provides a full-stack path tracing system based on network analysis, used to implement a full-stack path tracing method based on network analysis as described in any of the above claims. The system includes: The database module is used to store data packets collected at each key network node and provide query services; The path tracing module is used to trace the business path of a session through sorting, completion, and scoring processes. The display module is used to display path tracing results and the graphical user interface.

[0038] Based on the same inventive concept, the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, it implements a full-stack path tracing method based on network analysis as described in any of the above.

[0039] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A full-stack path tracing method based on network analysis, characterized by the following steps: include: S1: Collect data packets from each key network node, extract key indicators of the data packets through TCP analysis, and construct a session record with a composite key as the unique identifier and aggregating the key indicators; S2: Input the identification information of the target session, and query all related sessions of the bidirectional traffic of the target session in the session record; S3: Divide the queried sessions into a set of sessions for request paths from the client to the server and a set of sessions for response paths from the server to the client. Use a non-insertionable association algorithm to sort the set of sessions for request paths and response paths respectively. Determine the final path based on the sorting results.

2. The full-stack path tracing method based on network analysis according to claim 1, characterized in that, In step S1, the composite key includes: service layer source IP address and source port, service layer destination IP address and destination port, tunnel layer source IP address and source MAC address, and tunnel layer destination IP address and destination MAC address; The key metrics include: client's first packet TCP sequence number, client's first packet TTL in the tunnel layer, client's first packet TTL in the service layer, client's first packet timestamp, server's first packet TCP sequence number, server's first packet TTL in the tunnel layer, server's first packet TTL in the service layer, and server's first packet timestamp.

3. The full-stack path tracing method based on network analysis according to claim 2, characterized in that, In step S2, the query includes at least three levels of queries: First query: Based on the service layer source IP address and source port, as well as the service layer destination IP address and destination port of the target session, query the first packet IP identifier and the first packet TCP sequence number of the target session; Second query: Based on the first packet IP identifier and the first packet TCP sequence number, query the service layer source IP address and source port, as well as the service layer destination IP address and destination port of the relevant session; The third query: Based on the source IP address and source port of the service layer and the destination IP address and destination port of the service layer, query all related sessions of the bidirectional traffic of the target session.

4. The full-stack path tracing method based on network analysis according to claim 2, characterized in that, In step S3, request path sessions are filtered out based on the condition that the timestamp of the first packet from the client is greater than 0; response path sessions are filtered out based on the condition that the timestamp of the first packet from the server is greater than 0.

5. The full-stack path tracing method based on network analysis according to claim 2, characterized in that, In step S3, the method of sorting using the non-insertionable association algorithm includes: If the service layer IPs of two sessions are exactly the same or opposite to each other, or if the tunnel layer IPs of two sessions are exactly the same, they are sorted first by the service layer TTL value from largest to smallest; if the service layer TTLs are the same, they are sorted by the tunnel layer TTL value from largest to smallest; if both the service layer TTL and the tunnel layer TTL are the same, they are sorted by the first packet timestamp value from smallest to largest. If two sessions do not meet the conditions of having identical or opposite service layer IPs and identical tunnel layer IPs, they are directly sorted by the timestamp value of the first packet from smallest to largest.

6. The full-stack path tracing method based on network analysis according to claim 2, characterized in that, To address path discontinuities caused by missing data collection, a path completion algorithm is used to insert virtual paths at the beginning and end of paths or between breakpoints within paths; the path completion algorithm includes path beginning and end completion and path breakpoint completion. The path completion steps include: sequentially checking the start and end sessions of the request path and the response path; if the source or destination IP address of the service layer does not match the IP address of the path endpoint, then completing a virtual path connecting the endpoint and the service layer IP address at the corresponding endpoint. The path breakpoint completion step includes: sequentially checking the session sequence within the request path and response path; when the tunnel layer destination IP address of the previous session and the tunnel layer source IP address of the next session do not match, a virtual path with the tunnel layer destination IP address as the source and the tunnel layer source IP address as the destination is inserted between the two adjacent sessions.

7. The full-stack path tracing method based on network analysis according to claim 2, characterized in that, If a path has multiple possibilities, the score of each adjacent session pair is calculated based on preset rules and accumulated to form the total path score. Finally, the path with the highest cumulative score is selected as the final path. If a path has only one possibility, it is directly determined as the final path.

8. The full-stack path tracing method based on network analysis according to claim 2, characterized in that, It also includes drawing a visualization of the final path, and the method steps include: Sessions with the same source IP address and destination IP address at the service layer and the same source IP address and destination IP address at the tunnel layer will be merged into one session; Grouping sessions based on the network segment of the tunnel layer IP address or the vendor of the tunnel layer MAC address, and grouping sessions with the same network segment or vendor into the same group; Combine the established request and response paths into a complete communication link, and create attributes for each element in the path for interface display.

9. A full-stack path tracing system based on network analysis, characterized in that, For implementing a full-stack path tracing method based on network analysis as described in any one of claims 1-8, the system comprises: The database module is used to store data packets collected at each key network node and provide query services; The path tracing module is used to trace the business path of a session through sorting, completion, and scoring processes. The display module is used to display path tracing results and the graphical user interface.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements as described in claim 1.

8. A full-stack path tracing method based on network analysis as described in any one of the following.

Images