Link tracking method and apparatus, electronic device, and medium

Abstract

The present disclosure discloses a link tracking method and device, electronic equipment and storage medium. The link tracking method executed by an application node can include: using a proxy module integrated in a transmission control protocol (TCP) module in the kernel of the application node, inserting first tracking data in a TCP packet transmitted and received by the TCP module; generating and caching second tracking data according to the first tracking data; wherein the second tracking data is used at least for link tracking of a service call corresponding to the TCP packet; and when a state of the second tracking data satisfies a first condition, sending the second tracking data to a collection module.

Description

Technical Field

[0001] This disclosure relates to the field of network technology, and in particular to a link tracing method and apparatus, electronic device and storage medium. Background Technology

[0002] With the development of cloud computing technology, more and more systems are being deployed using complex and large-scale distributed architectures. These systems are often composed of numerous software modules, which may use different programming languages ​​and are deployed as microservices in numerous cloud servers or containers.

[0003] If microservices are used to respond to service requests, multiple application nodes will be involved, and tracing will be required. In this way, when an anomaly is detected, the cloud server and / or container that caused the anomaly can be located based on the tracing data.

[0004] There are two ways to perform link tracing in related technologies:

[0005] The first method relies on log data and business data from cloud servers and containers. By analyzing the correlation between log data and business data, the location of anomalies can be determined. However, this method suffers from the problems of a large amount of data to be processed and significant delays.

[0006] The second approach, replacing the first, involves embedding tracking points within the applications involved in the microservices to perform tracing. However, this method requires authorization from the application developers, and if multiple applications are involved, code modifications are needed for each. Therefore, while this approach can reduce latency in anomaly localization based on tracing data, it is technically challenging to implement. Summary of the Invention

[0007] This disclosure provides a link tracing method, apparatus, electronic device, and storage medium to simplify link tracing while reducing code modifications to applications.

[0008] The first aspect of this disclosure provides a link tracing method, executed by an application node, comprising:

[0009] Using the proxy module of the Transmission Control Protocol (TCP) module integrated in the kernel of the application node, first tracking data is inserted into the TCP packets sent and received by the TCP module;

[0010] Based on the first tracking data, second tracking data is generated and cached; wherein, the second tracking data is used at least for link tracing of the service call corresponding to the TCP packet;

[0011] When the status of the second tracking data meets the first condition, the second tracking data is sent to the collection module.

[0012] A second aspect of this disclosure provides a link tracing method, executed by an electronic device including an analysis module, comprising:

[0013] After receiving the second tracking data sent by the proxy module integrated in the application node kernel, the integrity of the second tracking data is verified.

[0014] The second trace data that passes the integrity check is cached and sent to the analysis module. The second trace data cached by the collection module is used at least for link tracing of the service call corresponding to the TCP packet.

[0015] A third aspect of this disclosure provides a link tracing method, executed by an electronic device including an analysis module, comprising:

[0016] Receive second tracking data sent by the collection module, wherein the second tracking data is generated by the agent module integrated into the application node kernel;

[0017] Based on the second tracking data, a topology diagram of the service call chain represented by the tracking ID is generated.

[0018] A fourth aspect of this disclosure provides a link tracking device, comprising:

[0019] The insertion module is configured to use a proxy module of the Transmission Control Protocol (TCP) module integrated into the kernel of the application node to insert first tracking data into TCP packets sent and received by the TCP module.

[0020] The first generation module is configured to generate and cache second tracking data based on the first tracking data; wherein the second tracking data is used at least for link tracing of the service call corresponding to the TCP packet;

[0021] The first sending module is configured to send the second tracking data to the collection module when the status of the second tracking data meets the first condition.

[0022] A fifth aspect of this disclosure provides an electronic device, the electronic device comprising:

[0023] Memory;

[0024] A processor, connected to the memory, is configured to implement the link tracing method provided by any one of the technical solutions in the first to third aspects by executing computer-executable instructions stored in the memory.

[0025] A sixth aspect of this disclosure provides a computer storage medium storing computer-executable instructions; when executed by a processor, the computer-executable instructions can implement the link tracing method provided by any one of the technical solutions in the first to third aspects.

[0026] The technical solution provided in this disclosure integrates the proxy module within the kernel of the server operating system. Therefore, the proxy module is no longer integrated into the application that needs monitoring. When an application node has multiple applications installed, this proxy module integrated into the kernel can be used to track TCP packets sent and received by different applications. This tracking involves inserting first trace data into the TCP packet, and simultaneously generating second trace data. The first trace data inserted into the TCP packet can be used by other application nodes to generate second trace data. After the second trace data from each node is submitted to the analysis module, it can be used for precise and easy tracking of the service calls involved in the TCP packet. Attached Figure Description

[0027] Figure 1 A flowchart illustrating a link tracing method provided in an embodiment of this disclosure;

[0028] Figure 2 A schematic diagram of a tracking ID provided for an embodiment of this disclosure;

[0029] Figure 3 This is a schematic diagram of a TCP packet without an inserted tracking portion, provided as an embodiment of this disclosure.

[0030] Figure 4 This is a schematic diagram illustrating the insertion of a tracking portion of a TCP packet, as provided in an embodiment of this disclosure.

[0031] Figure 5 A flowchart illustrating a link tracing method provided in an embodiment of this disclosure;

[0032] Figure 6 A flowchart illustrating a link tracing method provided in an embodiment of this disclosure;

[0033] Figure 7 This is a schematic diagram illustrating a link tracing operation as provided in an embodiment of the present disclosure;

[0034] Figure 8 A flowchart illustrating a link tracing method provided in an embodiment of this disclosure;

[0035] Figure 9 A flowchart illustrating a link tracing method provided in an embodiment of this disclosure;

[0036] Figure 10 A flowchart illustrating a link tracing method provided in an embodiment of this disclosure;

[0037] Figure 11 A flowchart illustrating a link tracing method provided in an embodiment of this disclosure;

[0038] Figure 12 A flowchart illustrating a link tracing method provided in an embodiment of this disclosure;

[0039] Figure 13 This is a schematic diagram of the structure of a link tracking device provided in an embodiment of the present disclosure;

[0040] Figure 14 This is a schematic diagram of the structure of a link tracking device provided in an embodiment of the present disclosure;

[0041] Figure 15 This is a schematic diagram of the structure of a link tracking device provided in an embodiment of this disclosure;

[0042] Figure 16 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0043] To gain a more detailed understanding of the features and technical content of this disclosure, the implementation of this disclosure will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit this disclosure.

[0044] like Figure 1 As shown, this disclosure provides a link tracing method, executed by an application node, including:

[0045] S110: Using the proxy module integrated into the kernel of the application node, insert first trace data into the TCP packets sent and received by the TCP module;

[0046] S120: Generate and cache second tracking data based on the first tracking data; wherein, the second tracking data is used at least for link tracking of the service call corresponding to the TCP packet;

[0047] S130: When the status of the second tracking data meets the first condition, the second tracking data is sent to the collection module.

[0048] The service call trace can be performed using a distributed tracing tool. This service call can be a distributed call within a microservice.

[0049] The application node can use a Linux system or a Linux-derived system, and has a kernel. One application node can correspond to one physical machine, host, or virtual machine. One agent module is deployed within the operating system kernel of each application node.

[0050] The kernel has a TCP module that is specifically designed to handle the sending and receiving of TCP packets for the application node.

[0051] In this embodiment, the proxy module is integrated into the kernel of the server operating system, thus eliminating the need to integrate it into the application being monitored. When an application node has multiple applications installed, this proxy module integrated into the kernel can be used to track TCP packets sent and received by different applications. This tracking is specifically: inserting the first tracking data into the TCP packet.

[0052] In this embodiment of the disclosure, the first tracking data may include at least: a global ID and a span ID of the service call, i.e., the tracking ID.

[0053] Figure 2 The diagram shows the composition of a tracking ID.

[0054] The tracking ID may include one or more of the following components:

[0055] The first part consists of the timestamp used to generate the tracking ID;

[0056] The second part consists of the device ID that generated the tracking ID;

[0057] The third part, the serial number, is determined based on the tracking IDs currently generated by the device that produced this tracking ID.

[0058] exist Figure 2 A tracking ID consists of 64 bits; these 64 bits may include: 41 bits carrying a timestamp, which is the timestamp when the tracking ID was generated; 10 bits carrying the device ID; and 12 bits being a serial number.

[0059] exist Figure 2 The highest bit of the tracking ID is a reserved bit.

[0060] Of course, the above is just one way to generate a tracking ID.

[0061] The node ID and tracking ID can be generated in the same or similar ways.

[0062] Figure 3The image shows a TCP packet, which includes a TCP header and a TCP data portion.

[0063] The TCP header includes various fixed information such as source port, destination port, sequence number, acknowledgment number, and checksum.

[0064] The TCP data portion consists of the data content transmitted in a TCP message.

[0065] In this embodiment of the disclosure, the proxy module inserts first tracking data into the TCP packets sent and received by the TCP module.

[0066] like Figure 4 As shown, after the first trace data is inserted into the TCP packet, a trace portion is inserted between the TCP header (i.e., the TCP data portion). This trace portion includes at least the first trace data.

[0067] If this node is not the first node in the service call, then this node will have a parent node. In this case, the first trace data may include: trace ID, node ID, and parent node ID.

[0068] The second tracking data includes the first tracking data, and the amount of data included in the first tracking data is larger. For example, the second tracking data may include some status information of this node, thereby facilitating subsequent monitoring of the response quality of the microservices of this node.

[0069] The length of the first trace data inserted into the TCP packet can be preset. For example... Figure 4 As shown, the first trace data may include 32 bits. In specific implementations, the length of the first trace data inserted into the TCP packet is not limited to the aforementioned 32 bits.

[0070] In this embodiment, the collection module can be deployed on the same application node as the agent module, or on different application nodes. In a microservice distributed system, each application node can deploy one agent module, while only some application nodes may have the collection module deployed. If an application node does not have the collection module deployed, it sends the second tracing data collected by its own agent module to the application node with the collection module deployed.

[0071] In some embodiments, the agent module can report all the second tracking data it caches in real time. For example, once the agent module has new second tracking data cached, it sends the newly cached second tracking data to the collection module.

[0072] In some embodiments, in order to reduce the amount of second tracking data sent by the agent module to the collection module, the second tracking data in the cache is only reported when the first condition is met, thereby reducing the number of times the geographic module reports data to the collection module.

[0073] In some embodiments, the insertion of first tracking data into TCP packets sent and received by the TCP module using a proxy module integrated in the application node kernel includes at least one of the following:

[0074] The first trace data is inserted into the TCP packet containing the start command by using the proxy module of the Transmission Control Protocol (TCP) module integrated in the kernel of the application node.

[0075] The first tracking data is inserted into the TCP packet containing the sending instruction by using the proxy module of the Transmission Control Protocol (TCP) module integrated in the kernel of the application node.

[0076] The first tracking data is inserted into the TCP packet containing the receive instruction by using the proxy module of the Transmission Control Protocol (TCP) module integrated in the kernel of the application node.

[0077] The first trace data is inserted into a TCP packet containing a stop instruction by using a proxy module of the Transmission Control Protocol (TCP) module integrated into the kernel of the application node.

[0078] TCP messages come in many types. A single TCP interaction between a client and an application node involves a start (START) instruction to initiate data transmission, an end (END) instruction to terminate data transmission, a send (SEND) instruction to indicate data to be sent, and a receive (RCV) instruction to indicate data to be received.

[0079] In this embodiment of the disclosure, in order to reduce unnecessary insertion of the first tracking data, the first tracking data will only be inserted into the TCP packets carrying the above four instructions.

[0080] In some embodiments, sending the second tracking data to the collection module when the state of the second tracking data meets the first condition includes:

[0081] When the amount of the second tracking data reaches the first data amount threshold, the second tracking data is sent to the collection module;

[0082] And / or,

[0083] When the time interval between the last reported time of the second tracking data and the current time reaches the first interval period, the second tracking data is sent to the collection module.

[0084] If the data is reported according to the cached data volume, when the data volume cached by the agent node reaches the first data volume threshold, the tracking node reports a second tracking data to the collection module.

[0085] If the reporting is periodic, and the time interval between the current moment and the last report has reached one reporting cycle, the second tracking data that has not yet been reported will be sent to the collection module once.

[0086] In some embodiments, generating and caching the second tracking data based on the first tracking data includes:

[0087] Based on the service call status information of this node, generate second trace data that includes the first trace data and the service call status information;

[0088] Cache the second tracking data.

[0089] The service call status information includes, but is not limited to, the timestamp of the service call, for example, the accuracy of the timestamp can be down to the millisecond or microsecond level.

[0090] The proxy module locally caches the second trace data, while the first trace data is the data inserted into the TCP packet.

[0091] In some embodiments, caching the second tracking data includes:

[0092] The second tracking data is cached using a double-ended queue.

[0093] The agent module can use various queues to cache the second tracking data, such as a first-in-first-out queue, a last-in-first-out queue, and / or a double-ended queue. Since both the tail and the head of a double-ended queue are operable, using a double-ended queue to cache the second tracking data facilitates subsequent processing of the second tracking data in the double-ended queue by the agent module.

[0094] In some embodiments, sending the second tracking data to the collection module when the state of the second tracking data meets the first condition includes:

[0095] When the state of the second tracking data meets the first condition, a data sequence of the second tracking data is generated according to the protocol cache sequence partitioning protocol;

[0096] The compressed data sequence is sent to the collection module.

[0097] For example, the second tracking data is converted into a protobuf sequence according to the protocol buffer, and the protobuf sequence is sent to the collection module. The protobuf sequence has the advantages of fast serialization speed and small data volume after serialization.

[0098] In some embodiments, the method further includes:

[0099] The system receives sampling configuration information sent by the collection module, wherein the sampling configuration information includes at least: sampling frequency;

[0100] The method of inserting first tracking data into TCP packets sent and received by the TCP module using the proxy module integrated in the application node kernel includes:

[0101] The proxy module of the Transmission Control Protocol (TCP) module integrated in the kernel of the application node is used to intercept TCP packets sent and received by the TCP module according to the sampling frequency.

[0102] And insert the first trace data into the intercepted TCP packet.

[0103] For example, assuming a frequency of 1000, the proxy module intercepts one TCP packet out of 1000 TCP packets sent and received by the TCP module, and then inserts the first tracking data.

[0104] In this embodiment of the disclosure, the proxy module can dynamically adjust the TCP packet embedding frequency according to the sampling configuration information issued by the collection module.

[0105] For example, the first tracking data includes at least: tracking identifier ID and node ID.

[0106] As another example, the first tracking data also includes a parent node identifier.

[0107] In some embodiments, the second tracking data includes the first tracking data and further includes at least one of the following: the operation name of the operation involved in the TCP packet; node label; network protocol IP address of the local node host; timestamp; log events of the link; log information; process identifier; and the length of the TCP packet.

[0108] The timestamp here is the timestamp of the service call's response within this node.

[0109] The length of this TCP packet can be: the packet length excluding the first trace data inserted.

[0110] For example, the second tracking data may specifically include one or more of the contents shown in Table 1.

[0111]

[0112]

[0113] In some embodiments, the method further includes:

[0114] If the second tracking data includes sensitive data, the sensitive data is encrypted before the second tracking data is sent to the collection module.

[0115] For example, the sensitive data may include, but is not limited to, user's personal data. Sensitive data in the second tracking data is protected by encrypting it using symmetric or asymmetric encryption methods.

[0116] like Figure 5 As shown, this disclosure provides a link tracing method, executed by an electronic device including a collection module, comprising:

[0117] S210: After receiving the second tracking data sent by the proxy module integrated in the application node kernel, perform integrity verification on the second tracking data;

[0118] S220: Cache the second trace data that has passed the integrity check and send the second trace data that has passed the integrity check to the analysis module, wherein the second trace data cached by the collection module is used at least for link tracing of the service call corresponding to the TCP packet.

[0119] The execution device in this embodiment of the disclosure is an electronic device that includes a collection module.

[0120] The collection module receives the second tracking data sent by the agent module that integrates the application node kernel, performs a completeness check on the second tracking data, caches it after the check is completed, and sends it to the analysis module.

[0121] The analysis module can at least analyze the link of service calls corresponding to TCP packets.

[0122] The analysis module can be located on the same electronic device as the collection module, or it can be a module deployed on a different electronic device than the collection module.

[0123] Integrity verification ensures that the data received from the agent module has not been tampered with, thereby ensuring the correctness and accuracy of link tracing.

[0124] In some embodiments, the method further includes:

[0125] Discard the second tracking data that fails the integrity check.

[0126] In other words, the second tracking data that fails the integrity check is dirty data that will be discarded by the collection module, thereby reducing the impact of dirty data on the accuracy and precision of link tracking.

[0127] In some embodiments, the method further includes:

[0128] When the state of the cached second tracking data meets the second condition, the second tracking data cached by the collection module is written into the storage module.

[0129] After the collection module caches a certain amount of second tracking data, it writes the second tracking data to the storage module. The storage module can persist the second tracking data. For example, after receiving the second tracking data, the storage module distributes the second tracking data to the database of multiple nodes.

[0130] If the second tracking data is persisted, it will be convenient to view it again later.

[0131] In some embodiments, writing the second tracking data cached by the collection module into the storage module when the state of the cached second tracking data reaches a second condition includes at least one of the following:

[0132] When the amount of the second tracking data reaches the second data amount threshold, the second tracking data is written to the storage module;

[0133] And / or,

[0134] When the time interval between the last reported time of the second tracking data and the current time reaches the second interval period, the second tracking data is written to the storage module.

[0135] For example, the second data volume threshold may be greater than the aforementioned first data volume threshold, and the second interval period may be equal to or greater than the aforementioned first interval period.

[0136] In summary, in this embodiment of the disclosure, the second tracking data cached by the collection module and written to the storage module are both data that have undergone integrity verification.

[0137] like Figure 6 As shown, this disclosure provides a link tracing method, executed by an electronic device including an analysis module, comprising:

[0138] S310: Receive second tracking data sent by the collection module, wherein the second tracking data is generated by the agent module integrated in the application node kernel;

[0139] S320: Based on the second tracking data, generate a topology diagram of the service call chain represented by the tracking ID.

[0140] In some embodiments, the method further includes:

[0141] S330: Based on the topology diagram, obtain the DAG of the service call link.

[0142] This disclosure embodiment is executed by an electronic device with an analysis module deployed.

[0143] The analysis module and the collection module can be deployed on the same electronic device (e.g., an application node) or on different electronic devices.

[0144] In summary, in this embodiment of the disclosure, the analysis module receives second tracking data sent by the collection module. This second tracking data is then generated by a proxy module integrated on the application node kernel.

[0145] After receiving the second trace data, the analysis module will use the trace ID to identify the topology of the two service calls.

[0146] The topology diagram here can be a topology diagram of the application nodes corresponding to the call chain of the service call.

[0147] By determining the topology, the call chain of service calls can be traced.

[0148] After obtaining the topology graph, the analysis module further converts it into a DAG.

[0149] In some embodiments, the method obtains a directed acyclic graph (DAG) of the service call links based on the topology graph, including:

[0150] When a tracking ID involves multiple topologies, the multiple topologies are aggregated into a directed acyclic graph (DAG).

[0151] If multiple topology graphs are generated based on the second tracking data of the same tracking ID, the multiple topology graphs are aggregated into one topology graph based on the dependencies and / or connections between nodes in the multiple topology graphs, thereby obtaining the DAG of the service call chain corresponding to a tracking ID.

[0152] In some embodiments, generating a topology map of the service call chain represented by a trace ID based on the second trace data includes:

[0153] Grouping the second tracking data according to the tracking ID, we obtain second tracking data with the same tracking ID;

[0154] Based on the node identifier and parent node identifier contained in the second tracking data of the same tracking ID, a topology map of the calling nodes involved in the link corresponding to the tracking ID is generated.

[0155] Starting from the second application node involved in the service call, the second trace data with the same trace ID will not only include the trace ID, but also the node ID (i.e., the current node ID) and the parent node ID. In this way, a topology graph can be constructed based on the current node ID and the parent node ID.

[0156] In some embodiments, the method further includes at least one of the following:

[0157] Based on the topology diagram, the call relationship information between different nodes is written into the tracking table;

[0158] Based on the topology graph, the dependency information between different nodes is written into the dependency table;

[0159] Based on the second tracking data, the node's indicator data is written into the indicator table;

[0160] Based on the second tracking data, the node's log data is written to the log table.

[0161] While performing link tracing, the call relationship information of service calls can be obtained from the topology diagram and written into the tracing table for easy subsequent querying.

[0162] Write the dependencies between nodes themselves and / or the dependencies between nodes that reflect the dependencies between business functions into the dependency table.

[0163] Different application nodes have different working performance, which can be reflected by indicator data. Therefore, the indicator data for the corresponding node can be written into the indicator table based on the indicator data contained in the second tracking data.

[0164] In another embodiment, the second tracking data may also include log data, which can be written to a log table.

[0165] refer to Figure 7 and Figure 8 As shown, the standard Opentracing links represented by Skywalking all adopt an annotation-based scheme. It relies on the application or middleware to explicitly mark a global ID for all services (i.e., the aforementioned tracing ID), thereby associating a series of requests. Compared to the black-box mode, this scheme does not rely on statistical inference, which improves the accuracy of service call dependencies. The Skywalking link implementation scheme consists of four parts: (1) Data collection module (agent): collects data through tracing dependency probe technology.

[0166] (2) Data transmission module (i.e., collection module): transmits data to the Skywalking Reveiver module via Kafka, Grpc, and HTTP.

[0167] (3) Data parsing and analysis module (also known as analysis module, (Flink jobs)): The OAP system performs data parsing and analysis.

[0168] (4) Data storage module (e.g., database (DB) or cache (Bookie): The backend interface supports multiple storage implementations, such as ES, MySQL, etc.

[0169] (5) User Interface (UI) module: Queries are performed using GraphQL, and then displayed using a front-end built with Vue.

[0170] This disclosure proposes a non-intrusive distributed link tracing method based on kernel-based instrumentation.

[0171] The overall architecture diagram designed in the embodiments of this disclosure is as follows: Figure 8 and Figure 9 As shown:

[0172] The TCP protocol implementation of the link scheme is divided into four parts, and the functions of each module are described below:

[0173] Agent: Installed on the host machine, it implements TCP layer data entry, trace data reporting, and dynamic adjustment of the sampling rate.

[0174] Collector: A horizontally scalable data receiving module that implements trace data format verification.

[0175] Flink Jobs: Uses a Flink cluster to perform aggregate computations on trace data. The aggregate results are stored in the database according to the link results, link metrics results, and dependency topology results.

[0176] The UI provides three views: 1. The Link View: View the overall link based on the service instance, including the trace map, the hierarchical relationship between spans, and relevant information for each span. 2. The Dependency Topology Graph: Displays the topological dependencies of each service instance. 3. The Performance Metrics View: Displays various performance metrics for each service instance.

[0177] The Agent module in this embodiment is integrated into the Linux kernel and performs data entry points in TCP request and response sequences.

[0178] like Figure 9The Agent performs event tracking for each TCP request and response. When a request enters the kernel's TCP receive module, the Agent intercepts it, focusing on the START / SEND / RCV / END instructions. Figure 9 illustrates three hosts or containers responding to service calls (Requests). Clearly, each host or container has an agent module deployed within the kernel.

[0179] Once this condition is met, the TCP header is obtained. The data format definition of the TCP header is as follows: Figure 3 As shown, the Agent then achieves the purpose of data tracking by adding no more than 40 bytes of tracking-related data to the options. This corresponds to a space of no more than 40 bytes for the options section. This embodiment of the disclosure designs a data format that achieves the effect of data tracking without excessively increasing the message size. The specific design is as follows... Figure 4 As shown.

[0180] Both the trace ID and the span ID can be generated using the Snowflake algorithm. The 64-bit ID generated by the Snowflake algorithm consists of 1 sign bit + 41-bit timestamp + 10-bit machine ID + 12-bit sequence. The ID structure is as follows: Figure 2 As shown.

[0181] After receiving the call request, the Agent will parse the trace portion of the TCP header and check the traceID and span ID.

[0182] If it does not exist, the trace ID and span ID are generated using the snowflake algorithm and inserted into the TCP header. Simultaneously, a data entry point is generated and reported to the Collector module, with the data format conforming to... Figure 8 Format definition. Before sending a TCP packet, the Agent will still insert the traceID, spanID, and parentSpanID into the packet header and pass them to the next node, while sending data to the Collector module according to the data format in Table 1.

[0183] In the trace data, the timestamp represents the current machine's timestamp, while the trace ID, span ID, and parent span ID are trace data in the TCP header.

[0184] The process ID is the process ID that the kernel-mode TCP module receives or sends to user space. The thread ID is the thread ID that the kernel-mode TCP module receives or sends to user space. The operationName is the TCP instruction operation, which only includes the four instructions: START, SEND, RCV, and END. The host refers to the IP address of the host. The msgSize refers to the size of the packet in this TCP transmission. The tags are a map data format used to store supplementary tag content. The logs are used to save log information when a link anomaly occurs, including the occurrence time and specific time description.

[0185] The agent compresses and serializes the trace data using protobuf serialization before sending it to the collector. Sensitive data in the logs is encrypted using the AES asymmetric encryption algorithm. A bidirectional queue is designed on the agent's sending end to buffer the trace data to be sent. By setting a time window and a size window, whichever is satisfied first triggers a data report. The agent compresses each batch of trace data using zlib's compression algorithm to reduce the amount of data transmitted over the network.

[0186] The sampling rate configuration in the Agent can be dynamically changed by receiving instructions from the Collector. For example, if the sampling rate is set to 1 / 1000, the Agent will implement TCP header tracking and report trace data to the Collector module in the next TCP request or response after this point. Subsequent TCP requests or responses with the same Process ID will not be tracked. When a new sampling rate is received, TCP header tracking and trace data reporting will be re-implemented, and the number of discarded packets will be recalculated.

[0187] Figure 10 This is a diagram of the architecture of the Collector module. The Collector module is designed as a stateless component with separate computation and storage, so it can theoretically achieve unlimited horizontal scaling, supporting independent deployment and deployment of the Collector through Kubernetes.

[0188] The Dispatcher consists of a TCP module that is responsible for receiving and sending messages externally, and an HTTP module that is used to manage the internal configuration of the service.

[0189] The TCP module is mainly used to receive tracking data reported by the Agent and send control commands for the sampling rate, and to complete the decompression of compressed data.

[0190] The HTTP module is primarily used for the management interface of the Collector cluster, including settings such as the retention period for tracking data, the number of replicas, and the sampling rate.

[0191] First, Ledger will check the format of the reported trace data and whether the required fields are complete, and will discard incomplete trace data.

[0192] For performance reasons, messages are distributed to Ledgers for caching. When the number of cached messages reaches a certain threshold, the data is persisted to the Bookie cluster via the BK Client. Each instance then has a scheduled task responsible for creating replicas of the cached data on the Ledger. The Sampling Controller is a configuration module used to adjust the sampling rate of the Agents. It receives modification requests from the frontend through the Dispatcher and then distributes them to all Agents via the TCP module.

[0193] Figure 11 This demonstrates the complete data timing relationship of the trace data reporting completed by the Collector module.

[0194] First, the TCP module in the Dispatcher starts a TCP listening port.

[0195] Then, after receiving the trace data reported by the Agent, it is decompressed and distributed to the Ledger module.

[0196] Ledger performs integrity verification on trace data. If verification fails, the data is discarded. If verification succeeds, the data is cached in the cache module. After a batch of data entries and a time window (either one needs to be satisfied) are completed, the batch data is written to Bookie. Then, the result is returned to Agent asynchronously.

[0197] Analysis modules (e.g., Flink Jobs),

[0198] To enable link and topology analysis of real-time tracking data, this embodiment employs the Flink real-time computing engine. Here, the Collector consumer is set as the Source of the Flink task, allowing trace data to continuously flow from the Collector to the Flink task.

[0199] DataStream <Iterable >traces=spans

[0200] ①.filter((FilterFunction )span->span.isClient()||span.isServer())

[0201] .name(FILTER_LOCAL_SPANS)

[0202] ②.keyBy((KeySelector<Span,String>)span->String

[0203] .format("%d:%d",span.getTraceIdHigh(),span.getTraceIdLow()))

[0204] ③.window(EventTimeSessionWindows.withGap(Time.minutes(3)))

[0205] ④.apply(new SpanToTraceWindowFunction()).name(SPANS_TO_TRACES)

[0206] ⑤.map(new AdjusterFunction<>()).name(DEDUPE_SPAN_IDS)

[0207] ⑥.map(new CountSpansAndLogLargeTraceIdFunction()).name(COUNT_SPANS);

[0208] ⑦DataStream <dependency>dependencies=traces .flatMap(newTraceToDependencies()).name(TRACE_TO_DEPENDENCIES).keyBy(key->key.getParent()+ key.getChild()).timeWindow(Time.minutes(30)).sum("callCount").name(PREAGGREGATE_DEPENDENCIE S);

[0209] filter((FilterFunction `span->span.isClient()||span.isServer())` only retains spans with the `client` or `server` tag.

[0210] keyBy((KeySelector<Span,String> `span->String.format("%d:%d",span.getTraceIdHigh(), span.getTraceIdLow())` - Groups span data based on a combination of traceIdHigh and traceIdLow.

[0211] `window(EventTimeSessionWindows.withGap(Time.minutes(3)))` - Creates a session window with a gap of 3 minutes for each trace. This means that for a certain trace, if no new span data arrives within 3 minutes, the window is considered to end, thus triggering subsequent aggregation operations.

[0212] The SpanToTraceWindowFunction is responsible for collecting node data from the session window. A node's span data includes a second trace of that node.

[0213] AdjusterFunction is responsible for removing duplicate spans from a trace.

[0214] The CountSpansAndLogLargeTraceIdFunction counts the distribution of the number of spans over time in 10-second increments.

[0215] `flatMap(new TraceToDependencies())` - Iterates through all spans of a trace, collecting the topological relationships represented by the trace. `keyBy(key->key.getParent()+key.getChild())` - Groups dependencies with the same parent-child relationship into a single group.

[0216] timeWindow(Time.minutes(30)).sum("callCount") - Creates a 30-minute time window for each dependency pair, and triggers a counting operation when the window ends.

[0217] Here, Elasticsearch (ES) is set as the sink for the Flink task. Once the dependencies have been computed due to meeting the triggering conditions of the time window, they will be persisted to ES in the form of dependencies(ts, ts_index, dependencies).

[0218] Figure 12 This describes the process by which a Flink Job, based on trace data, ultimately converges into a call topology.

[0219] First, the Flink task continuously consumes data from the Collector according to the protobuf serialization method and groups it according to the traceID;

[0220] Then, the call topology graph for each trace ID is calculated based on spanID and parentSpanID;

[0221] Finally, the topology graphs of each trace are aggregated into a DAG graph. Trace call relationships are also written to the trace table, dependency relationships to the dependency table, and metric data for each node to the metrics table.

[0222] Log data for each node (span) is written to the log table.

[0223] Sensitive data will be decrypted before being written to the log table.

[0224] This disclosed embodiment calculates the competitive relationship between two nodes. By tracking the IP field in the data, it is easy to identify competing nodes among different nodes. This relationship can be presented on the final topology graph through Flink job calculation, and metrics of the competing nodes will be displayed. This function is very useful for analyzing the root cause of anomalies, which may be due to resource contention on the same host causing a node in another link to fail.

[0225] The UI displays different dimensions of the pipeline by using trace, dependency, and metrics tables calculated from the Flink Job.

[0226] This disclosure presents a non-intrusive distributed link tracing method based on kernel-level instrumentation, as detailed below:

[0227] This non-intrusive link tracing of applications is achieved by embedding points in the sending and receiving modules of the TCP module in the kernel. The TCP module can obtain relevant tag content in user space and relevant data in IP layer to complete the encapsulation and reporting of trace data, which is something that previous distributed tracing systems have not considered or addressed.

[0228] By adding sampling control to the Collector module, user-defined sampling rate settings can be dynamically received, and then broadcast to the Agent module to dynamically modify the sampling rate in real time. The Agent module then, based on the sampling rate setting, avoids setting TCP headers for trace data that doesn't need to be reported, thus reducing TCP data transmission and minimizing side effects on TCP transmission. This design is not fully considered in Opentracing solutions such as Skywalking.

[0229] While calculating the node call relationships in a single tracing chain using Flink Jobs, it also incorporates the competition relationships of the same host or container into the final topology display. This will improve the accuracy of problem delimitation when performing root cause analysis. This design has not been considered or involved in previous distributed tracing systems.

[0230] By encrypting sensitive fields in the tracking data, even if the data is intercepted during transmission, the information of these sensitive fields cannot be obtained.

[0231] While Skywalking achieves low intrusion for Java applications through bytecode enhancement technology—requiring only the addition of the agent's JAR file and agent configuration at application startup—it still necessitates modifying the application's startup method and restarting the application. This disclosed embodiment, however, only requires installing a TCP-layer agent service on the application nodes to be traced to complete application tracking and trace data reporting.

[0232] Dynamically modifying the sampling rate is problematic because Skywalking only supports setting the sampling rate before application startup and cannot receive sampling rate settings from the upper layer in real time. This results in performance overhead during periods of high traffic, potentially impacting business operations. This embodiment addresses this by establishing a control command broadcasting mechanism for the sampling rate of the Collector and Agent modules to achieve dynamic configuration of the sampling rate. Furthermore, the method by which the Agent module's sampling rate takes effect, and by adding trace data to the TCP header to indicate whether sampling is enabled, significantly reduces data transmission size and minimizes the impact on business operations.

[0233] The topological presentation of competitive relationships in this embodiment adds the display of competitive relationships and the statistical display of competitive relationship metrics to the topological display diagram of Skywalking, so as to provide a strong basis for fault location and root cause analysis and improve the accuracy of problem analysis.

[0234] For the encryption processing of sensitive data, this embodiment of the disclosure implements encrypted data transmission through the AES asymmetric encryption algorithm during the data tracking and reporting stage, thereby ensuring data security.

[0235] like Figure 13 As shown, this disclosure provides a link tracing device, including:

[0236] Insertion module 110 is configured to insert first trace data into TCP packets sent and received by the TCP module using a proxy module integrated into the application node kernel;

[0237] The first generation module 120 is configured to generate and cache second tracking data based on the first tracking data; wherein the second tracking data is used at least for link tracing of the service call corresponding to the TCP packet;

[0238] The first sending module 130 is configured to send the second tracking data to the collection module when the state of the second tracking data meets the first condition.

[0239] In some embodiments, the link tracing device may be an application node containing a proxy module.

[0240] In some embodiments, the insertion module 110, the first generation module 120, and the first sending module 130 may be program modules; after the program modules are executed by the processor, they can perform the above operations.

[0241] In some embodiments, the insertion module 110, the first generation module 120, and the first transmission module 130 may be hardware-software hybrid modules; the hardware-software hybrid modules include, but are limited to, various programmable arrays; the programmable arrays include, but are not limited to, field-programmable arrays and / or complex programmable arrays.

[0242] In some embodiments, the insertion module 110, the first generation module 120, and the first sending module 130 may be pure hardware modules; the pure hardware modules include, but are not limited to, application-specific integrated circuits.

[0243] In some embodiments, the insertion module 110 is configured to perform at least one of the following:

[0244] The first trace data is inserted into the TCP packet containing the start command by using the proxy module of the Transmission Control Protocol (TCP) module integrated in the kernel of the application node.

[0245] The first tracking data is inserted into the TCP packet containing the sending instruction by using the proxy module of the Transmission Control Protocol (TCP) module integrated in the kernel of the application node.

[0246] The first tracking data is inserted into the TCP packet containing the receive instruction by using the proxy module of the Transmission Control Protocol (TCP) module integrated in the kernel of the application node.

[0247] The first trace data is inserted into a TCP packet containing a stop instruction by using a proxy module of the Transmission Control Protocol (TCP) module integrated into the kernel of the application node.

[0248] In some embodiments, the first sending module 130 is configured to send the second tracking data to the collection module when the amount of the second tracking data reaches a first data amount threshold; and / or,

[0249] When the time interval between the last reported time of the second tracking data and the current time reaches the first interval period, the second tracking data is sent to the collection module.

[0250] In some embodiments, the generation module is configured to generate second tracking data containing the first tracking data and the service call status information based on the service call status information of this node;

[0251] The device further includes:

[0252] The first caching module is used to cache the second tracking data.

[0253] In some embodiments, the first caching module is specifically used to cache the second tracking data using a bidirectional queue.

[0254] In some embodiments, the first sending module 130 is configured to generate a data sequence of the second tracking data according to a protocol cache sequence partitioning protocol when the state of the second tracking data meets the first condition; and send the compressed data sequence to the collection module.

[0255] In some embodiments, the apparatus further includes:

[0256] The first receiving module is configured to receive sampling configuration information sent by the collection module, wherein the sampling configuration information includes at least: sampling frequency;

[0257] The insertion module 110 is configured to use a proxy module of the Transmission Control Protocol (TCP) module integrated in the application node kernel to intercept TCP packets sent and received by the TCP module according to the sampling frequency.

[0258] And insert the first trace data into the intercepted TCP packet.

[0259] In some embodiments, the first tracking data includes at least: a tracking identifier ID and a node ID.

[0260] In some embodiments, the first tracking data may further include a parent node identifier.

[0261] In some embodiments, the second tracking data includes the first tracking data and further includes at least one of the following.

[0262] The operation name involved in the TCP message;

[0263] Node labels;

[0264] The network protocol IP address of this node host;

[0265] Timestamp;

[0266] Log events of the link;

[0267] Log information;

[0268] Process identifier;

[0269] The length of the TCP packet.

[0270] In some embodiments, the first sending module 130 is further configured to encrypt the sensitive data before sending the second tracking data to the collection module if the second tracking data includes sensitive data.

[0271] like Figure 14 As shown, this disclosure provides a link tracing device, including:

[0272] The second receiving module 210 is used to perform integrity verification on the second tracking data after receiving the second tracking data sent by the proxy module integrated in the application node kernel.

[0273] The second caching module 220 is used to cache the second trace data that has passed the integrity check and send the second trace data that has passed the integrity check to the analysis module. The second trace data cached by the collection module is used at least for link tracing of the service call corresponding to the TCP packet.

[0274] In some embodiments, the link tracking device may be an electronic device that includes a collection module.

[0275] In some embodiments, the second receiving module 210 and the second buffer module 220 may be program modules; after being executed by the processor, the program modules can perform the above operations.

[0276] In some embodiments, the second receiving module 210 and the second buffer module 220 may be hardware-software hybrid modules; the hardware-software hybrid modules include, but are limited to, various programmable arrays; the programmable arrays include, but are not limited to, field-programmable arrays and / or complex programmable arrays.

[0277] In some embodiments, the second receiving module 210 and the second buffer module 220 may be pure hardware modules; the pure hardware modules include, but are not limited to, application-specific integrated circuits.

[0278] In some embodiments, the apparatus further includes:

[0279] A discard module is used to discard the second tracking data that fails the integrity check.

[0280] In some embodiments, the apparatus further includes:

[0281] The second sending module is used to write the second tracking data cached by the collection module into the storage module when the state of the cached second tracking data reaches the second condition.

[0282] In some embodiments, the second sending module is specifically configured to perform at least one of the following:

[0283] When the amount of the second tracking data reaches the second data amount threshold, the second tracking data is written to the storage module;

[0284] And / or,

[0285] When the time interval between the last reported time of the second tracking data and the current time reaches the second interval period, the second tracking data is written to the storage module.

[0286] like Figure 15 As shown, this disclosure provides a link tracing device including:

[0287] The third receiving module 310 is used to receive the second tracking data sent by the collection module, wherein the second tracking data is generated by the proxy module integrated in the application node kernel;

[0288] The second generation module 320 is used to generate a topology diagram of the service call links represented by the trace ID based on the second trace data.

[0289] In some embodiments, the link tracing device may be an electronic device that includes an analysis module.

[0290] In some embodiments, the third receiving module 310 and the second generating module 320 may be program modules; after being executed by the processor, the program modules can perform the above operations.

[0291] In some embodiments, the third receiving module 310 and the second generating module 320 may be hardware-software hybrid modules; the hardware-software hybrid modules include, but are limited to, various programmable arrays; the programmable arrays include, but are not limited to, field-programmable arrays and / or complex programmable arrays.

[0292] In some embodiments, the third receiving module 310 and the second generating module 320 may be pure hardware modules; the pure hardware modules include, but are not limited to, application-specific integrated circuits.

[0293] In some embodiments, the apparatus further includes:

[0294] An aggregation module is used to aggregate multiple topology graphs into a directed acyclic graph (DAG) when a tracking ID involves multiple topology graphs.

[0295] In some embodiments, the second generation module 320 is specifically used to group the second tracking data according to the tracking ID to obtain second tracking data with the same tracking ID; and to generate a topology map of the calling nodes involved in the link corresponding to the tracking ID based on the node identifier and parent node identifier contained in the second tracking data with the same tracking ID.

[0296] In some embodiments, the apparatus further includes a processing module for performing at least one of the following:

[0297] Based on the topology diagram, the call relationship information between different nodes is written into the tracking table;

[0298] Based on the topology graph, the dependency information between different nodes is written into the dependency table;

[0299] Based on the second tracking data, the node's indicator data is written into the indicator table;

[0300] Based on the second tracking data, the node's log data is written to the log table.

[0301] like Figure 16 As shown, this disclosure provides an electronic device, the electronic device comprising:

[0302] Memory;

[0303] The processor, connected to the memory, is configured to implement the link tracing method provided in any of the foregoing embodiments by executing computer-executable instructions stored in the memory, such as executing... Figure 1 , Figure 5 , Figures 6 to 12 Any link tracing method shown.

[0304] The electronic device may be a terminal device and / or a server in a service platform.

[0305] like Figure 16 As shown, the electronic device may also include a network interface, which can be used to interact with peer devices via a network.

[0306] This disclosure provides a computer storage medium storing computer-executable instructions; when executed by a processor, these computer-executable instructions can implement the link tracing method provided in any of the foregoing embodiments, for example, performing... Figure 1 , Figure 5 , Figures 6 to 12 Any link tracing method shown.

[0307] The computer's storage medium is a non-instantaneous storage medium.

[0308] The technical solutions described in the embodiments of this disclosure can be combined arbitrarily without conflict.

[0309] In the several embodiments provided in this disclosure, it should be understood that the disclosed methods and smart devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0310] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0311] In addition, each functional unit in the various embodiments of this disclosure can be integrated into a second processing unit, or each unit can be a separate unit, or N or more units can be integrated into a unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0312] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure. < / dependency>

Claims

1. A link trace method characterized by, Executed by the application node, including: Using the proxy module integrated into the operating system kernel of the application node, first trace data is inserted into the option field of the TCP packets sent and received by the Transmission Control Protocol (TCP) module; wherein, the option field of the TCP packet is located between the fixed header of the TCP packet and the data portion of the TCP packet; Based on the first tracking data, second tracking data is generated and cached; wherein, the second tracking data is used at least for link tracing of the service call corresponding to the TCP packet; When the status of the second tracking data meets the first condition, the second tracking data is sent to the collection module.

2. The method of claim 1, wherein, The step of generating and caching second tracking data based on the first tracking data includes: Based on the service call status information of this node, generate second trace data that includes the first trace data and the service call status information; Cache the second tracking data.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The system receives sampling configuration information sent by the collection module, wherein the sampling configuration information includes at least: sampling frequency; The step of inserting first trace data into the option field of TCP packets sent and received by the Transmission Control Protocol (TCP) module using the proxy module integrated into the application node's operating system kernel includes: Using a proxy module integrated into the operating system kernel of the application node, TCP packets sent and received by the TCP module are intercepted according to the sampling frequency; And insert the first trace data into the options field of the intercepted TCP packet.

4. The method according to claim 1, characterized in that, The method further includes: If the second tracking data includes sensitive data, the sensitive data is encrypted before the second tracking data is sent to the collection module.

5. A link tracing method, characterized in that, Performed by an electronic device containing a collection module, including: After receiving the second trace data sent by the proxy module integrated in the application node operating system kernel, the integrity of the second trace data is verified; wherein, the second trace data is obtained based on the first trace data, which is obtained by the proxy module inserting it into the option field of the TCP packet sent and received by the Transmission Control Protocol (TCP) module, and the option field of the TCP packet is located between the fixed header of the TCP packet and the data part of the TCP packet; The second trace data that passes the integrity check is cached and sent to the analysis module. The second trace data cached by the collection module is used at least for link tracing of the service call corresponding to the TCP packet.

6. A link tracing method, characterized in that, Performed by an electronic device containing an analysis module, including: The system receives second tracking data sent by the collection module. The second tracking data is generated by the proxy module integrated into the application node operating system kernel based on the first tracking data. The first tracking data is obtained by the proxy module inserting it into the option field of the TCP packet sent and received by the Transmission Control Protocol (TCP) module. The option field of the TCP packet is located between the fixed header of the TCP packet and the data part of the TCP packet. Based on the second tracking data, a topology diagram of the service call chain represented by the tracking ID is generated.

7. The method according to claim 6, characterized in that, The method further includes: When a tracking ID involves multiple topologies, the multiple topologies are aggregated into a directed acyclic graph (DAG).

8. A link tracking device, characterized in that, include: An insertion module is configured to use a proxy module integrated into the application node's operating system kernel to insert first trace data into the option field of TCP packets sent and received by the Transmission Control Protocol (TCP) module; wherein the option field of the TCP packet is located between the fixed header of the TCP packet and the data portion of the TCP packet; The first generation module is configured to generate and cache second tracking data based on the first tracking data; wherein the second tracking data is used at least for link tracing of the service call corresponding to the TCP packet; The first sending module is configured to send the second tracking data to the collection module when the status of the second tracking data meets the first condition.

9. An electronic device, characterized in that, The electronic device includes: Memory; A processor, connected to the memory, is configured to implement the link tracing method provided by any one of claims 1 to 4, 5, or 6 to 7 by executing computer-executable instructions stored in the memory.

10. A computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions; when executed by a processor, the computer-executable instructions can implement the link tracing method provided in any one of claims 1 to 4, 5, or 6 to 7.

Images