Full-link tracking method and system based on network message

Abstract

The invention provides a full-link tracking method and system based on a network message. The full-link tracking method comprises the following steps: S1, capturing a data packet through bypass equipment in a network environment to generate an independent Span record; and S2, triggering full-link query to generate a full-link Trace record. According to bypass-based network data packet acquisition, complete request parameters and response data can be obtained through decoding, so that the problem of data acquisition is solved under the condition that an existing system is not changed at all.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed systems and application performance management, and in particular to a method and system for full-link tracing based on network messages. Background Art

[0002] Full-link tracing is one of the important means to troubleshoot and locate problems in distributed systems. In a distributed system, each user request needs to call multiple subsystems to return the final result. If a user request fails or a request takes a long time, it is necessary to locate which subsystem has a problem. Full-link tracing is an effective way to solve such problems.

[0003] There are already some solutions for full-link tracking, such as: client-based tracking, Java bytecode injection, log collection, etc. Client-based tracking requires all services on the link to be modified in order to complete data collection, which requires a large amount of work and a long completion period. Some systems are even impossible to modify. The Java bytecode injection-based method only supports Java applications, but most current distributed systems are multi-language mixed, and this method is difficult to achieve full-link coverage. Log collection-based tracking relies on the output of the current log information, and it is difficult to ensure that the required related information exists in the log. At the same time, these methods all rely on the global association ID to obtain a complete full-link record, which can only be achieved after the entire system is uniformly modified.

[0004] Patent document CN109359094B (application number: CN201810877216.4) discloses a distributed system log full-link tracking method and device. The method includes: S1, creating a configuration file to configure the system, and executing according to the following process: configuring cluster servers; configuring cluster dependencies; configuring data logical relationships; S2, reading and parsing the configuration file through the application to obtain key information; S3, obtaining the life cycle of system data based on key information. The device includes a system configuration unit, an information acquisition unit and a cluster traversal unit. The full-link tracking of data logs is abstracted into three configuration files. By reading and parsing these configuration files through the corresponding application, the key information such as the server information corresponding to each cluster, the upstream and downstream order of the system, and the logical relationship between the system data can be obtained. This can be achieved without code tracking, and the life cycle of a system data can be obtained in one click, and problems encountered online or during testing can be located. Summary of the invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a full-link tracking method and system based on network messages.

[0006] A full-link tracking method based on network messages provided by the present invention includes:

[0007] Step S1: Capturing data packets through a bypass device in a network environment to generate a separate Span record;

[0008] Step S2: trigger a full-link query to generate a full-link Trace record.

[0009] Preferably, the step S1 comprises:

[0010] Step S1.1: Collecting network traffic of each segment through a bypass device using a TAP method in a network environment;

[0011] Step S1.2: Perform protocol decoding and transaction association on each segment of collected network traffic to obtain transaction records;

[0012] Step S1.3: Convert the transaction record format to obtain a span record in opentelemetry / opentracing format.

[0013] Preferably, the transaction association in step S1.2 is represented by a mesh structure; wherein the mesh structure includes points and edges, each point represents each subsystem; and each edge represents that two subsystems have a mutual calling relationship.

[0014] Preferably, the transaction record is represented by Biz_ID, Trans_ID, duration, ret_code, req content, and resp content;

[0015] Wherein, the Biz_ID represents the business ID requested by the global user;

[0016] The Trans_ID represents a single segment association field;

[0017] The duration represents a single-segment response time;

[0018] The ret_code represents a response code, which is used to indicate whether the request is successful;

[0019] The req content and resp content represent the request and response respectively, and contain all the business information in the request and response.

[0020] Preferably, the span record is represented by name, traceID, spanID, startTime, endTime, parent, status, links, events, attributes;

[0021] Wherein, the name indicates a name, which is used to describe the meaning of the current segment;

[0022] The traceID represents the global ID requested by the user;

[0023] The spanID represents the unique ID of a single span;

[0024] The startTime and endTime represent the start time and the end time respectively;

[0025] The parent represents the parent node of the current span;

[0026] The status indicates the request status;

[0027] The links represent the relationship between the records and other spans;

[0028] The events represent events that occur within a single-segment record;

[0029] The attributes represent the properties associated with the span record, and the user describes the characteristics of the span.

[0030] A full-link tracking system based on network messages provided by the present invention includes:

[0031] Module M1: Generate separate Span records by capturing packets through bypass devices in a network environment;

[0032] Module M2: Trigger full-link query to generate full-link Trace records.

[0033] Preferably, the module M1 comprises:

[0034] Module M1.1: Collect network traffic of each segment through bypass devices using TAP method in the network environment;

[0035] Module M1.2: Perform protocol decoding and transaction association on each segment of collected network traffic to obtain transaction records;

[0036] Module M1.3: Convert the transaction records into span records in opentelemetry / opentracing format.

[0037] Preferably, the transaction association in the module M1.2 is represented by a mesh structure; wherein the mesh structure includes points and edges, each point represents each subsystem; and each edge represents that there is a mutual calling relationship between two subsystems.

[0038] Preferably, the transaction record is represented by Biz_ID, Trans_ID, duration, ret_code, req content, and resp content;

[0039] Wherein, the Biz_ID represents the business ID requested by the global user;

[0040] The Trans_ID represents a single segment association field;

[0041] The duration represents a single-segment response time;

[0042] The ret_code represents a response code, which is used to indicate whether the request is successful;

[0043] The req content and resp content represent the request and response respectively, and contain all the business information in the request and response.

[0044] Preferably, the span record is represented by name, traceID, spanID, startTime, endTime, parent, status, links, events, attributes;

[0045] Wherein, the name indicates a name, which is used to describe the meaning of the current segment;

[0046] The traceID represents the global ID requested by the user;

[0047] The spanID represents the unique ID of a single span;

[0048] The startTime and endTime represent the start time and the end time respectively;

[0049] The parent represents the parent node of the current span;

[0050] The status indicates the request status;

[0051] The links represent the relationship between the records and other spans;

[0052] The events represent events that occur within a single-segment record;

[0053] The attributes represent the properties associated with the span record, and the user describes the characteristics of the span.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] 1. The present invention is based on bypass network data packet collection, and can obtain complete request parameters and response data through decoding, thereby solving the problem of data collection without changing the existing system at all;

[0056] 2. The present invention can output records into standard opentelemetry / opentracing formats by configuring the association logic of single-segment request responses and multi-segment association logic, and can implement a hybrid system with new and old subsystems;

[0057] 3. The present invention searches and associates the tracking records of all subsystems through a parallel deep traversal algorithm, which can effectively solve the problem of the absence of a global association ID. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0059] Figure 1 This is a system framework diagram for full-link tracking based on network messages. DETAILED DESCRIPTION

[0060] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0061] Example 1

[0062] According to a full-link tracking method and system based on network messages provided by the present invention, Figure 1 As shown, packet capture is performed through bypass, and through real-time packet decoding, transaction correlation output conforms to the opentelemetry / opentracing format records, thereby realizing a full-link tracking system that can mix "systems that meet the standard format" and "completely unmodified old systems."

[0063] The full-link tracking method based on network messages includes the following steps:

[0064] Step 1: Generate a separate tracing record, that is, generate a Span record compatible with the opentelemetry / opentracing format by bypassing the packet mirroring method.

[0065] Step 2: Execute a full-link query to generate a complete full-link Trace record, that is, associate all the Span records requested by a single user together based on the Span records generated by each subsystem.

[0066] Wherein, the step 1 comprises the following steps:

[0067] Step 1.1: Data collection in each subsystem: The present invention requires a "full-link tracking system" that collects the inlet network traffic of each subsystem. Network data collection can generally adopt, but is not limited to, the TAP method.

[0068] Step 1.2: Perform protocol decoding and transaction association on each segment of the collected network traffic. The entire system is a mesh structure, and each point is a subsystem. If two subsystems call each other, an edge is formed, and each edge is a segment. Through protocol decoding and transaction association, the transaction records of all segments can be obtained. Transaction records can be described using (Biz_ID, Trans_ID, duration, ret_code, req content, resp content), where Biz_ID represents the business ID of the global user request, the Trans_ID field represents the single-segment association field; duration represents the single-segment response time; ret_code represents the response code, which can be used to indicate whether the request is successful; req and resp represent request and response respectively, and contain all business information in the request and response.

[0069] Biz_ID is a field specified by the user through configuration, which generally corresponds to a field in the request, such as serial number, order number, etc.

[0070] Trans_ID is generated by the user through configuration of a specified field or several fields. For synchronous protocols, a four-tuple (source IP address, source port, destination IP address, destination port) can be used. For asynchronous protocols, fields in both the request and response are generally used, such as serial number, order number, etc.

[0071] Step 1.3: Convert the obtained transaction records to generate span records in opentelemetry / opentracing format. Its format is: (name, traceID, spanID, startTime, endTime, parent, status, links, events, attributes)

[0072] name indicates the name, which can be used to describe the meaning of the current segment. It can be extracted from the transaction record request or response content through configuration. For example, req.url uses the requested URL to describe the current segment.

[0073] traceID represents the global ID requested by the user. Each span record usually has the same value. It usually corresponds to Biz_ID in the transaction record.

[0074] SpanID represents the unique ID of a single span. Since the present invention uses the mirroring method of network traffic and does not modify the single request data, the spanID cannot be passed backward. Therefore, the present invention extracts the operation name of the request as the spanID.

[0075] startTime and endTime represent the start time and end time respectively; in the transaction record, they correspond to the time in req and the time in resp respectively.

[0076] parent represents the parent node of the current span (that is, the previous preceding segment of this segment. The user request will first pass through the previous preceding segment, and then the next step will reach the current segment). Parent is represented by a tuple (traceID, spanID); parent.traceID is equivalent to traceID, but additional configuration is required when the transaction number changes, otherwise the subsequent full link association will be broken. parent.spanID needs to be specified by the user, usually extracting the operation of the current request, and obtaining the preceding operation of the current operation through a table lookup, and using the preceding operation as the value of parent.spanID.

[0077] Status indicates the request status, which can be obtained through ret_code conversion. The conversion logic is specified by the user through configuration.

[0078] Links represent the relationship between span records and other span records, and usually do not require configuration.

[0079] Events represent events that occur within a single segment record; it contains the req and resp content in the transaction record.

[0080] Attributes represent the properties associated with span records, and users describe the characteristics of spans. Usually, users specify some fields in req and resp through configuration. Extracting attributes is beneficial for subsequent retrieval of span records, because the fields in attributes are usually indexed when spans are stored.

[0081] The present invention also provides a full-link tracking system based on network messages. The full-link tracking system based on network messages can be implemented by executing the process steps of the full-link tracking method based on network messages, that is, those skilled in the art can understand the full-link tracking method based on network messages as a preferred implementation of the full-link tracking system based on network messages.

[0082] Those skilled in the art know that, in addition to implementing the system, device and its various modules provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers and embedded microcontrollers by logically programming the method steps. Therefore, the system, device and its various modules provided by the present invention can be considered as a hardware component, and the modules included therein for implementing various programs can also be considered as structures within the hardware component; the modules for implementing various functions can also be considered as both software programs for implementing the method and structures within the hardware component.

[0083] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A full-link tracking method based on network messages, characterized in that: include: Step S1: Capturing data packets through a bypass device in a network environment to generate a separate Span record; Step S2: trigger a full-link query to generate a full-link Trace record.

2. The full-link tracking method based on network messages according to claim 1 is characterized in that: The step S1 comprises: Step S1.1: Collecting network traffic of each segment through a bypass device using a TAP method in a network environment; Step S1.2: Perform protocol decoding and transaction association on each segment of collected network traffic to obtain transaction records; Step S1.3: Convert the transaction record format to obtain a span record in opentelemetry / opentracing format.

3. The full-link tracking method based on network messages according to claim 2 is characterized in that: In step S1.2, the transaction association is represented by a mesh structure; wherein the mesh structure includes points and edges, each point represents each subsystem; and each edge represents that two subsystems have a mutual calling relationship.

4. The full-link tracking method based on network messages according to claim 2 is characterized in that: The transaction record is represented by Biz_ID, Trans_ID, duration, ret_code, req content, and resp content; Wherein, the Biz_ID represents the business ID requested by the global user; The Trans_ID represents a single segment association field; The duration represents a single-segment response time; The ret_code represents a response code, which is used to indicate whether the request is successful; The req content and resp content represent the request and response respectively, and contain all the business information in the request and response.

5. The full-link tracking method based on network messages according to claim 2 is characterized in that: The span record is represented by name, traceID, spanID, startTime, endTime, parent, status, links, events, and attributes; Wherein, the name indicates a name, which is used to describe the meaning of the current segment; The traceID represents the global ID requested by the user; The spanID represents the unique ID of a single span; The startTime and endTime represent the start time and the end time respectively; The parent represents the parent node of the current span; The status indicates the request status; The links represent the relationship between the records and other spans; The events represent events that occur within a single-segment record; The attributes represent the properties associated with the span record, and the user describes the characteristics of the span.

6. A full-link tracking system based on network messages, characterized in that: include: Module M1: Generate separate Span records by capturing packets through bypass devices in a network environment; Module M2: Trigger full-link query to generate full-link Trace records.

7. The full-link tracking system based on network messages according to claim 6 is characterized in that: The module M1 comprises: Module M1.1: Collect network traffic of each segment through bypass devices using TAP method in the network environment; Module M1.2: Perform protocol decoding and transaction association on each segment of collected network traffic to obtain transaction records; Module M1.3: Convert the transaction records into span records in opentelemetry / opentracing format.

8. The full-link tracking system based on network messages according to claim 7 is characterized in that: The transaction association in the module M1.2 is represented by a mesh structure; wherein the mesh structure includes points and edges, each point represents each subsystem; and each edge represents that there is a mutual calling relationship between two subsystems.

9. The full-link tracking system based on network messages according to claim 7 is characterized in that: The transaction record is represented by Biz_ID, Trans_ID, duration, ret_code, req content, and resp content; Wherein, the Biz_ID represents the business ID requested by the global user; The Trans_ID represents a single segment association field; The duration represents a single-segment response time; The ret_code represents a response code, which is used to indicate whether the request is successful; The req content and resp content represent the request and response respectively, and contain all the business information in the request and response.

10. The full-link tracking system based on network messages according to claim 7, characterized in that: The span record is represented by name, traceID, spanID, startTime, endTime, parent, status, links, events, and attributes; Wherein, the name indicates a name, which is used to describe the meaning of the current segment; The traceID represents the global ID requested by the user; The spanID represents the unique ID of a single span; The startTime and endTime represent the start time and the end time respectively; The parent represents the parent node of the current span; The status indicates the request status; The links represent the relationship between the records and other spans; The events represent events that occur within a single-segment record; The attributes represent the properties associated with the span record, and the user describes the characteristics of the span.

Images