Otel-based distributed tracing data processing method and system

Abstract

The application relates to an OTel-based distributed tracking data processing method, which comprises the following steps: receiving tracking data in an OTLP protocol through an OTel native receiving component, caching the tracking data into a cache pipeline of a custom export component, continuously reading the tracking data from the cache pipeline by a write process of the custom export component, judging whether to export the tracking data based on a throughput threshold value, and if yes, storing the tracking data into a time series database table based on a gRPC protocol. Through the application, the problem of high cost of self-developed components in the OTel architecture is solved, only the OTel native receiving component and the custom export component are reserved, other components need not to be coordinated, the dependency tree of the receiving component and the custom export component has no intersection, and the receiving component and the custom export component can be independently upgraded; the data is temporarily stored in the cache pipeline, and the strong time sequence coupling of receiving and exporting is broken; through the OTel native receiving component and the custom export component, semi-dynamic table building is realized under the premise of OTLP compatibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular to a distributed tracing data processing method and system based on OTel. BACKGROUND

[0002] OpenTelemetry (OTel) is an open-source observability framework that collects, processes, and exports traces, performance metrics, and logs of distributed systems through standardized APIs, SDKs, and other tools.

[0003] In OTel, if the existing modules do not meet the requirements, corresponding components need to be developed. Under the existing OTel architecture, all dependencies (external libraries, frameworks, or modules) of the self-written components must be kept in complete version consistency with the common dependencies of the OpenTelemetry existing components, so as to ensure the successful self-compilation of the Collector. However, coordinating all common dependencies of all components to the same version is complex and time-consuming, and the development cost is high. Moreover, the Trace data in the traditional OTLP format is semi-structured nested data, which is difficult to store and query efficiently. SUMMARY

[0004] Embodiments of the present application provide a distributed tracing data processing method and system based on OTel, electronic equipment and storage medium, to at least solve the problem of high cost of developing components in the OTel architecture in related technologies.

[0005] In a first aspect, embodiments of the present application provide a distributed tracing data processing method based on OTel, the method comprising:

[0006] receiving tracing data in OTLP protocol through a receiving component of OTel;

[0007] caching the tracing data into a cache pipeline of a custom export component;

[0008] a write process of the custom export component continuously reads the tracing data from the cache pipeline, and judges whether to export the tracing data based on a throughput threshold, and if so, stores the tracing data into a time series database table based on gRPC protocol.

[0009] In some embodiments, the tracing data includes a plurality of span data units, and the caching of the tracing data into the cache pipeline of the custom export component comprises:

[0010] splitting the tracing data into a plurality of storage units, one storage unit corresponding to one span data unit;

[0011] cache the storage unit into the cache pipeline.

[0012] In some embodiments, determining whether to export the trace data based on the throughput threshold comprises:

[0013] periodically obtaining a current throughput threshold from a configuration service, determining whether a current data throughput exceeds the throughput threshold, wherein the data throughput is a throughput of reading data by the write process to the cache pipeline,

[0014] if yes, triggering a throttling mechanism, discarding the current storage unit, and generating a discard log, the discard log recording an event of discarding the storage unit,

[0015] if no, storing the current storage unit into the time series database table based on an Arrow Flight SQL protocol in gRPC.

[0016] In some embodiments, determining whether the current data throughput exceeds the throughput threshold comprises:

[0017] determining a speed threshold according to the throughput threshold, and controlling a timer to generate timing data based on the speed threshold;

[0018] comparing a generation speed of the timing data with a generation speed of the trace data, the generation speed of the trace data being a speed of reading trace data by the write process to the cache pipeline,

[0019] if the generation speed of the trace data is greater than the generation speed of the timing data, considering that the current data throughput exceeds the throughput threshold.

[0020] In some embodiments, generating the discard log comprises:

[0021] generating the discard log immediately when discarding the storage unit triggered by the throttling mechanism for the first time;

[0022] after a preset time interval, generating the discard log based on a total discard number of the storage unit from a time when the throttling mechanism was triggered last time to a current time when the throttling mechanism is triggered again.

[0023] In some embodiments, each of the storage units comprises an index field, a non-index field, and other fields; and storing the trace data into the time series database table based on the gRPC protocol comprises:

[0024] determining a field type of the time series database table based on a first piece of the trace data, and constructing the time series database table based on the field type;

[0025] storing an index field and a non-index field of each of the storage units into corresponding fields of the time series database table;

[0026] encoding other fields of the storage units into a json format string and storing the json format string into a catch-all storage field of the time series database table.

[0027] In a second aspect, an embodiment of the present application provides a distributed tracking data processing system based on OTel, the system comprising: an OTel native receiving component and a custom export component,

[0028] The OTel native receiving component is configured to receive tracking data in OTLP protocol.

[0029] The custom export component comprises a cache pipeline and a write process,

[0030] The cache pipeline is configured to cache the tracking data.

[0031] The write process is configured to continuously read the tracking data from the cache pipeline, and determine whether to export the tracking data based on a throughput threshold, and if so, store the tracking data into a time series database table based on gRPC protocol.

[0032] In some embodiments, the tracking data comprises a plurality of span data units, and the cache pipeline is further configured to split the tracking data into a plurality of storage units, wherein one storage unit corresponds to one span data unit.

[0033] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the distributed tracking data processing method based on OTel as described in the first aspect.

[0034] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium having a computer program stored thereon, wherein the program is executable on a processor to implement the distributed tracking data processing method based on OTel as described in the first aspect.

[0035] Compared with the related art, the distributed tracking data processing method based on OTel provided by the embodiment of the present application only retains two necessary components, i.e., the OTel native receiving component and the custom export component, without the need to coordinate other components; and the OTel native receiving component is only related to the dependency of OTel, and the custom export component is related to the dependency of storage requirement, and the dependency trees of the two components have no intersection, and can be independently upgraded, thereby solving the problem of high cost of self-developed components in the OTel architecture.

[0036] The data is temporarily stored in the cache pipeline, breaking the strong timing coupling of receiving and exporting, realizing receiving and exporting asynchronously, and avoiding transmission blocking.

[0037] Through the OTel native receiving component and the custom export component, the OTLP format tracking data is converted into data that can be stored in the time series database table based on the gRPC protocol under the premise of OTLP compatibility, and semi-dynamic table building is realized. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0039] Figure 1 is a flow chart of an OTel-based distributed tracking data processing method according to an embodiment of the application;

[0040] Figure 2 is a principle diagram of an OTel-based distributed tracking data processing method according to an embodiment of the application;

[0041] Figure 3 is a structure block diagram of an OTel-based distributed tracking data processing system according to an embodiment of the application;

[0042] Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical scheme and advantages of the application clearer, the application is described and explained below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application. Based on the embodiments provided in the application, all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the application.

[0044] Obviously, the drawings in the following description are only some examples or embodiments of the application, and for those of ordinary skill in the art, the application can be applied to other similar scenarios without making creative efforts based on these drawings. In addition, it can be understood that although the efforts made in this development process can be complex and lengthy, for those of ordinary skill in the art related to the content disclosed in the application, some design, manufacture or production changes based on the technical content disclosed in the application are only routine technical means and should not be understood as insufficient disclosure of the application.

[0045] Reference to an "example" in this application means that a particular feature, structure, or characteristic described in connection with the example can be included in at least one example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same example, nor are they necessarily mutually exclusive of other examples. Of course, one of ordinary skill in the art would understand that a reference to an example in this application can be a reference to at least one example of claim 1 and a phrase in the specification can have the same meaning as the phrase is used in claim 1. It is explicitly contemplated that combinations of features, structures, or characteristics can be made with other examples without departing from the scope of the application.

[0046] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. Unless otherwise defined, the terms "one", "a", "an", "the" and like terms referring to an element will not be limited to the singular but can comprise one or more elements. The terms "including", "containing", "having", and the like are not limiting and are understood to mean "comprising". The terms "connected", "coupled", or the like, unless otherwise defined, are not limited to direct or physical connections or attachments, but can include an electrical connection, whether direct or indirect. The term "multiple" means two or more. The term "and / or" describes associated objects in association with the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally means that the associated objects are in an "or" relationship. The terms "first", "second", "third", and the like, merely distinguish similar objects, and do not represent a specific order.

[0047] OpenTelemetry (OTel) is an open-source observability framework led by the Cloud Native Computing Foundation (CNCF). It standardizes the collection, processing, and export of three pillars of data for distributed systems through standardized Application Programming Interfaces (APIs), Software Development Kits (SDKs), and other tools:

[0048] Traces: Record the call path and time consumption of requests among microservices.

[0049] Metrics: Quantify system performance (e.g., CPU, request latency).

[0050] Logs: Record discrete events and error information.

[0051] The core advantages of OTel lie in cross-language support (including but not limited to Java / Go / Python), vendor neutrality (avoiding locking specific monitoring tools), and simplifying the monitoring integration of cloud-native applications.

[0052] The OpenTelemetry Collector is an independent data processing agent in the OTel ecosystem, responsible for receiving, processing, and forwarding telemetry data. Its core components include:

[0053] Receivers: Support multiple protocols (such as OTLP, Jaeger, Prometheus).

[0054] Processors: Perform data filtering, batching, or enhancement (such as adding attributes).

[0055] Exporters: Send data to the backend (such as Jaeger, Prometheus, log storage). The deployment of the Collector decouples the application from the monitoring backend, supports centralized data processing, reduces network overhead, and improves configuration flexibility (such as dynamically adjusting sampling rules without restarting the service).

[0056] Applications generate data through the SDK of OTel → send to the Collector → Collector processes and distributes to the monitoring backend, forming an end-to-end observability pipeline.

[0057] OTLP (OpenTelemetry Protocol) is the core transmission protocol of the OpenTelemetry project, used to standardize the encoding and transmission of telemetry data (tracing, metrics, logs). It supports gRPC and HTTP protocols, defines data models through protocol buffers (Protobuf), and ensures cross-language and cross-platform interoperability.

[0058] Datalayers is a distributed multi-modal database designed for industrial Internet of Things, Internet of Vehicles, and other scenarios. Datalayers supports time series data storage, key-value storage, and native SQL queries, with features such as storage-compute separation, adaptive compression, and edge-cloud synchronization. Its core advantage lies in handling large-scale time series data, such as sensor monitoring metrics, and providing interpolation functions (such as linear interpolation) to fix data missing problems.

[0059] The embodiment provides a distributed tracing data processing method based on OTel. Figure 1 The flowchart of the distributed tracing data processing method based on OTel according to the embodiment of the application is shown in Figure 1 The flowchart includes the following steps:

[0060] Step S101, receiving the trace data of the OTLP protocol through the receiving component of the OTel native.

[0061] This embodiment no longer uses the self-compilation party recommended by OpenTelemetry Collector, but uses our own new Collector construction method. This embodiment mainly involves the processing of trace data (traces), and the new Collector is named trace data exchanger (trace-exchanger).

[0062] Figure 2 is a kind of based on OTel distributed trace data processing method implementation schematic diagram according to the embodiment of the present application, as Figure 2 The receiving component of the trace data exchanger directly uses the otlp receiver of OpenTelemetry Collector, so that OTLP protocol can be completely compatible to receive OTLP trace data. Directly reuse the official otlp receiver, which is convenient for future upgrade, and does not need to be maintained by yourself, reducing the operation and maintenance cost. The control program (trace-exchanger main) of the trace data exchanger is the registration and control logic of the receiving component and the custom export component. The control program will parse the configuration file when starting the entire trace data exchanger, and will initialize and pull up the receiving component and the custom export component as its child process according to the configuration file.

[0063] Step S102, cache the trace data to the cache pipeline of the custom export component.

[0064] Continue to refer to Figure 2 When the trace data is transmitted from the receiving component (otlp receiver) to the custom export component, the trace data will be placed in a cache pipeline (traceCache channel) first.

[0065] The data is temporarily stored in the cache pipeline, breaking the strong time sequence coupling of receiving and exporting, realizing the asynchronization of receiving and exporting, and avoiding transmission blockage.

[0066] Step S103, the write process of the custom export component continuously reads the trace data from the cache pipeline, and judges whether to export the trace data based on the throughput threshold value, if yes, stores the trace data into the time sequence database table based on the gRPC protocol.

[0067] The background of the custom export component has a write process (TryWriteTraceRecord) that always tries to read trace data from the cache pipeline, and then determines whether to trigger the trace data flow control logic according to the current throughput and the received throughput upper limit, preferably by the number of transactions that can be successfully processed per second (tps) to describe the throughput.

[0068] Through the above method, only two necessary components, the OTel native receiving component and the custom export component, are retained, without the need to coordinate other components, greatly simplifying the customization process; and the OTel native receiving component is only related to the dependency of OTel, and the custom export component is related to the storage requirement dependency, and the dependency trees of the two have no intersection and can be independently upgraded, solving the problem of high cost of self-developed components in the OTel architecture.

[0069] Through the OTel native receiving component and the custom export component, the OTLP format trace data is converted into data that can be stored in the time series database table based on the gRPC protocol under the premise of compatibility with OTLP, and semi-dynamic table building is achieved.

[0070] In some embodiments, the trace data includes a plurality of span data units, and step S102 specifically includes:

[0071] The trace data is split into a plurality of storage units, one storage unit corresponding to one span data unit; and the storage units are cached in the cache pipeline.

[0072] When the receiving component goes to the export component, the export component processes the logic of the trace data, splits all spans in multiple traces into individual storage unit instances through a storage unit (TraceRecord) structure, and puts these storage units into the cache pipeline, and waits for subsequent throughput checking. After the throughput checking is passed, one storage unit is converted into one SQL statement.

[0073] In some embodiments, the judgment of whether to export the trace data based on the throughput threshold in step S103 includes:

[0074] In step S1031, the current throughput threshold is obtained from the configuration service periodically, and it is judged whether the current data throughput exceeds the throughput threshold, wherein the data throughput is the throughput of the write process reading data from the cache pipeline.

[0075] The control of tps_limit can be realized by interacting with the ECP service through ecp.endpoint in the configuration file, and the upper limit value of tps of each trace data exchanger can be dynamically adjusted. It should be noted that the entire communication is encrypted and can prevent tampering.

[0076] Step S1032: If yes, trigger the rate limiting mechanism, discard the current storage unit, and generate a discard log. The discard log records the event of discarding the storage unit.

[0077] Step S1033: If not, store the current storage unit into the time-series database table based on the Arrow Flight SQL protocol in gRPC.

[0078] If the current throughput exceeds the throughput threshold (tps_limit), the write process will discard the current trace data record to prevent excessive pressure on the backend storage service, and simultaneously print a log entry to record the event of discarding trace data. Optionally, to prevent short-term bursts of traffic from affecting rate limiting, the comparison between the throughput threshold and the current throughput can be performed with a specified multiplier, or the average throughput over a specified time period can be compared. If the current throughput is less than the throughput threshold, the trace data record will be converted into a row in the time-series database table and stored in the table.

[0079] Preferably, the current storage unit is stored in the time-series database table based on the Arrow Flight SQL protocol. The ArrowFlight SQL protocol combines Arrow's columnar in-memory format with SQL query capabilities, achieving efficient data movement through the Flight RPC framework. Replacing HTTP / 1.1 with Flight RPC reduces latency, and columnar storage combined with a vectorization engine improves analysis efficiency, making it suitable for scenarios requiring real-time analysis of massive amounts of tracking data.

[0080] In some embodiments, determining whether the current data throughput exceeds the throughput threshold in step S1031 includes:

[0081] Step S201: Determine the speed threshold based on the throughput threshold, and control the timer to generate timing data based on the speed threshold.

[0082] Step S202: Compare the generation speed of the timing data with the generation speed of the trace data. The generation speed of the trace data is the speed at which the writing process reads the trace data from the cache pipe.

[0083] Step S203: If the rate of generation of tracking data is greater than the rate of generation of timed data, then the current data throughput is considered to exceed the throughput threshold.

[0084] There is no component in the OpenTelemetry Collector native component that can control the speed of trace data consumption. The trace data exchanger of the present embodiment controls the trace tps through its own logic. The basic logic includes: when a span data of trace data, i.e., a storage unit, is written into the cache pipeline through the writing process, a timer is used to control the speed first. For example, tps_limit = 1000, and the timer generates a timing data every 1 / 1000 second. As long as the generation speed of trace data is slower than 1 / 1000 second, the writing process will convert the current storage unit into a SQL statement and then write it into the database. If the generation speed of trace data is faster than 1 / 1000 second, the flow control will be triggered, and the current storage unit will be discarded directly.

[0085] In order to prevent short-term traffic impact from causing data loss, a larger time period average flow rate check can also be added. Only when the average tps in the preset time period (for example, ten minutes) exceeds tps_limit, the current storage unit will be discarded.

[0086] In some embodiments, the generating the discard log in step S1032 includes:

[0087] The discard log is generated immediately when the first time the flow limiting mechanism discards the storage unit.

[0088] After a preset time interval, the total discard number of storage units from the last time the flow limiting mechanism was triggered to the current time is counted when the flow limiting mechanism is triggered again, and the discard log is generated based on the total discard number.

[0089] When the storage unit is discarded due to throughput limitation, a log will be printed. In order to prevent too many repeated logs from being printed in a short time, the present embodiment uses the preventTooManyLogs mechanism to control the amount of log output.

[0090] Optionally, the number of printed logs can be controlled according to the time interval. A log is printed every fixed time (for example, one minute). The first time the tps flow limiting is triggered, a log is printed immediately. The tps flow limiting is triggered again within one minute, only the discard number is accumulated, but no log is printed. The tps flow limiting is triggered again after one minute, and the total discard number from the last time the tps was triggered to the current time is printed.

[0091] In some embodiments, each storage unit includes an index field, a non-index field, and other fields; and the storing the trace data into the time series database table based on the gRPC protocol in step S103 includes:

[0092] Step S1034, determining the field type of the time series database table based on the first piece of tracking data, and constructing the time series database table based on the field type.

[0093] The field type will be determined by the first piece of tracking data, and the table will be dynamically built in combination with the index field and the non-index field in the configuration file. Since the data type of each field of the span in the tracking data is fixed, only the first piece of data is needed for dynamic table building.

[0094] Step S1035, storing the index field and the non-index field of each storage unit into the corresponding fields of the time series database table.

[0095] Step S1036, encoding the other fields of the storage unit into a json format string and storing them into the catch-all storage field of the time series database table.

[0096] The span is composed of the index field, the non-index field and the other fields, and one storage unit corresponds to one span, that is, the storage unit is also composed of the index field, the non-index field and the other fields.

[0097] When the value of the tracking data matching the field is equal to any one of the key value list of a certain time series database table, the current tracking data is stored in the time series database table.

[0098] The index field of the span is filled into the index field of the table structure, the non-index field of the span is filled into the non-index field of the table structure, and if a certain field in the span is neither in the index field nor in the non-index field, all these fields are encoded into a json format string and placed in the catch-all storage field (contributes field).

[0099] Through the above process, the conversion of one piece of OTLP format tracking data to time series database table data is completed.

[0100] It should be noted that the steps shown in the above process or the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that here.

[0101] The embodiment also provides an OTel-based distributed tracking data processing system, which is used to implement the above-mentioned embodiments and preferred embodiments, and has been described above. As used below, the terms "module", "unit", "sub-unit" and the like can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, hardware or a combination of software and hardware is also possible and contemplated.

[0102] Figure 3 is a structural block diagram of an OTel-based distributed trace data processing system according to an embodiment of the present application, as shown in the figure, the system comprises an OTel native receiving component 31 and a custom export component 32, Figure 3

[0103] The OTel native receiving component 31 is configured to receive trace data in the OTLP protocol.

[0104] The custom export component 32 comprises a cache pipeline 321 and a write process 322.

[0105] The cache pipeline 321 is configured to cache the trace data.

[0106] The write process 322 is configured to continuously read the trace data from the cache pipeline 321, and determine whether to export the trace data based on a throughput threshold value, and if so, store the trace data into a time series database table based on the gRPC protocol.

[0107] In some embodiments, the trace data comprises a plurality of span data units, and the cache pipeline 321 is further configured to split the trace data into a plurality of storage units, wherein one storage unit corresponds to one span data unit.

[0108] In some embodiments, the write process 322 comprises a flow limiting determination module, a discarding module and a storage module.

[0109] The flow limiting determination module is configured to periodically obtain a current throughput threshold value from a configuration service, and determine whether a current data throughput exceeds the throughput threshold value, wherein the data throughput is a throughput of reading data from the cache pipeline by the write process,

[0110] The discarding module is configured to, in a case where the current data throughput exceeds the throughput threshold value, trigger a flow limiting mechanism, discard the current storage unit, and generate a discarding log, the discarding log recording an event of discarding the storage unit.

[0111] The storage module is configured to, in a case where the current data throughput does not exceed the throughput threshold value, store the current storage unit into the time series database table based on an Arrow Flight SQL protocol in the gRPC.

[0112] In some embodiments, the flow limiting determination module comprises:

[0113] A timing data module is configured to determine a speed threshold value according to the throughput threshold value, and control the timer to generate timing data based on the speed threshold value.

[0114] ​a comparison module configured to compare a generation speed of the timing data to a generation speed of the trace data, the generation speed of the trace data being a speed at which the write process reads the trace data from the cache pipeline.

[0115] a throttling module configured to consider that the current data throughput exceeds the throughput threshold in a case where the generation speed of the trace data is greater than the generation speed of the timing data.

[0116] In some embodiments, the discarding module comprises a log generation module configured to generate a discarding log immediately when the throttling mechanism is triggered for the first time, and to generate a discarding log based on a total discarding number of the storage units from a last time when the throttling mechanism is triggered to a current time each time a preset time interval elapses and the throttling mechanism is triggered again.

[0117] In some embodiments, each storage unit comprises an index field, a non-index field and other fields, and the write process 322 comprises:

[0118] a table building module configured to determine a field type of the time series database table based on the first trace data, and to build the time series database table based on the field type.

[0119] a first storage module configured to store the index field and the non-index field of each storage unit into corresponding fields of the time series database table.

[0120] a second storage module configured to encode the other fields of the storage unit into a json-formatted string, and to store the json-formatted string into a catch-all storage field of the time series database table.

[0121] Through the above system, only two necessary components, the OTel native receiving component and the custom export component, are retained, and other components do not need to be coordinated, greatly simplifying the customization process; and the OTel native receiving component is only related to the dependency of OTel, and the custom export component is related to the storage requirement dependency, and the dependency trees of the two components have no intersection, and the two components can be upgraded independently, solving the problem of high cost of self-developed components in the OTel architecture.

[0122] The data is temporarily stored in the cache pipeline, breaking the strong time sequence coupling of receiving and exporting, realizing asynchronous receiving and exporting, and avoiding transmission blockage.

[0123] Through the OTel native receiving component and the custom export component, the trace data in the OTLP format is converted into data that can be stored in the time series database table based on the gRPC protocol on the premise of compatibility with OTLP, realizing semi-dynamic table building.

[0124] It should be noted that each of the above modules can be a functional module or a program module, and can be implemented by software or hardware. For the module implemented by hardware, each of the above modules can be located in the same processor; or each of the above modules can also be located in different processors in any combination.

[0125] The embodiment also provides an electronic device including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps in any of the method embodiments.

[0126] Optionally, the electronic device can further include a transmission device and an input / output device, wherein the transmission device is connected with the processor, and the input / output device is connected with the processor.

[0127] Optionally, in the embodiment, the processor can be configured to execute the following steps by the computer program:

[0128] S1, receiving the trace data of the OTLP protocol through the OTel native receiving component.

[0129] S2, caching the trace data into the cache pipeline of the custom export component.

[0130] S3, the write process of the custom export component continuously reads the trace data from the cache pipeline, judges whether to export the trace data based on the throughput threshold, and if so, stores the trace data into the time series database table based on the gRPC protocol.

[0131] It should be noted that the specific examples in the embodiment can refer to the examples described in the above embodiments and optional implementation manners, and the embodiment will not be described here.

[0132] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the application, as Figure 4 shown, an electronic device is provided, which can be a server, and the internal structure diagram of the electronic device can be as Figure 4As shown. The electronic device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium, an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement an OTel-based distributed tracking data processing method.

[0133] Those skilled in the art can understand that Figure 4 The skilled in the art can understand that the structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device can include more or less components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM) and the like.

[0135] Those skilled in the art should understand that each technical feature of the above-mentioned embodiments can be combined arbitrarily, and in order to make the description simple, not all possible combinations of each technical feature in the above-mentioned embodiments are described, however, as long as the combination of technical features does not exist contradictory, it should be considered as the scope of the present application.

[0136] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the patent. It should be noted that for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these are all within the scope of the present application. Therefore, the scope of protection of the patent of the present application should be subject to the appended claims.

Claims

1. A distributed tracking data processing method based on OTel, characterized in that, The method includes: OTel receives OTLP protocol tracking data through its native receiving component; The tracking data is cached in the cache pipeline of the custom export component. The native OTel receiving component is only related to the dependency of OTel, while the custom export component is related to the dependency of storage requirements. The dependency trees of the two have no intersection and can be upgraded independently. The writing process of the custom export component continuously reads the trace data from the cache pipeline and determines whether to export the trace data based on the throughput threshold. If so, the trace data is stored in the time series database table based on the gRPC protocol. The tracking data is divided into multiple storage units, each including an index field, a non-index field, and other fields; storing the tracking data into a time-series database table based on the gRPC protocol includes: Based on the tracking data mentioned in the first article, the field types of the time series database table are determined, and the time series database table is constructed based on the field types; The index field and non-index field of each storage unit are stored in the corresponding field of the time-series database table; The other fields of the storage unit are encoded into JSON format strings and stored in the fallback storage field of the time series database table.

2. The method according to claim 1, characterized in that, The tracking data includes multiple span data units, and caching the tracking data into the cache pipeline of the custom export component includes: The tracking data is divided into multiple storage units, with one storage unit corresponding to one span data unit; The storage unit is cached in the cache pipeline.

3. The method according to claim 2, characterized in that, The step of determining whether to export the tracking data based on the throughput threshold includes: The system periodically retrieves the current throughput threshold from the configuration service to determine whether the current data throughput exceeds the threshold. The data throughput refers to the throughput of the write process reading data from the cache pipeline. If so, the rate limiting mechanism is triggered, the current storage unit is discarded, and a discard log is generated. The discard log records the event of discarding the storage unit. If not, based on the Arrow Flight SQL protocol in gRPC, the current storage unit is stored in the time-series database table.

4. The method according to claim 3, characterized in that, The step of determining whether the current data throughput exceeds the throughput threshold includes: A speed threshold is determined based on the throughput threshold, and a timer is controlled to generate timing data based on the speed threshold. The generation rate of the timing data is compared with the generation rate of the tracking data, where the generation rate of the tracking data is the rate at which the writing process reads the tracking data from the cache pipe. If the rate at which the tracking data is generated is greater than the rate at which the timing data is generated, then the current data throughput is considered to exceed the throughput threshold.

5. The method according to claim 3, characterized in that, The generation of the discard log includes: A discard log is generated immediately when the rate limiting mechanism is triggered for the first time to discard a storage unit. Each time the rate limiting mechanism is triggered again after a preset time interval, the total number of data dropped from the last time the rate limiting mechanism was triggered to the current time is counted, and a drop log is generated based on the total number of data dropped.

6. A distributed tracking data processing system based on OTel, characterized in that, The system includes: a native OTel receiving component and a custom export component. The native OTel receiving component is only related to OTel's dependencies, while the custom export component is related to storage requirements. Their dependency trees do not overlap and they can be upgraded independently. The native receiving component of OTel is used to receive tracking data from the OTLP protocol; The custom export component includes a cache pipeline and a write process. The cache pipeline is used to cache the tracking data. The write process is used to continuously read the trace data from the cache pipeline and determine whether to export the trace data based on the throughput threshold. If so, the trace data is stored in the time series database table based on the gRPC protocol. The tracking data is divided into multiple storage units, each including an index field, a non-index field, and other fields; storing the tracking data into a time-series database table based on the gRPC protocol includes: Based on the tracking data mentioned in the first article, the field types of the time series database table are determined, and the time series database table is constructed based on the field types; The index field and non-index field of each storage unit are stored in the corresponding field of the time-series database table; The other fields of the storage unit are encoded into JSON format strings and stored in the fallback storage field of the time series database table.

7. The system according to claim 6, characterized in that, The tracking data includes multiple span data units, and the cache pipeline is also used to split the tracking data into multiple storage units, wherein one storage unit corresponds to one span data unit.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the OTel-based distributed tracking data processing method as described in any one of claims 1 to 5.

9. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the OTel-based distributed tracing data processing method as described in any one of claims 1 to 5.

Images