Distributed tracking data processing method and system based on OTel

Abstract

The invention relates to an OTel-based distributed tracking data processing method, which comprises the following steps of: receiving tracking data of an OTLP protocol through a native receiving component of an OTel, caching the tracking data into a cache pipeline of a user-defined export component, continuously reading the tracking data from the cache pipeline by a write-in process of the user-defined export component, and judging whether to export the tracking data based on a throughput threshold, and if so, storing the tracking data into a time sequence database table based on a gRPC protocol. Through the method and the device, the problem of high cost of self-development of components in an OTel architecture is solved, only the original receiving component and the user-defined export component of the OTel are reserved, other components do not need to be coordinated, and dependency trees of the receiving component and the user-defined export component are not intersected and can be upgraded independently; temporarily storing the data in a cache pipeline, and breaking strong time sequence coupling of receiving and exporting; and semi-dynamic table establishment is realized on the premise of being compatible with OTLP through a native receiving component and a user-defined export component of the OTel.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a distributed tracing data processing method and system based on OTeI. Background Art

[0002] OpenTelemetry (OTel) is an open source observability framework that uniformly collects, processes, and exports trace data, performance metrics, and logs of distributed systems through standardized APIs, SDKs, and other tools.

[0003] In OTeI, if existing modules don't meet your needs, you must develop your own components. Under the current OTeI architecture, all dependencies (external libraries, frameworks, or modules) of your own components must be fully version-matched with the common dependencies of existing OpenTelemetry components to ensure successful Collector self-compilation. However, coordinating all common dependencies of all components to the same version is complex, time-consuming, and expensive to develop. Furthermore, trace data in the traditional OTLP format is semi-structured and nested, making it difficult to efficiently query directly when stored. Summary of the Invention

[0004] The embodiments of the present application provide an OTeI-based distributed tracing data processing method, system, electronic device, and storage medium to at least address the high cost of developing components in the OTeI architecture in related technologies.

[0005] In a first aspect, an embodiment of the present application provides a distributed tracing data processing method based on OTeI, the method comprising: Receive OTLP protocol tracking data through Otel's native receiving component; Cache the tracking data into the cache pipeline of the custom export component; The writing process of the custom export component continuously reads the tracking data from the cache pipeline and determines whether to export the tracking data based on the throughput threshold. If so, the tracking data is stored in the time series database table based on the gRPC protocol.

[0006] In some embodiments, the tracking data includes a plurality of span data units, and caching the tracking data in a cache pipeline of a custom export component includes: Splitting the tracking data into multiple storage units, where one storage unit corresponds to one span data unit; The storage unit is cached in the cache pipeline.

[0007] In some embodiments, determining whether to export the tracking data based on the throughput threshold includes: Periodically obtain a current throughput threshold from the configuration service and determine whether the current data throughput exceeds the throughput threshold, wherein the data throughput is the throughput of the write process reading data from the cache pipeline. If so, the current limiting mechanism is triggered, the current storage unit is discarded, and a discard log is generated, which records the event of discarding the storage unit. If not, based on the Arrow Flight SQL protocol in gRPC, the current storage unit is stored in the time series database table.

[0008] In some embodiments, determining whether the current data throughput exceeds the throughput threshold includes: determining a speed threshold according to the throughput threshold, and controlling a timer to generate timing data based on the speed threshold; Comparing the generation speed of the timing data with the generation speed of the tracking data, the generation speed of the tracking data being the speed at which the writing process reads the tracking data into the cache pipeline, If the generation speed of the tracking data is greater than the generation speed of the timing data, it is considered that the current data throughput exceeds the throughput threshold.

[0009] In some embodiments, generating a discard log includes: When the current limiting mechanism is triggered for the first time to discard a storage unit, a discard log is immediately generated; Each time the current limiting mechanism is triggered again after a preset time interval, the total number of discards in the storage unit from the last time the current limiting mechanism was triggered to the current time is counted, and a discard log is generated based on the total number of discards.

[0010] In some embodiments, each of the storage units includes an index field, a non-index field, and other fields; and storing the tracking data in a time series database table based on the gRPC protocol includes: Determining a field type of the time series database table based on the first piece of tracking data, and constructing the time series database table based on the field type; Storing the index field and the non-index field of each storage unit into the corresponding fields of the time series database table; The other fields of the storage unit are encoded into a string in json format and stored in the backup storage field of the time series database table.

[0011] In a second aspect, an embodiment of the present application provides an OTeI-based distributed tracing data processing system, the system comprising: an OTeI native receiving component and a custom exporting component, The OTeI native receiving component is used to receive tracking data of the OTLP protocol; The custom export component includes a cache pipeline and a writing process. The cache pipeline is used to cache the tracking data, The write process is used to continuously read the tracking data from the cache pipeline and determine whether to export the tracking data based on a throughput threshold. If so, the tracking data is stored in a time series database table based on the gRPC protocol.

[0012] In some embodiments, the tracing data includes multiple span data units, and the cache pipeline is further used to split the tracing data into multiple storage units, wherein one storage unit corresponds to one span data unit.

[0013] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the OTeI-based distributed tracing data processing method as described in the first aspect above is implemented.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the distributed tracing data processing method based on OTeI as described in the first aspect above is implemented.

[0015] Compared with related technologies, the distributed tracing data processing method based on OTeI provided in the embodiment of the present application only retains two necessary components: the native receiving component of OTeI and the custom export component, without the need to coordinate other components. In addition, the native receiving component of OTeI is only related to the dependencies of OTeI, and the custom export component is related to the dependencies of storage requirements. The dependency trees of the two have no intersection and can be upgraded independently, solving the problem of high cost of developing components in the OTeI architecture.

[0016] Temporarily storing data in the cache pipeline breaks the strong timing coupling between reception and export, realizes asynchronous reception and export, and avoids transmission congestion.

[0017] Through Otel's native receiving components and custom export components, and while maintaining compatibility with OTLP, OTLP-formatted tracking data is converted based on the gRPC protocol into data that can be stored in a time series database table, enabling semi-dynamic table creation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a flowchart of a distributed tracing data processing method based on OTeI according to an embodiment of the present application; Figure 2 This is a schematic diagram of an implementation principle of an OTeI-based distributed tracing data processing method according to an embodiment of the present application; Figure 3 is a structural block diagram of an OTeI-based distributed tracing data processing system according to an embodiment of the present application; Figure 4 Schematic diagram of the internal structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0020] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can, without inventive effort, apply the present application to other similar scenarios based on these drawings. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, changes in design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0021] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0022] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "an," "the," and similar expressions used herein do not denote quantitative limitations and may refer to either the singular or the plural. The terms "comprise," "include," "have," and any variations thereof, used herein, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules (units) is not limited to the listed steps or units but may also include steps or units not listed, or may include other steps or units inherent to the process, method, product, or apparatus. The terms "connected," "connected," "coupled," and similar expressions used herein are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used herein, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" may mean: A exists alone; A and B exist simultaneously; or B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0023] OpenTelemetry (OTel) is an open source observability framework developed by the Cloud Native Computing Foundation (CNCF). It uses standardized application programming interfaces (APIs), software development kits (SDKs), and other tools to uniformly collect, process, and export data from three pillars of distributed systems: Traces: Record the call path and duration of requests between microservices.

[0024] Metrics: Quantify system performance (e.g., CPU, request latency).

[0025] Logs: Record discrete events and error messages.

[0026] OTel's core advantages lie in cross-language support (including but not limited to Java / Go / Python), vendor neutrality (avoiding lock-in to specific monitoring tools), and simplified monitoring integration for cloud-native applications.

[0027] The OpenTelemetry Collector is an independent data processing agent in the OTeI ecosystem, responsible for receiving, processing, and forwarding telemetry data. Its core components include: Receivers: Supports multiple protocols (e.g., OTLP, Jaeger, Prometheus).

[0028] Processors: perform data filtering, batch processing, or enhancement (such as adding attributes).

[0029] Exporters: Send data to backends (such as Jaeger, Prometheus, and log storage). Deploying a Collector decouples applications from the monitoring backend, enabling centralized data processing, reducing network overhead, and increasing configuration flexibility (e.g., dynamically adjusting sampling rules without restarting the service).

[0030] Applications generate data through the OTeI SDK, send it to the Collector, and then distribute it to the monitoring backend after processing, forming an end-to-end observability pipeline.

[0031] OTLP (OpenTelemetry Protocol) is the core transport protocol of the OpenTelemetry project, used to standardize the encoding and transmission of telemetry data (tracing, metrics, and logs). It supports gRPC and HTTP protocols and defines the data model through Protocol Buffers (Protobuf), ensuring cross-language and cross-platform interoperability.

[0032] Datalayers is a distributed multimodal database designed for scenarios such as the Industrial Internet of Things and the Internet of Vehicles. It supports time-series data storage, key-value storage, and native SQL queries, and features storage-computation separation, adaptive compression, and edge-cloud synchronization. Its core strength lies in processing large-scale time-series data, such as sensor monitoring metrics, and providing interpolation capabilities (such as linear interpolation) to correct missing data.

[0033] This embodiment provides a distributed tracing data processing method based on OTeI. Figure 1 is a flow chart of a distributed tracing data processing method based on OTeI according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps: Step S101: Receive OTLP protocol tracking data through the OTel native receiving component.

[0034] This example no longer uses the self-compiled method recommended by the OpenTelemetry Collector, but instead uses our own new Collector assembly method. This example primarily involves processing traces, and the new Collector is named trace-exchanger.

[0035] Figure 2 This is a schematic diagram of a distributed tracking data processing method based on OTeI according to an embodiment of the present application. Figure 2 As shown, the trace data exchanger's receiving component directly uses the OpenTelemetry Collector's built-in OTLP receiver, ensuring full compatibility with the OTLP protocol for receiving OTLP trace data. Directly reusing the official OTLP receiver facilitates future upgrades and eliminates the need for maintenance, reducing operational costs. The trace data exchanger's control program (trace-exchanger main) provides the registration and control logic for the receiving component and custom exporter. When the trace data exchanger starts, the control program parses the configuration file and initializes and starts the receiving component and custom exporter as its own child processes according to the configuration file.

[0036] Step S102: Cache the tracking data into the cache pipeline of the custom export component.

[0037] Continue to refer to Figure 2 When the trace data is passed from the receiving component (otlp receiver) to the custom export component, it is first placed in a cache channel (traceCache channel).

[0038] Temporarily storing data in the cache pipeline breaks the strong timing coupling between reception and export, realizes asynchronous reception and export, and avoids transmission congestion.

[0039] In step S103, the writing process of the custom export component continuously reads the tracking data from the cache pipeline and determines whether to export the tracking data based on the throughput threshold. If so, the tracking data is stored in the time series database table based on the gRPC protocol.

[0040] The custom export component has a writing process (TryWriteTraceRecord) in the background that constantly attempts to read trace data from the cache pipeline. It then decides whether to trigger the trace data flow rate control logic based on the current throughput and the received throughput limit. Preferably, the throughput is described by the number of transactions per second (tps) that can be successfully processed.

[0041] Through the above method, only two necessary components, the native OTeI receiving component and the custom exporting component, are retained, eliminating the need to coordinate other components and greatly simplifying the customization process. Furthermore, the native OTeI receiving component is only related to OTeI's dependencies, while the custom exporting component is related to the dependencies of storage requirements. The dependency trees of the two components do not intersect, and they can be upgraded independently, solving the high cost of developing components in the OTeI architecture.

[0042] Through Otel's native receiving components and custom export components, and while maintaining compatibility with OTLP, OTLP-formatted tracking data is converted based on the gRPC protocol into data that can be stored in a time series database table, enabling semi-dynamic table creation.

[0043] In some embodiments, the tracking data includes multiple span data units, and step S102 specifically includes: Split the tracking data into multiple storage units, one storage unit corresponds to one span data unit; cache the storage units in the cache pipeline.

[0044] When the receiving component transfers to the exporting component, the exporting component processes the logic of the tracing data. It uses a storage unit (TraceRecord) structure to split all spans in multiple traces into storage unit instances, and puts these storage units into the cache pipeline. After the subsequent throughput check passes, each storage unit is converted into an SQL statement.

[0045] In some embodiments, determining whether to export the tracking data based on the throughput threshold in step S103 includes: Step S1031 , periodically obtain the current throughput threshold from the configuration service, and determine whether the current data throughput exceeds the throughput threshold, wherein the data throughput is the throughput of the write process reading data from the cache pipe.

[0046] The tps_limit can be controlled by interacting with the ECP service through the ecp.endpoint in the configuration file, dynamically adjusting the tps limit for each tracking data exchange. It should be noted that the entire communication is encrypted to prevent tampering.

[0047] Step S1032: If yes, trigger the current limiting mechanism, discard the current storage unit, and generate a discard log, which records the event of discarding the storage unit.

[0048] Step S1033: If not, based on the Arrow Flight SQL protocol in gRPC, store the current storage unit into the time series database table.

[0049] If the current throughput exceeds the throughput threshold (tps_limit), the write process discards the current trace data record to prevent excessive pressure on the backend storage service. A log message is also printed to record the discarded trace data event. Optionally, to prevent short bursts of traffic from affecting throttling, the throughput threshold can be compared against the current throughput by adding a specified multiplier, or the average throughput over a specified time period can be compared. If the current throughput is less than the throughput threshold, the trace data record is converted into a row in the time series database table and stored in the table.

[0050] Preferably, the current storage unit is stored in a time series database table based on the Arrow Flight SQL protocol. The ArrowFlight SQL protocol combines Arrow's columnar in-memory format with SQL query capabilities, enabling efficient data movement through the Flight RPC framework. Using Flight RPC instead of HTTP / 1.1 reduces latency, while columnar storage combined with a vectorized engine improves analysis efficiency, making it suitable for scenarios requiring real-time analysis of massive amounts of tracking data.

[0051] In some embodiments, determining whether the current data throughput exceeds the throughput threshold in step S1031 includes: Step S201: determining a speed threshold according to a throughput threshold, and controlling a timer to generate timing data based on the speed threshold.

[0052] Step S202 : comparing the generation speed of the timing data with the generation speed of the tracking data. The generation speed of the tracking data is the speed at which the writing process reads the tracking data from the cache pipeline.

[0053] In step S203 , if the generation speed of the tracking data is greater than the generation speed of the timing data, it is considered that the current data throughput exceeds the throughput threshold.

[0054] The native OpenTelemetry Collector component does not have any function to control the speed of tracing data consumption. The tracing data exchanger in this embodiment controls trace TPS through its own logic. The basic logic includes: when a span of tracing data, that is, a storage unit, is written to the cache pipeline through the writing process, a timer is first used to control the speed. For example, if tps_limit = 1000, the timer will generate a timed data every 1 / 1000 second. As long as the tracing data generation speed is slower than 1 / 1000 second, the writing process will convert the current storage unit into an SQL statement and then write it to the database; if the tracing data generation speed is faster than 1 / 1000 second, the flow rate control will be triggered and the current storage unit will be directly discarded.

[0055] To prevent data loss caused by short-term traffic shocks, you can also add an average flow rate check for a larger time period. Only when the average TPS within the preset time period (for example, ten minutes) exceeds tps_limit will the current storage unit be discarded.

[0056] In some embodiments, generating a discard log in step S1032 includes: A discard log is generated immediately when the current limiting mechanism is triggered for the first time to discard a storage unit.

[0057] Each time the current limiting mechanism is triggered again after a preset time interval, the total number of discards from the storage unit from the last time the current limiting mechanism was triggered to the current time is counted, and a discard log is generated based on the total number of discards.

[0058] When a storage unit is discarded due to throughput limitations, a log is printed. To prevent too many duplicate logs from being printed in a short period of time, this embodiment uses a log storm prevention mechanism (preventTooManyLogs) to control the log output volume.

[0059] Optionally, you can control the number of logs printed based on a time interval, printing a log every fixed time (for example, one minute). The first time TPS throttling is triggered, a log is printed immediately. If TPS throttling is triggered again within one minute, only the number of discarded requests is accumulated but no log is printed. If TPS throttling is triggered again one minute later, the total number of discarded requests from the last time TPS was triggered to the current time is printed.

[0060] In some embodiments, each storage unit includes an index field, a non-index field, and other fields; and storing the tracking data in the time series database table based on the gRPC protocol in step S103 includes: Step S1034: determine the field type of the time series database table based on the first tracking data, and construct the time series database table based on the field type.

[0061] The field type is determined by the first traced data entry, and the table is dynamically created based on the indexed and non-indexed fields in the configuration file. Since the data type of each span field in the traced data is fixed, only the first data entry is needed for dynamic table creation.

[0062] Step S1035 : Store the index field and non-index field of each storage unit into corresponding fields of the time series database table.

[0063] Step S1036: Encode the other fields of the storage unit into a string in json format and store them in the backup storage field of the time series database table.

[0064] A span consists of index fields, non-index fields, and other fields. One storage unit corresponds to one span, which means that a storage unit also consists of index fields, non-index fields, and other fields.

[0065] When the value of the tracking data matching field is equal to any item in the key value list of a time series database table, the current tracking data is stored in the time series database table.

[0066] Fill the index fields of the span into the index fields of the table structure, and fill the non-index fields of the span into the non-index fields of the table structure. If a field in the span is neither in the index field nor in the non-index field, encode all these fields into JSON format strings and put them into the backup storage field (contributes field).

[0067] Through the above process, the conversion of a piece of OTLP format tracking data into time series database table data is completed.

[0068] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0069] This embodiment also provides an OTeI-based distributed tracing data processing system, which is used to implement the above-mentioned embodiments and preferred implementations. Details that have already been described will not be repeated. As used below, the terms "module," "unit," "subunit," etc. may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0070] Figure 3 is a structural block diagram of a distributed tracking data processing system based on OTeI according to an embodiment of the present application, such as Figure 3 As shown, the system includes: OTeI native receiving component 31 and custom export component 32, The OTel native receiving component 31 is used to receive tracking data of the OTLP protocol.

[0071] The custom export component 32 includes a cache pipeline 321 and a writing process 322 .

[0072] The cache pipeline 321 is used to cache trace data.

[0073] The write process 322 is used to continuously read the tracking data from the cache pipeline 321 and determine whether to export the tracking data based on the throughput threshold. If so, the tracking data is stored in the time series database table based on the gRPC protocol.

[0074] In some embodiments, the tracing data includes multiple span data units, and the cache pipeline 321 is further used to split the tracing data into multiple storage units, wherein one storage unit corresponds to one span data unit.

[0075] In some embodiments, the write process 322 includes a current limiting determination module, a discarding module, and a storage module.

[0076] The current limiting judgment module is used to periodically obtain the current throughput threshold from the configuration service and judge whether the current data throughput exceeds the throughput threshold, where the data throughput is the throughput of the write process reading data from the cache pipeline. The discard module is used to trigger the current limiting mechanism when the current data throughput exceeds the throughput threshold, discard the current storage unit, and generate a discard log, which records the event of discarding the storage unit.

[0077] The storage module is used to store the current storage unit in the time series database table based on the Arrow Flight SQL protocol in gRPC when the current data throughput does not exceed the throughput threshold.

[0078] In some embodiments, the current limiting determination module includes: The timing data module is used to determine a speed threshold according to a throughput threshold, and to control a timer to generate timing data based on the speed threshold.

[0079] The comparison module is used to compare the generation speed of the timing data with the generation speed of the tracking data. The generation speed of the tracking data is the speed at which the writing process reads the tracking data from the cache pipeline.

[0080] The current limiting module is used to determine that the current data throughput exceeds the throughput threshold when the generation rate of tracking data is greater than the generation rate of timing data.

[0081] In some embodiments, the discard module includes: a log generation module, which is used to immediately generate a discard log when the current limiting mechanism is triggered for the first time to discard the storage unit; each time the current limiting mechanism is triggered again after a preset time interval, the total number of discards of the storage unit from the last time the current limiting mechanism was triggered to the current time is counted, and a discard log is generated based on the total number of discards.

[0082] In some embodiments, each storage unit includes an index field, a non-index field, and other fields; the writing process 322 includes: The table creation module is used to determine the field type of the time series database table based on the first tracking data and build the time series database table based on the field type.

[0083] The first storage module is used to store the index field and non-index field of each storage unit into the corresponding fields of the time series database table.

[0084] The second storage module is used to encode other fields of the storage unit into a string in json format and store it in the backup storage field of the time series database table.

[0085] Through the above system, only two necessary components, OTeI's native receiving component and the custom export component, are retained, eliminating the need to coordinate other components, greatly simplifying the customization process. Furthermore, OTeI's native receiving component is only related to OTeI's dependencies, while the custom export component is related to the dependencies of storage requirements. The dependency trees of the two components do not intersect, and they can be upgraded independently, solving the high cost of developing components in the OTeI architecture.

[0086] Temporarily storing data in the cache pipeline breaks the strong timing coupling between reception and export, realizes asynchronous reception and export, and avoids transmission congestion.

[0087] Through Otel's native receiving components and custom export components, and while maintaining compatibility with OTLP, OTLP-formatted tracking data is converted based on the gRPC protocol into data that can be stored in a time series database table, enabling semi-dynamic table creation.

[0088] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0089] This embodiment further provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0090] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0091] Optionally, in this embodiment, the processor may be configured to execute the following steps through a computer program: S1, receives OTLP protocol tracking data through OTeI's native receiving component.

[0092] S2, caches the tracking data into the cache pipeline of the custom export component.

[0093] In S3, the writing process of the custom export component continuously reads the tracing data from the cache pipeline and determines whether to export the tracing data based on the throughput threshold. If so, the tracing data is stored in the time series database table based on the gRPC protocol.

[0094] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be repeated here.

[0095] In one embodiment, Figure 4 is a schematic diagram of the internal structure of an electronic device according to an embodiment of the present application, such as Figure 4 As shown, an electronic device is provided, which may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The electronic device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the electronic device is used to store data. The network interface of the electronic device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a distributed tracking data processing method based on OTeI is implemented.

[0096] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0097] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0098] Those skilled in the art should understand that the various technical features of the above-described embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the various technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A distributed tracing data processing method based on OTeI, characterized in that: The method comprises: Receive OTLP protocol tracking data through Otel's native receiving component; Cache the tracking data into the cache pipeline of the custom export component; The writing process of the custom export component continuously reads the tracking data from the cache pipeline and determines whether to export the tracking data based on the throughput threshold. If so, the tracking data is stored in the time series database table based on the gRPC protocol.

2. The method according to claim 1, characterized in that The tracking data includes a plurality of span data units, and caching the tracking data in a cache pipeline of the custom export component includes: Splitting the tracking data into multiple storage units, where one storage unit corresponds to one span data unit; The storage unit is cached in the cache pipeline.

3. The method according to claim 2, characterized in that The determining whether to export the tracking data based on the throughput threshold includes: Periodically obtain a current throughput threshold from the configuration service and determine whether the current data throughput exceeds the throughput threshold, wherein the data throughput is the throughput of the write process reading data from the cache pipeline. If so, the current limiting mechanism is triggered, the current storage unit is discarded, and a discard log is generated, which records the event of discarding the storage unit. If not, based on the Arrow Flight SQL protocol in gRPC, the current storage unit is stored in the time series database table.

4. The method according to claim 3, characterized in that Determining whether the current data throughput exceeds the throughput threshold includes: determining a speed threshold according to the throughput threshold, and controlling a timer to generate timing data based on the speed threshold; Comparing the generation speed of the timing data with the generation speed of the tracking data, the generation speed of the tracking data being the speed at which the writing process reads the tracking data into the cache pipeline, If the generation speed of the tracking data is greater than the generation speed of the timing data, it is considered that the current data throughput exceeds the throughput threshold.

5. The method according to claim 3, characterized in that Generating a discard log includes: When the current limiting mechanism is triggered for the first time to discard a storage unit, a discard log is immediately generated; Each time the current limiting mechanism is triggered again after a preset time interval, the total number of discards in the storage unit from the last time the current limiting mechanism was triggered to the current time is counted, and a discard log is generated based on the total number of discards.

6. The method according to claim 2, characterized in that Each of the storage units includes an index field, a non-index field, and other fields; storing the tracking data in a time series database table based on the gRPC protocol includes: Determining a field type of the time series database table based on the first piece of tracking data, and constructing the time series database table based on the field type; Storing the index field and the non-index field of each storage unit into the corresponding fields of the time series database table; The other fields of the storage unit are encoded into a string in json format and stored in the backup storage field of the time series database table.

7. A distributed tracking data processing system based on OTeI, characterized in that: The system includes: OTeI native receiving components and custom export components, The OTeI native receiving component is used to receive tracking data of the OTLP protocol; The custom export component includes a cache pipeline and a writing process. The cache pipeline is used to cache the tracking data, The write process is used to continuously read the tracking data from the cache pipeline and determine whether to export the tracking data based on a throughput threshold. If so, the tracking data is stored in a time series database table based on the gRPC protocol.

8. The system according to claim 7, characterized in that The tracing data includes multiple span data units, and the cache pipeline is further used to split the tracing data into multiple storage units, wherein one storage unit corresponds to one span data unit.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the OTeI-based distributed tracing data processing method according to any one of claims 1 to 6 is implemented.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the OTeI-based distributed tracing data processing method according to any one of claims 1 to 6 is implemented.

Images