A method for sampling SkyWalking call links

Abstract

The application relates to a SkyWalking call link sampling method based on a computer software technology field.The method comprises the following steps: reporting link data in an agent end to a server end in full amount; forwarding the link data to a Kafka asynchronous consumption; judging the link needing sampling in a first round of consumption and recording a TraceId, writing the TraceId into a sampling Bloom filter, and storing the TraceId into a first storage unit in full amount; judging whether the TraceId exists in the sampling Bloom filter in a second round of delayed consumption, storing the TraceId into a second storage unit and writing the TraceId into a query Bloom filter if the TraceId exists; and routing to the corresponding storage unit according to the TraceId and the query Bloom filter during the query.The application solves the problems of incomplete sampling, inaccurate indexes, incomplete links, high storage cost and low query efficiency through full amount reporting, asynchronous decoupling, two rounds of consumption, double Bloom filters and storage separation.

Description

Technical Field

[0001] This invention relates to the field of computer software technology, and more specifically to a method for sampling call chains based on SkyWalking. Background Technology

[0002] As the company's business expands and system functions iterate, the backend system architecture becomes increasingly complex, leading to a surge in service call volume. SkyWalking, as a core component for observability in distributed systems, needs to handle terabytes (TB) or even petabytes (PB) of service chain data. However, in actual operation and maintenance, developers only focus on a very small portion of the service chain, mainly including erroneous requests, slow requests, test requests, and requests with specific business implications. How to extract the complete service chain of interest to developers from massive amounts of service chain data in a low-cost and accurate manner and store it long-term has become a critical problem that companies implementing SkyWalking urgently need to solve.

[0003] In existing technologies, SkyWalking's trace sampling mechanism is mainly divided into two categories: Agent-side sampling and Server-side sampling. The Agent side employs a header sampling strategy, where upon receiving a request, it randomly decides whether to sample the entire trace based on a preset sampling rate. If sampling is deemed appropriate, the flag is passed down to all downstream services along with the trace context, and all downstream traces are collected and reported. If no sampling is deemed appropriate, no data is reported, and the flag continues to be passed downstream, where the downstream service re-performs the sampling decision. The Server side employs a unit sampling strategy. After trace data is reported to the Server, the hash value of the TraceId is calculated and modulo the preset sampling rate to mark whether the data should be stored. This sampling method only applies to the current service node and is not transitive. At the processing and storage level, the Server synchronously receives Trace data reported from channels such as HTTP, gRPC, and Kafka, and directly performs sampling judgment and analysis. All Trace data is stored in the same Elasticsearch index. Data expiration cleanup cannot distinguish between long-term and short-term storage needs; only a unified setting for the overall Trace storage time can be implemented. Queries directly retrieve the entire storage index.

[0004] The existing technical solutions described above have several drawbacks. First, the sampling at the agent end determines whether to collect data when the link is generated. At this point, it is impossible to predict whether errors or timeouts will occur later in the link, leading to the discarding of faulty links. Furthermore, the sampling markers are only passed downstream, failing to trace back to upstream unsampled nodes, resulting in incomplete links. Second, because the agent-side sampling filters only a portion of the link data, the server cannot obtain full data when calculating metrics such as call count and average response time, leading to discrepancies between statistical results and actual operating conditions. Third, existing solutions only support sampling of faulty or slow links, failing to meet the needs of test links, special business links, and other business scenarios requiring precise filtering and long-term storage. Fourth, some language agents only support synchronous reporting via HTTP and gRPC, requiring synchronous processing by the server, which places high demands on server configuration and increases deployment costs. Fifth, all link data is stored in the same index, with massive amounts of data without long-term retention value consuming storage resources, interfering with query efficiency, and failing to achieve separation of hot and cold data. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides a method for sampling call chains based on SkyWalking. Through full reporting, asynchronous decoupling, two-round consumption, dual Bloom filters, and storage separation mechanisms, it achieves complete collection of call chains, accurate statistical analysis of metrics, flexible business sampling, server-side stress reduction, and optimized storage and querying. This method enables the accurate extraction and long-term storage of complete call chains of interest to developers from massive amounts of call chain data at low cost and high efficiency.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for skywalking-based call chain sampling includes the following steps: The agent side disables the sampling and filtering mechanism and reports all link data to the server side in full. After receiving the link data synchronously via HTTP and gRPC, the server forwards it to Kafka for asynchronous consumption. First round of consumption: Consume the link data in Kafka, determine the link that needs to be sampled and record the TraceId, write the TraceId into the sampling Bloom filter, and then store the link data in full into the first storage unit; Second round of consumption: Delayed consumption of link data in Kafka, determining whether the TraceId of the link data exists in the sampling Bloom filter. If it exists, the link data is stored in the second storage unit, and the TraceId is written into the query Bloom filter. When the query conditions include a specified TraceId, check if the TraceId exists in the query Bloom filter. If it exists, query the second storage unit; otherwise, query the first storage unit.

[0007] Furthermore, the process of reporting all link data to the server includes the following steps: The Agent provides a marking function for business links in the link data, identifies business links that need to be stored for a long time through the marking information, and reports the marking information to the Server. In the first round of consumption, the links that need to be sampled include those identified based on the tagging information.

[0008] Furthermore, the tagging information includes: Add the first tag information to the link using the application programming interface provided by the software development kit; By injecting a second tagging information into the protocol layer carrier.

[0009] Furthermore, in the first round of consumption, the links that need to be sampled include those identified based on faulty or slow links.

[0010] Furthermore, faulty / slow link identification includes: Identify whether there are error status codes in the link; It identifies whether the time taken for each operation in the link exceeds the slow link threshold. The slow link threshold can be dynamically adjusted and differentiated thresholds can be set for different types of components. Different types of components include at least one of database connection and cache query.

[0011] Furthermore, in the first round of consumption, the links that need to be sampled include those identified based on user-generated markings.

[0012] Furthermore, the data expiration times of the first storage unit and the second storage unit are configured independently, with the data expiration time of the first storage unit being shorter than that of the second storage unit.

[0013] Furthermore, when the query conditions do not include a specified TraceId, the first storage unit or the second storage unit is queried according to the query mode selected by the user. The query modes include immediate query and delayed query. Immediate query corresponds to querying the first storage unit, and delayed query corresponds to querying the second storage unit.

[0014] Furthermore, it also includes the following steps: The server performs metric calculations on the link data and writes the calculated metric data to the metric storage cluster separately. The metric storage cluster is physically isolated from the first and second storage units that store the link data. The metrics calculation includes: aggregating and calculating link data by time bucket; periodically checking the metrics data before a preset time window (which is longer than the time bucket), and writing the retrieved metrics data into the metrics storage cluster.

[0015] Furthermore, it also includes the following steps: During the first round of consumption, all link data reported in full is obtained, the link data is formatted into Segment objects and sent to the listener; The listener parses the Segment object, iterates through the Exit Span in the Segment object, and determines whether the middleware called by the Exit Span has encountered an error. If an error has occurred, a middleware error object is constructed. Determine whether the link data contains errors or whether any Exit Span has an error. If an error occurs, use TraceId as the unique key to deduplicate and construct a link error object. The middleware error objects and link error objects are converted into time-series metrics and sent to Prometheus. The Prometheus AlertManager component then uses these metrics to configure alarm rules and trigger alarms.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention disables the sampling filtering mechanism on the agent side, reporting all link data in full to the server side, thus avoiding the problem of discarding erroneous or slow requests due to header sampling in existing technologies. Simultaneously, since the server side obtains all the data, it can perform statistical analysis based on the actual full data when generating various observable metrics, fundamentally solving the technical problem of inaccurate metric calculations caused by agent-side sampling.

[0017] 2. This invention utilizes the buffering mechanism of the Kafka message queue and employs a two-round consumption strategy combining first-round and second-round consumption. In particular, the second-round consumption uses a delayed consumption mechanism to ensure that sampling, judgment, and storage are only performed after all data from upstream and downstream nodes in the link have been reported. Compared to existing technologies where unit sampling can only capture fragments of the link and header sampling cannot trace back upstream, this invention can accurately identify and store complete end-to-end data, providing reliable data support for fault location.

[0018] 3. In this invention, after the server synchronously receives link data via HTTP and gRPC, it does not process it but directly forwards it to Kafka for asynchronous consumption. This mechanism allows the server to quickly release connections and utilizes Kafka's high throughput to buffer data, significantly reducing the real-time processing pressure on the server. With the same data volume, stable operation is possible without configuring high-specification servers, thereby reducing server deployment costs and operational complexity.

[0019] 4. This invention introduces a hot and cold data separation architecture between a first storage unit and a second storage unit. All data is only temporarily stored in the first storage unit, while only critical link data identified through sampling (e.g., based on tagging information, faulty / slow link identification, or user-manual tagging) is stored long-term in the second storage unit. Simultaneously, by combining a sampling Bloom filter and a query Bloom filter, queries can be quickly routed to the corresponding storage unit, avoiding invalid retrieval from massive amounts of temporary data. This solution significantly reduces the storage cost of massive link data and significantly improves the query response speed of core business links.

[0020] 5. This invention abandons the traditional, single random sampling mode and constructs a scalable sampling and marking system. On the one hand, it supports dynamically adding marking information to business logic through software development kits; on the other hand, it supports injecting marking information into the protocol layer carrier; in addition, it supports automatic identification of faulty and slow links (supporting dynamic adjustment of slow link thresholds and fine-grained configuration at the component level) and manual marking by users. The diverse sampling strategies enable the system to accurately capture the business links that are truly of interest to R&D, testing, and operations personnel, greatly improving the value density of observable data. Attached Figure Description

[0021] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0022] Terminology explanation.

[0023] SkyWalking: An open-source distributed systems observability platform under the Apache Foundation, specifically designed for monitoring, tracing, and diagnosing microservices, cloud-native, and containerized architectures.

[0024] HTTP: An application layer protocol used for data transmission between web servers and clients.

[0025] gRPC: A high-performance remote procedure call framework developed by Google, based on the HTTP / 2 protocol, supporting features such as bidirectional streaming and multiplexing.

[0026] Kafka: A distributed message queue system.

[0027] Segment object: Represents a single call fragment within a single service instance.

[0028] endpoint: Represents the access point in the service, usually corresponding to a specific URL path or API interface name.

[0029] url: Uniform Resource Locator, used to identify the address of a network resource.

[0030] Exit Span: Represents the exit span for cross-service calls, such as HTTP client calls to downstream services, database driver calls, message queue sending, etc.

[0031] Prometheus: An open-source system monitoring and alerting toolkit that uses a multidimensional data model and the PromQL query language.

[0032] AlertManage: An alarm management component in the Prometheus ecosystem, responsible for handling alarms triggered by Prometheus, and supporting functions such as alarm deduplication, grouping, silencing, suppression, routing, and notification sending.

[0033] HTTP Header: The header field area in the HTTP protocol, used to pass additional information (such as content type, authentication token, custom identifiers, etc.) between the client and the server.

[0034] Dubbo Attachment: The attachment mechanism of the Dubbo framework.

[0035] Mq properties: Message Queue Properties.

[0036] Grafana: An open-source, multi-platform data visualization tool that supports querying data from various data sources such as Prometheus, Elasticsearch, and InfluxDB.

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] In the description of this invention, it should be noted that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0039] like Figure 1 As shown, the present invention provides a method for skywalking-based call chain sampling, comprising the following steps: Full Reporting on the Agent Side: Configure the SkyWalking Agent to disable token bucket sampling and all sampling filtering mechanisms. When a business request enters the service, the Agent generates a TraceId, collects link data, and reports all link data to the Server side in full, ensuring that the Server side obtains complete link data for subsequent processing and metric calculation.

[0040] The purpose of the full reporting of the Agent side in this invention is to solve two key problems caused by Agent header sampling in the prior art: First, it ensures the accuracy of server-side metric calculations. In existing solutions, the agent randomly discards a large amount of link data according to a preset sampling rate. This causes the server to be able to calculate observable metrics such as call count, requests per minute (CPM), average response time, and error rate based on only a portion of the samples, resulting in a systematic deviation from the actual operating status. This invention disables all sampling filtering mechanisms on the agent side, ensuring that the server obtains all raw data. Therefore, all metric calculations are based on complete, real data, fundamentally solving the problem of inaccurate metrics.

[0041] Second, it ensures the complete collection of link data. Existing header sampling strategies determine whether to collect the entire link at the request entry point, but at this time, it is impossible to predict whether errors or timeouts will occur in that link later. If it is determined not to sample, even if a serious failure occurs in that link later, its link data will be directly discarded on the Agent side, and the Server side will be completely unaware of the existence of the failure. This invention, through full reporting, sends all link data to the Server side without discrimination, providing a complete data foundation for subsequent fine-grained sampling and storage, ensuring that no faulty link is missed due to early sampling decisions.

[0042] Asynchronous forwarding: After synchronously receiving the link data reported by the Agent via HTTP or gRPC protocol, the Server immediately forwards the link data to the Kafka message queue for asynchronous consumption without performing any synchronous processing. The Server releases the connection with the Agent after successful data forwarding, achieving asynchronous decoupling of data reception and processing. The Server can release the connection without waiting for processing to complete after receiving data, leveraging Kafka's high throughput to buffer data, effectively reducing the real-time processing pressure on the Server and lowering deployment costs. Simultaneously, it provides the technical foundation for subsequent delayed consumption (second-round consumption), ensuring the integrity of the link data.

[0043] First-round consumption: The first-round consumer subscribes to the link data in Kafka in real time and performs the following operations: Consume the trace data in Kafka, determine the traces that need to be sampled, and record the TraceId; Write the recorded TraceId into the sampled Bloom filter; Then, all the link data is stored in the first storage unit. The first storage unit is a temporary storage unit for Elasticsearch indexes, used to store short-term valid link data. The links that need to be sampled include those marked by the service, those that are faulty or slow, and those marked manually by the user.

[0044] Second round of consumption: The second round of consumers consumes the linked data in Kafka after a delay (e.g., a 30-minute delay), and performs the following operations: Consume the trace data in Kafka and determine whether the TraceId of the trace data exists in the sampling Bloom filter; If it exists, the trace data is stored in the second storage unit, which is a long-term Elasticsearch index used to store trace data that needs to be retained for a long time, and the TraceId is written into the query Bloom filter. If the link does not exist, the data will be discarded and not stored long-term.

[0045] The delayed consumption setting ensures that sampling, storage, and judgment are performed only after all data from upstream and downstream nodes in the link have been reported, thus guaranteeing the integrity of the link.

[0046] The purpose of this invention's delayed consumption mechanism is to solve the problem of inconsistent data arrival times caused by network transmission delays and differences in processing speeds among different service nodes in distributed data transmission. Specifically: A complete distributed call chain typically spans multiple service nodes, with its corresponding multiple Segment objects (one Segment per service instance) being successively reported to the server and written to Kafka. During the first round of consumption, the upstream Segment may have already arrived and been processed, but the downstream Segment may still be in transit or not yet generated. If sampling and long-term storage decisions are made immediately at this point, the following situation may occur: the upstream Segment is determined not to require long-term storage due to the absence of errors or slow calls, but the downstream Segment arrives subsequently and is found to have a fault, causing the entire chain to fail to store data completely due to inconsistent decisions.

[0047] This invention provides a sufficient time window by setting a delayed consumption period (e.g., 30 minutes) to ensure that all segments of the link (from the upstream entry service to the downstream database or cache access) have been fully reported. When delayed consumption is triggered, the second round of consumers processes the segments sequentially, starting from the upstream segment. At this time, the sampling Bloom filter has already recorded all TraceIds (including links determined based on various conditions such as errors, slow calls, and business flags) that were deemed necessary for sampling during the first round of consumption. For any link whose TraceId exists in the sampling Bloom filter, the second round of consumers can write all its segments completely into the second storage unit, thereby ensuring the end-to-end integrity of the long-term storage link and avoiding fragmented storage of the link due to data arrival time sequence issues.

[0048] Query routing: When a user specifies a TraceId in the query criteria, the system performs the following operations: Determine if the TraceId exists in the Bloom filter. If it exists, then query the second storage unit (long-term storage); If it does not exist, then query the first storage unit (temporary storage).

[0049] Finally, the query results are returned.

[0050] By using the above query routing mechanism, retrieval in massive amounts of temporary data is avoided, thus improving query performance.

[0051] In one embodiment, the invention further includes a marking step: The Agent provides a marking function for business links in the link data. Business links refer to service call links with specific business meanings or that require special attention, such as test requests, large transaction requests, VIP user requests, etc.

[0052] The agent identifies business links that require long-term storage by using tagging information and reports the tagging information to the server.

[0053] In the first round of consumption, links that need to be sampled are identified based on the tagging information. Specifically, when the link data contains tagging information, the link is determined to be a link that needs to be sampled.

[0054] In one embodiment, the tagging information includes the following two types: Add the first tag information to the link using the application programming interface provided by the software development kit; By injecting a second tagging information into the protocol layer carrier.

[0055] The first tagging information is the tag identifier added through the SDK API. Specifically, a Java SDK is provided. During operation, the business service determines whether a link is worth storing long-term (even if no errors or slow calls occur) based on business logic (such as order amount exceeding a threshold, VIP user requests, etc.). It then calls the application interface provided by the SDK to add a tag identifier to the current link. This identifier is recognized by the Agent and reported to the Server as marking information.

[0056] The second tagging information is a special identifier injected through the protocol layer carrier. Specifically, for the test link, the second tagging information is manually injected into the protocol layer carrier. The protocol layer carrier includes at least one of HTTP Header, Dubbo Attachment, and MQ properties. After the agent identifies the identifier, it adds the link to the Tag and reports the identifier as tagging information to the server.

[0057] In one embodiment, during the first round of consumption, the links identified as needing sampling include those identified as faulty or slow links. Specifically, when link data is identified as a faulty or slow link, that link is determined to be the link that needs sampling.

[0058] In one specific implementation, faulty / slow link identification includes the following two methods: Method 1: Error status code recognition.

[0059] Identify whether there are error status codes in the link. If any span in the link contains an error status code (such as HTTP 500, database connection error, etc.), then the link is identified as an error link.

[0060] Method 2: Slow link identification.

[0061] This feature identifies whether the time taken for each operation in the process chain exceeds the slow link threshold. The slow link threshold can be dynamically adjusted and modified in real time through the configuration center. Furthermore, differentiated thresholds can be set for different types of components. For example, for database connections, the slow link threshold can be set to >10ms; for cache queries (such as Redis queries), the slow link threshold can be set to >10ms.

[0062] When the time taken for any operation in a link exceeds the slow link threshold of the corresponding component, the link is identified as a slow link.

[0063] In one embodiment, during the first round of consumption, the links identified as requiring sampling include those manually marked by the user. Specifically, when querying on the page, the user manually marks links that need to be stored long-term through the user interface. The system then synchronizes the TraceId of that link to the sampling judgment process, thus identifying that link as one requiring sampling.

[0064] In one embodiment, the data expiration times of the first storage unit and the second storage unit are configured independently. The first storage unit is used to store all link data, and its data expiration time is relatively short; in this embodiment, it is configured to be two days by default. The second storage unit is used to store link data that needs to be retained for a long time after sampling and judgment, and its data expiration time is relatively long; in this embodiment, it is configured to be 30 days by default. Through the above differentiated configuration, both the automatic cleanup of temporary data and the long-term traceability of critical link data are ensured.

[0065] In one embodiment, when the query conditions do not include a specified TraceId, the user needs to manually select a query mode. Query modes include immediate query and delayed query. Immediate query corresponds to querying the first storage unit (temporary storage), used for quickly retrieving recently reported link data. Delayed query corresponds to querying the second storage unit (long-term storage), used for retrieving archived long-term storage link data; delayed query is the default query mode. Users can select the appropriate query mode according to their actual needs, and the system routes the query to the corresponding storage unit based on the user's selected mode.

[0066] In one embodiment, the present invention further includes a step of separating index calculation and storage: Metric Calculation: The server performs metric calculations on the link data. Metric calculations include: aggregating the link data by time bucket (e.g., 1 minute), such as calculating the number of calls per minute, average response time, and other metrics. The aggregation results are first written to the primary storage cluster.

[0067] Delayed Backtesting: Due to network latency, asynchronous processing, and other reasons, some indicator data may only arrive after the time bucket (1 minute / 1 hour / 1 day) has closed. To ensure data accuracy, the system periodically backtests indicator data before a preset time window. In this embodiment, the scheduled task executes once every minute, backtesting indicator data from two minutes ago. The preset time window (2 minutes) is longer than the time bucket (1 minute), ensuring that the backtested indicator data has stabilized and is no longer changing.

[0068] Metric storage isolation: Stable metric data obtained from backtracking is written to a separate metric storage cluster (metric Elasticsearch cluster). The metric storage cluster is physically isolated from the first and second storage units of the storage link data to avoid write performance conflicts between metric data and link data, and allows for targeted operation and maintenance optimization of the metric cluster.

[0069] In one embodiment, the present invention further includes an alarm step: Data formatting and listening: During the first round of consumption, all link data reported in full is obtained, the link data is formatted into Segment objects and sent to the listeners (including Segment listeners, Span listeners, etc.).

[0070] Parsing and Logging: The listener parses the Segment object and records the following key information contained within it: whether the operation encountered an error, the request object, the endpoint (service endpoint), and the URL (request path). The recorded operation error status is used to determine if the link data contains errors; if so, a link error object is constructed. The recorded request object, service endpoint, and request path serve as attribute information for the link error object.

[0071] Middleware error identification: Iterate through the Exit Span in the Segment object. The Exit Span represents the exit span for cross-service calls, such as HTTP client calls, database driver calls, message queue sending, etc. Determine whether an error occurred in the middleware (such as database, message queue, registry center) called by the Exit Span. If an error occurred, construct a middleware error object. The middleware error object contains the following information: middleware type (such as MQ, DB, registry center), address, service name, Topic (for MQ), time, error message, and TraceId.

[0072] Link error identification and deduplication: Determine if the link data contains errors or if an Exit Span error has occurred. If an error occurs, deduplication is performed using the TraceId as the unique key to avoid multiple error Spans from being recorded repeatedly for the same link, and a link error object (errorTrace) is constructed. The link error object contains the following information: error code type, error code, error message, service name, instance name, interface name, and TraceId.

[0073] Convert to timing metrics and send: Convert the constructed middleware error objects and link error objects into timing metrics and send them to Prometheus via the Remote Write protocol.

[0074] Unified Alert Configuration: Alerts are triggered based on reported time-series metrics configured using Prometheus' AlertManager component. For example, configuring an alert rule `count by (component_name) (middleWare_metric{middle_type='Nacos',service=~"prd-.*"}) > 500` will immediately trigger an alert when multiple services request the same middleware and experience a large number of errors. Simultaneously, monitoring dashboards can be configured using Grafana and Prometheus to visualize metric data, and AlertManager's capabilities can be leveraged to complete the entire alert handling process, including alert aggregation, escalation, distribution, notification, and processing.

[0075] Through the above alarm steps, this invention integrates SkyWalking link data with the Prometheus monitoring system, enabling link-driven real-time alarms and significantly shortening fault detection time. At the same time, it uses TraceId for deduplication to avoid alarm storms and unifies the link tracing and infrastructure monitoring and alarm systems, reducing operational complexity.

[0076] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention used to illustrate the technical solutions of the present invention, and are not intended to limit the invention, nor are they intended to limit the patent scope of the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention. That is to say, any changes or refinements made to the main design concept and spirit of the present invention that are not of substantial significance, but whose technical problems are still consistent with the present invention, should be included within the protection scope of the present invention. In addition, the direct or indirect application of the technical solutions of the present invention to other related technical fields are similarly included within the patent protection scope of the present invention.

Claims

1. A method for call link sampling based on skywalking, characterized in that, Includes the following steps: The agent side disables the sampling and filtering mechanism and reports all link data to the server side in full. After receiving the link data synchronously via HTTP and gRPC, the server forwards it to Kafka for asynchronous consumption. First round of consumption: Consume the link data in Kafka, determine the link that needs to be sampled and record the TraceId, write the TraceId into the sampling Bloom filter, and then store the link data in full into the first storage unit; Second round of consumption: Delayed consumption of link data in Kafka, determining whether the TraceId of the link data exists in the sampling Bloom filter. If it exists, the link data is stored in the second storage unit, and the TraceId is written into the query Bloom filter. When the query conditions include a specified TraceId, check if the TraceId exists in the query Bloom filter. If it exists, query the second storage unit; otherwise, query the first storage unit.

2. The method according to claim 1, characterized in that, The process of uploading all link data to the server includes the following steps: The Agent provides a marking function for business links in the link data, identifies business links that need to be stored for a long time through the marking information, and reports the marking information to the Server. In the first round of consumption, the links that need to be sampled include those identified based on the tagging information.

3. The method according to claim 2, characterized in that, The tagging information includes: Add the first tag information to the link using the application programming interface provided by the software development kit; By injecting a second tagging information into the protocol layer carrier.

4. The method according to claim 1, characterized in that, In the first round of consumption, the links that need to be sampled include those identified based on faulty or slow links.

5. The method according to claim 4, characterized in that, Error and slow link identification includes: Identify whether there are error status codes in the link; It identifies whether the time taken for each operation in the link exceeds the slow link threshold. The slow link threshold can be dynamically adjusted and differentiated thresholds can be set for different types of components. Different types of components include at least one of database connection and cache query.

6. The method according to claim 1, characterized in that, In the first round of consumption, the links that need to be sampled include those identified based on user-generated markings.

7. The method according to claim 1, characterized in that, The data expiration times of the first storage unit and the second storage unit are configured independently, with the data expiration time of the first storage unit being shorter than that of the second storage unit.

8. The method according to claim 1, characterized in that, When the query conditions do not include a specified TraceId, the first storage unit or the second storage unit is queried according to the query mode selected by the user. The query modes include immediate query and delayed query. Immediate query corresponds to querying the first storage unit, and delayed query corresponds to querying the second storage unit.

9. The method according to claim 1, characterized in that, It also includes the following steps: The server performs metric calculations on the link data and writes the calculated metric data to the metric storage cluster separately. The metric storage cluster is physically isolated from the first and second storage units that store the link data. Metric calculation includes: aggregating and calculating link data by time bucket; Periodically check the indicator data before a preset time window. The preset time window is longer than the time bucket, and write the retrieved indicator data to the indicator storage cluster.

10. The method according to claim 1, characterized in that, It also includes the following steps: During the first round of consumption, all link data reported in full is obtained, the link data is formatted into Segment objects and sent to the listener; The listener parses the Segment object, iterates through the Exit Span in the Segment object, and determines whether the middleware called by the Exit Span has encountered an error. If an error has occurred, a middleware error object is constructed. Determine whether the link data contains errors or whether any Exit Span has an error. If an error occurs, use TraceId as the unique key to deduplicate and construct a link error object. The middleware error objects and link error objects are converted into time-series metrics and sent to Prometheus. The Prometheus AlertManager component then uses these metrics to configure alarm rules and trigger alarms.

Images