Telemetry metric real-time augmentation processing and apparatus

Abstract

The telemetry index real-time enhancement processing method and device provided in the application can complete most metadata query requests in microseconds through step-by-step query of a two-level cache architecture, thereby significantly reducing overall processing delay. Meanwhile, for a to-be-queried item that is not hit in the two-level caches, the field value is divided into batches according to a preset block size, the metadata API is called in batches for batch query, and the result is backfilled to the two-level caches, thereby greatly reducing the number of calls to external services and providing a cache hit basis for subsequent query of the same field, thereby forming a benign closed loop of cache query and backfill. In addition, the remaining survival time of a cache entry is detected when the second-level distributed cache is queried, and updating is triggered when the remaining survival time is less than a refresh threshold, thereby effectively avoiding query delay jitter after cache invalidation and further guaranteeing the stability of system processing performance, so that efficient and low-delay real-time enhancement processing can be achieved.

Description

Technical Field

[0001] This application relates to the field of big data processing technology, and more specifically, to a method and apparatus for real-time enhancement processing of telemetry indicators. Background Technology

[0002] In the construction of observability for large-scale distributed systems, raw telemetry data often lacks business context information, requiring real-time enhancement through queries to external metadata services. Existing solutions employ synchronous queries, calling the metadata API once for each data point encountered. In scenarios with millions of data points per second, frequent API calls lead to severe cumulative processing latency and place enormous pressure on the metadata service. To alleviate this problem, some solutions introduce local caching to store query results; however, all cached entries use a fixed TTL and are passively deleted upon expiration. Simultaneous invalidation of a large number of caches can easily trigger cache avalanches. Furthermore, for keys not present in the cache, there is a lack of effective empty result caching and fault tolerance mechanisms, causing query requests to repeatedly penetrate to the backend API, further exacerbating performance degradation. In addition, existing solutions lack batch merging mechanisms for query requests for multiple data points, still processing them field-by-field, failing to utilize the batch query capabilities of the metadata API, resulting in network round-trip overhead increasing linearly with data volume. Therefore, there is an urgent need for a real-time telemetry indicator enhancement method that can reduce API call frequency, decrease query latency, and avoid the impact of cache anomalies on system stability. Summary of the Invention

[0003] To overcome the aforementioned shortcomings in the prior art, the purpose of this application is to provide a real-time enhancement processing method for telemetry indicators, comprising the following steps: Obtain a list of query items, which includes multiple query items, where each query item is a combination of a telemetry indicator data point and its corresponding metadata fields to be queried; Query the first-level local cache to obtain the first query result corresponding to the query item in the query item list that can be hit in the first-level local cache; For the query item that is not found in the first-level local cache, the second-level distributed cache is queried to obtain the second query result, and the remaining lifetime of the cache entry is checked. If the remaining lifetime is less than a preset refresh threshold, the cache entry is updated. For the query item that is not found in either the first-level local cache or the second-level distributed cache, the corresponding field value is called in batches according to the preset block size to perform batch query on the metadata API, and the obtained batch query results are written to the first-level local cache and the second-level distributed cache. Based on the first query result, the second query result, and the batch query result, the telemetry indicator data points in the list of items to be queried are enhanced, and the enhanced indicator data is output.

[0004] In some possible implementations, the step of obtaining a list of query items that includes multiple query items includes: Receive batches of raw indicator data, wherein the batches of raw indicator data contain at least one raw telemetry indicator data point. Perform at least one of the following data quality control processes on the original telemetry index data points: data deduplication, consistent hash sampling, and cardinality control, to obtain a list of data points to be enhanced. Iterate through all the telemetry index data points in the list of data points to be enhanced, extract the metadata fields to be queried, combine each telemetry index data point and its corresponding metadata fields to be queried into the query item, and perform deduplication and merging processing on all query items to generate the query item list.

[0005] In some possible implementations, the step of updating the cache entry includes: If the current second query result is returned, an asynchronous task is started to attempt to acquire the distributed lock corresponding to the cached entry; If the distributed lock is successfully acquired, a cache update operation is performed, and the distributed lock is released after the update is completed. If the distributed lock cannot be acquired, the cache update operation will not be performed.

[0006] In some possible implementations, the step of performing the cache update operation includes: During the cache update operation, the cache entry is checked again before updating the cache to see if it has been updated by other nodes. When a cache update operation fails, the existing old data in the cache entry is retained.

[0007] In some possible implementations, the step of updating the cache entry further includes: A first time-to-live (TTL) is set for cached entries obtained through successful queries, and a second TTL is set for cached entries obtained through unsuccessful queries; the first TTL is longer than the second TTL.

[0008] In some possible implementations, the step of setting a first time-to-live (TTL) for cached entries obtained through successful queries and setting a second TTL for cached entries obtained through failed queries includes: For a cached entry that is successfully retrieved, a time period is randomly determined within the first time range as the first lifetime. For cached entries obtained after a failed query, a random duration is determined within the second duration range as the second lifetime.

[0009] In some possible implementations, the step of batch querying the corresponding field values ​​of the item to be queried in both the first-level local cache and the second-level distributed cache according to a preset block size by calling the metadata API in batches includes: The field values ​​corresponding to the query items that are not found in either the first-level local cache or the second-level distributed cache are aggregated into a query field set. The set of fields to be queried is divided into at least one subset of fields according to a preset block size; For each field subset, a batch query is performed by calling the metadata API, with each field subset corresponding to one API call. The query results obtained from each batch query are aggregated and written to the first-level local cache and the second-level distributed cache.

[0010] In some possible implementations, the step of enhancing the telemetry index data points in the list of items to be queried includes: The number of worker threads in the worker thread pool is adaptively determined based on the central processing unit resources of the operating environment, and the telemetry index data points in the list of items to be queried are enhanced in parallel through the worker thread pool.

[0011] In some possible implementations, the step of enhancing the telemetry index data points in the list of items to be queried includes: The query can perform at least one of the following calculations based on a single indicator in the telemetry data points in the query list: summation, average, count, minimum, maximum, 95th percentile, and 99th percentile; and / or at least one of the following calculations based on multiple indicators in the telemetry data points in the query list: summation, difference, multiplication, and division.

[0012] Another objective of this application is to provide a real-time telemetry indicator enhancement processing device, the real-time telemetry indicator enhancement processing device comprising: The data acquisition module is used to acquire a list of query items including multiple query items, where each query item is a combination of a telemetry indicator data point and its corresponding metadata fields to be queried; The first query module is used to query the first-level local cache and obtain the first query result corresponding to the query item in the query item list that can be hit in the first-level local cache; The second query module is used to query the second-level distributed cache for the query item that is not hit in the first-level local cache, obtain the second query result, and detect the remaining lifetime of the cache entry. If the remaining lifetime is less than a preset refresh threshold, the cache entry is updated. The third query module is used to perform batch queries on the corresponding field values ​​of the query item that is not found in the first-level local cache and the second-level distributed cache by calling the metadata API in batches according to the preset block size, and write the obtained batch query results into the first-level local cache and the second-level distributed cache. The data enhancement module is used to enhance the telemetry indicator data points in the list of items to be queried based on the first query result, the second query result, and the batch query result, and output the enhanced indicator data.

[0013] Another objective of this application is to provide an electronic device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by the processor, they implement the real-time telemetry indicator enhancement processing method provided in this application.

[0014] Another objective of this application is to provide a machine-readable storage medium storing machine-executable instructions, which, when executed by at least one processor, implement the real-time telemetry indicator enhancement processing method provided in this application.

[0015] Compared with the prior art, this application has the following beneficial effects: The telemetry indicator real-time enhancement processing method and apparatus provided in this application, through a two-level caching architecture and step-by-step querying, enables most metadata query requests to be completed in microseconds, significantly reducing overall processing latency. Simultaneously, for query items that are not found in either level of cache, the metadata API is called in batches according to a preset block size for querying field values, and the results are then backfilled into the two-level cache. This significantly reduces the number of calls to external services and provides a cache hit basis for subsequent queries of the same field, thus forming a virtuous cycle of cache query and backfilling. Furthermore, when querying the second-level distributed cache, the remaining lifetime of cache entries is checked, and an update is triggered when it is less than the refresh threshold. This effectively avoids query latency jitter after cache expiration, further ensuring the stability of system processing performance and enabling efficient, low-latency real-time enhancement processing. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the steps of the real-time enhancement processing method for telemetry indicators provided in this application embodiment; Figure 2 A schematic diagram of an electronic device provided in an embodiment of this application; Figure 3 A schematic diagram of the functional modules of the real-time telemetry index enhancement processing device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] In the description of this application, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application. In addition, the terms "first," "second," and "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] Furthermore, terms such as "horizontal," "vertical," and "sag" do not imply that components must be absolutely horizontal or suspended, but rather that they can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0023] In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set up," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0024] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a real-time enhancement processing method for telemetry indicators provided in this embodiment. The steps of the method are described in detail below.

[0025] Step S110: Obtain a list of items to be queried, which includes multiple items to be queried. Each item to be queried is a combination of a telemetry indicator data point and its corresponding metadata fields to be queried.

[0026] In this embodiment, telemetry index data points refer to the MetricDataPoint defined in the OpenTelemetry framework, which represents the specific values ​​and label set of the system's operating status at a specific point in time. Metadata fields refer to the query keys corresponding to the business context information that needs to be supplemented for the telemetry index data points, such as the service line identifier, data center identifier, cluster identifier, and responsible person identifier. The query item is a data structure formed by pairing and combining the telemetry index data points with the metadata fields to be queried. Each query item clearly indicates which data point is being queried and which field's value needs to be retrieved.

[0027] In step S110, the enhanced processing node obtains a list of query items including multiple query items, that is, obtains a set containing several query items, where each query item in the set points to a telemetry index data point and its corresponding query metadata field.

[0028] During the execution of the above steps, the generation of the query item list can be constructed based on the batch of raw indicator data received by the enhancement processing node. Specifically, when the enhancement processing node receives a batch of data containing at least one raw telemetry indicator data point, the enhancement processing node iterates through each raw telemetry indicator data point in the batch, and generates a query item for each metadata field that needs to be queried contained in each data point, thereby forming the query item list.

[0029] Step S120: Query the first-level local cache and obtain the first query result corresponding to the query item in the query item list that can be hit in the first-level local cache.

[0030] In this embodiment, the first-level local cache refers to the local memory cache space running within the current enhanced processing node process. This cache stores previously queried metadata field values, and its access latency is in the microsecond range, without the need for network transmission.

[0031] In step S120, for each query item in the query item list obtained in step S110, the enhancement processing node first searches in the first-level local cache using the metadata field of the query item as the cache key. If a cache entry matching the metadata field is found in the first-level local cache, the enhancement processing node determines that the query item has hit the first-level local cache and reads the corresponding metadata field value from the cache entry as the first query result. For a query item that has hit the first-level local cache, the first query result is the query return value of the metadata field corresponding to the query item. In practical applications, since the first-level local cache is located in the memory of the enhancement processing node process, the read speed is extremely fast, and most hot metadata query requests can be satisfied at this level, thereby avoiding the latency overhead caused by subsequent higher-level cache queries or external API calls.

[0032] Step S130: For the query item that is not found in the first-level local cache, query the second-level distributed cache, obtain the second query result, and check the remaining lifetime of the cache entry. If the remaining lifetime is less than the preset refresh threshold, update the cache entry.

[0033] In this embodiment, the second-level distributed cache refers to a distributed cache service that is independent of the current enhanced processing node and shared by multiple enhanced processing nodes.

[0034] It should be noted that the second-level distributed cache can be implemented using distributed caching middleware such as Redis. Access to this cache requires network transmission, resulting in a latency in the millisecond range. For a query item that fails to find a match in the first-level local cache in step S120, the enhancement processing node uses the metadata field of the query item as the cache key to search in the second-level distributed cache. If a cache entry matching the metadata field is found in the second-level distributed cache, the enhancement processing node determines that the query item has been found in the second-level distributed cache and reads the corresponding metadata field value from that cache entry as the second query result.

[0035] This can be understood as the second-level distributed cache serving as a supplement and extension to the first-level local cache, providing broader data sharing capabilities when the first-level local cache misses.

[0036] During the execution of the above steps, after querying the second-level distributed cache and obtaining the second query result, the enhancement processing node also needs to check the remaining lifetime of the cache entry. The remaining lifetime here refers to the length of time remaining between the current moment and its preset expiration time. The preset refresh threshold is a pre-configured time value used to determine whether the cache entry needs to be actively refreshed before expiration. If the enhancement processing node detects that the remaining lifetime of the cache entry is less than the preset refresh threshold, it triggers an update operation for that cache entry.

[0037] It should be understood that this update operation is performed asynchronously while returning the current query results, and does not block the return of the current query request.

[0038] Thus, by querying the second-level distributed cache and triggering the active refresh mechanism, the enhanced processing node achieves cross-node cache data sharing and active maintenance of cache entry timeliness.

[0039] In this embodiment, the first-level local cache uses an in-process cache space with an access latency in the microsecond range, suitable for storing frequently accessed metadata. The second-level distributed cache uses the Redis distributed caching service with an access latency in the millisecond range, suitable for sharing metadata cache among multiple enhancement processing nodes. When an enhancement processing node queries, it first queries the first-level local cache. Only if the first-level local cache is not found will it query the second-level distributed cache. Only if both caches are not found will the metadata API be called. This ensures that most query requests can be responded to by the first-level local cache with the lowest latency. The refresh threshold for the active refresh mechanism is set to five minutes by default, meaning that the enhancement processing node triggers an asynchronous refresh when it detects that the remaining lifespan of a cached entry is less than five minutes. Asynchronous refresh does not block the current query request; the current request directly returns the existing value in the cache, and the refresh operation is executed asynchronously in the background. This avoids the request blocking and increased latency caused by synchronous refresh.

[0040] Step S140: For the query item that is not found in both the first-level local cache and the second-level distributed cache, the corresponding field value is called in batches according to the preset block size to perform batch query on the metadata API, and the obtained batch query results are written to the first-level local cache and the second-level distributed cache.

[0041] In this embodiment, for query items that fail to hit the first-level local cache in step S120 and also fail to hit the second-level distributed cache in step S130, the metadata field values ​​corresponding to these query items do not exist in either of the current two-level caches and need to be obtained from an external metadata service. At this time, the enhanced processing node summarizes the field values ​​corresponding to these query items and processes them in batches according to a preset block size.

[0042] It should be noted that the preset chunk size refers to the maximum number of field values ​​included in each call to the metadata API. This value can be configured according to the single query load capacity of the metadata API. For each batch of field values, the enhancement processing node calls the metadata API once for a batch query. This API call returns the metadata corresponding to all field values ​​in the batch at once. The enhancement processing node then aggregates the results obtained from each batch of queries and writes them uniformly into the first-level local cache and the second-level distributed cache, so that subsequent query requests for the same metadata field can directly hit the cache.

[0043] This can be understood as follows: step S140 significantly reduces the number of network round trips and the frequency of API calls by merging multiple independent field query requests into a few batch API calls. Thus, the enhanced processing node not only retrieves the metadata of the query items that are not found in either of the two-level caches, but also builds a cached data foundation for subsequent queries by writing the query results back to both-level caches.

[0044] Step S150: Based on the first query result, the second query result, and the batch query result, perform enhancement processing on the telemetry indicator data points in the query item list and output the enhanced indicator data.

[0045] In this embodiment, after the processing of steps S120, S130 and S140, each item in the list of items to be queried has obtained the corresponding metadata field value. These metadata field values ​​are respectively from the first-level local cache (first query result), the second-level distributed cache (second query result) or the batch query of the metadata API (batch query result).

[0046] In step S150, the enhanced processing node fills all the above query results into the corresponding telemetry indicator data points according to the correspondence between the query items and the telemetry indicator data points, thus supplementing the original telemetry indicator data with complete business context information.

[0047] During the execution of the above steps, the enhancement processing refers to the enhancement processing node attaching the metadata field values ​​obtained from the query to the corresponding telemetry indicator data points in the form of tags or attributes, so that the telemetry indicator data points that originally only contained numerical values ​​and limited tags can obtain rich business context information.

[0048] After enhancement processing, the enhancement processing node outputs enhanced indicator data, which can be sent to downstream monitoring and alarm systems, time series databases, or other observability platforms for subsequent monitoring display, alarm judgment, and problem localization.

[0049] Based on the above design, in the solution provided in this embodiment, most metadata query requests can be completed in microseconds through the step-by-step query of the two-level caching architecture, significantly reducing the overall processing latency. At the same time, for query items that are not found in both levels of cache, the metadata API is called in batches according to the preset block size for batch querying of field values ​​and the results are backfilled into the two-level cache. This not only greatly reduces the number of calls to external services, but also provides a cache hit basis for subsequent queries of the same field, thus forming a virtuous cycle of cache query and backfill. In addition, when querying the second-level distributed cache, the remaining lifetime of the cache entry is detected and an update is triggered when it is less than the refresh threshold. This effectively avoids query latency jitter after cache expiration, further ensuring the stability of system processing performance and enabling efficient, low-latency real-time enhancement processing.

[0050] In some possible implementations, step S110 may include the following sub-steps.

[0051] Step S111: Receive a batch of raw indicator data, which contains at least one raw telemetry indicator data point.

[0052] A raw telemetry data batch refers to a data set received by the augmentation processing node from the data acquisition end (such as the receiving module of OpenTelemetryCollector), containing at least one raw telemetry data point. Each raw telemetry data point is a set of indicator data containing numerical values ​​and labels, collected at a specific time point.

[0053] Step S112: Perform at least one of the following data quality control processes on the original telemetry index data points: data deduplication, consistent hash sampling, and cardinality control, to obtain a list of data points to be enhanced.

[0054] During the above steps, data deduplication refers to the enhancement processing node generating a data fingerprint for each original telemetry indicator data point based on the SHA256 hash algorithm, according to the indicator name, sorting attribute, and timestamp. Within a preset time window, it checks for duplicate data fingerprints; if found, the current original telemetry indicator data point is considered duplicate data and is filtered out. Consistent hash sampling refers to the enhancement processing node using the SHA256 hash algorithm to calculate the hash value of the data key, comparing this hash value with a preset sampling threshold, and deciding whether to retain or discard the current original telemetry indicator data point based on the comparison result, ensuring that identical data keys always receive the same sampling decision. Cardinality control refers to the enhancement processing node tracking the number of current time series in real time. When the number of time series reaches a preset upper limit threshold, it performs discard or aggregation operations on newly generated time series according to the configured strategy. After the above data quality control processing, the enhancement processing node obtains a list of data points to be enhanced.

[0055] Step S113: Traverse all telemetry index data points in the list of data points to be enhanced, extract the metadata fields to be queried, combine each telemetry index data point and its corresponding metadata fields to be queried into a query item, and perform deduplication and merging on all query items to generate a query item list.

[0056] Specifically, the enhancement processing node iterates through each telemetry indicator data point in the list of data points to be enhanced, generating a query item for each data point's metadata fields that need to be queried. After generating the query items, the enhancement processing node performs deduplication and merging on all query items. That is, if multiple telemetry indicator data points need to query the same metadata field, these query items are merged into one to avoid duplicate queries. Through deduplication and merging, the metadata fields of each query item in the generated list of query items are unique, thereby reducing the number of subsequent cache queries and API calls.

[0057] Based on the above design, the data quality control process in steps S111 to S113 ensures the data quality entering the subsequent processing flow, while the deduplication and merging process optimizes query efficiency.

[0058] In this embodiment, data deduplication is based on generating data fingerprints using the SHA256 hash algorithm. The data fingerprint is obtained by combining the indicator name, sorting attribute, and timestamp, and then calculating the result using SHA256 hash. The enhancement processing node checks for duplicate data fingerprints within a preset time window. If a duplicate fingerprint is found, the current data point is determined to be duplicate data and filtered. In consistent hash sampling, the enhancement processing node uses the SHA256 hash algorithm to calculate the hash value of the data key. The hash value is compared with a sampling threshold to determine whether to retain the data point, ensuring that identical data keys always receive the same sampling decision, thereby guaranteeing the consistency and reproducibility of the sampling results. In cardinality control, the enhancement processing node tracks the number of time series in real time. When the number of time series reaches a preset upper limit threshold, newly generated time series are discarded or aggregated according to the configured strategy to prevent time series explosion caused by dynamic tags.

[0059] In some possible implementations, step S130 may include the following sub-steps.

[0060] Step S131: If the current second query result is returned, start an asynchronous task to attempt to acquire the distributed lock corresponding to the cached entry.

[0061] Specifically, after the enhancement processing node hits the second-level distributed cache, it returns the second query result to the current query request. This return operation is decoupled from the subsequent cache update operation. Simultaneously with returning the current query result, the enhancement processing node starts an asynchronous task that attempts to acquire the distributed lock corresponding to the cache entry. It should be noted that a distributed lock is a mutual exclusion mechanism used in distributed systems to coordinate access to shared resources by multiple nodes. In this embodiment, the distributed lock ensures that only one enhancement processing node performs an update operation on the same cache entry at any given time.

[0062] In step S132, if the distributed lock is successfully acquired, the cache update operation is performed, and the distributed lock is released after the update is completed.

[0063] Specifically, once the asynchronous task initiated by the enhancement processing node successfully acquires the distributed lock corresponding to the cache entry, the asynchronous task gains permission to perform a cache update operation. At this point, the enhancement processing node performs the cache update operation, that is, it retrieves the latest value of the cache entry from the metadata service and writes it to the cache. After the update operation is completed, the enhancement processing node releases the distributed lock, allowing other nodes waiting for the lock to continue performing update operations.

[0064] In step S133, if the distributed lock cannot be acquired, the cache update operation will not be performed.

[0065] Specifically, when an asynchronous task initiated by an enhancement processing node fails to acquire the distributed lock corresponding to a cache entry, it indicates that the cache entry is being updated by other enhancement processing nodes. The current asynchronous task does not need to repeat the update, so no cache update operation is performed.

[0066] During the execution of the above steps, the introduction of distributed locks effectively prevents duplicate queries and resource waste caused by multiple enhanced processing nodes refreshing the same cache entry at the same time, and also avoids the impact of concurrent pressure on the metadata service.

[0067] In some possible implementations, during the cache update operation in step S132, the cache entry is checked again to see if it has been updated by other nodes before updating the cache. If the cache update operation fails, the old data already in the cache entry is retained.

[0068] Specifically, after the asynchronous task successfully acquires the distributed lock and begins executing the cache update operation, there may be a time lag between acquiring the lock and actually executing the write operation. During this time, other nodes may have already completed updating the cache entry. Therefore, before executing the actual cache write operation, the enhancement processing node checks again whether the cache entry has already been updated by other nodes. If it has been updated, the current asynchronous task does not need to repeat the write operation; it can skip the write step and release the distributed lock. When the cache update operation fails, the enhancement processing node retains the existing old data in the cache entry. During the cache update operation, updates may fail due to reasons such as unavailability of the metadata service or network timeout. When the update operation fails, the enhancement processing node does not delete or overwrite the existing old data in the cache; instead, it retains the existing old data in the cache entry as a fallback solution.

[0069] In some possible implementations, in step S132, a first time-to-live (TTL) is set for cached entries obtained through successful queries, and a second TTL is set for cached entries obtained through unsuccessful queries, wherein the first TTL is greater than the second TTL.

[0070] Specifically, a successful query means that the cached entry stores a valid metadata query result, indicating that the enhancement processing node has successfully retrieved data from the metadata service. A query failure means that the enhancement processing node failed to retrieve a valid metadata query result due to the unavailability of the metadata service, the non-existent query key, or other abnormal reasons. For cached entries obtained from successful queries, the enhancement processing node sets a relatively long first lifetime, because these cached entries store valid data and can provide services for subsequent queries for a longer period. For cached entries obtained from failed queries, the enhancement processing node sets a relatively short second lifetime, because these cached entries store failure status information and need to expire quickly so that data can be retrieved from the metadata service again in subsequent queries.

[0071] It should be understood that by setting different lifetimes for successful and failed queries, both a longer cache duration for valid data is ensured to improve cache hit rate, while availability issues caused by prolonged caching of failed states are avoided.

[0072] In some possible implementations, for cached entries obtained through successful queries, a first time limit is randomly determined within a first time range. For cached entries obtained through failed queries, a second time limit is randomly determined within a second time range.

[0073] Specifically, the first time range is a preset time interval, and the enhancement processing node randomly selects a time range within this interval as the actual lifespan of the cached entry. The second time range is another preset time interval, and the enhancement processing node randomly selects a time range within this interval as the actual lifespan of the cached entry that failed to query.

[0074] During the execution of the above steps, the enhanced processing node randomly determines the lifetime within a preset time range, ensuring that the actual expiration times of different cache entries are staggered, thus avoiding the cache avalanche problem caused by a large number of cache entries expiring simultaneously. In practical applications, even if a large number of cache entries have the same basic lifetime setting, due to the random jitter mechanism, the actual expiration times of these cache entries will be dispersed within a time interval, thereby effectively reducing the risk of cache avalanche.

[0075] In some possible implementations, step S140 may include the following sub-steps.

[0076] Step S141: The field values ​​corresponding to the query items that were not found in either the first-level local cache or the second-level distributed cache are aggregated into a set of query fields.

[0077] After the two-level cache query in steps S120 and S130, some query items were not found in both caches. The enhanced processing node collected and summarized the field values ​​corresponding to these query items to form a set of query fields, which contains all the field values ​​that need to be obtained from the metadata API.

[0078] Step S142: Divide the set of fields to be queried into at least one subset of fields according to the preset block size.

[0079] The preset chunk size refers to the maximum number of field values ​​that can be queried each time the metadata API is called. This value can be configured according to the single query load capacity of the metadata API and network transmission limitations. The enhanced processing node divides the set of fields to be queried according to this chunk size. If the number of field values ​​in the set of fields to be queried is less than or equal to the chunk size, only one field subset is formed; if it is greater than the chunk size, multiple field subsets are formed.

[0080] Step S143: For each field subset, call the metadata API to perform a batch query, with each field subset corresponding to one API call.

[0081] For each field subset divided in step S142, the enhanced processing node initiates a call to the metadata API. This call sends all field values ​​in the field subset as query parameters to the metadata service, and the metadata service returns the metadata corresponding to all field values ​​in the field subset.

[0082] Step S144: After summarizing the query results obtained from each batch query, write them uniformly into the first-level local cache and the second-level distributed cache.

[0083] The enhanced processing node aggregates and merges the batch query results of all field subsets to form a complete batch query result, and then writes the result to the first-level local cache and the second-level distributed cache.

[0084] During the execution of the above steps, by processing the set of fields to be queried in batches according to the block size, the batch query capability of the metadata API is fully utilized, while avoiding timeout or denial-of-service issues caused by excessive load on a single API call. After the enhanced processing node writes the query results to a two-level cache, subsequent query requests for the same metadata field can be directly hit in the cache without having to call the metadata API again.

[0085] In this embodiment, the batch collection mechanism traverses the Resource, Scope, and DataPoint layers through a three-level nested loop to extract the field values ​​to be queried from each layer, forming a batch query list after deduplication. The Resource, Scope, and DataPoint layers are the three hierarchical structures of OpenTelemetry metric data. The Resource layer describes the resource information to which the metric belongs, the Scope layer describes the Instrumentation Scope information to which the metric belongs, and the DataPoint layer describes the specific value and label information of the metric at a specific time point. The enhanced processing node, through this three-level nested loop traversal, can comprehensively extract the metadata fields to be queried from each layer. In intelligent batch querying, the enhanced processing node first performs a batch query from the cache, collecting the missing values, and then calls the metadata API in batches according to the block size. Through the above batch query optimization technology, the number of API calls can be reduced by more than 95%.

[0086] In some possible implementations, in step S150, the number of worker threads in the worker thread pool can be adaptively determined according to the central processing unit resources of the operating environment, and the telemetry index data points in the list of query items can be enhanced in parallel through the worker thread pool.

[0087] In this embodiment, the worker pool refers to a pre-created set of threads (workers) used to execute augmented processing tasks in parallel. This worker pool runs inside the augmented processing node. During the execution of the above steps, the augmented processing node adaptively calculates the number of worker threads in the worker pool based on the CPU resources of the runtime environment.

[0088] Specifically, the enhanced processing node first reads the CPU quota from the container cgroup configuration. If the container CPU quota can be read, the number of worker threads is calculated based on the quota. If the container CPU quota cannot be read, the number of system CPU cores is read, and the number of worker threads is calculated based on the number of system CPU cores.

[0089] In one example, the final number of worker threads was set to four times the number of CPU cores. By adaptively determining the number of worker threads, the concurrency of the worker thread pool is matched with the CPU resources of the runtime environment. This avoids both CPU resource idleness due to too few threads and context switching overhead due to too many threads. After determining the number of worker threads, the enhancement processing node performs enhancement processing on the telemetry indicator data points in the query item list in parallel through the worker thread pool. That is, multiple worker threads simultaneously retrieve query items from the task queue and independently perform enhancement operations such as label filling and numerical transformation, thereby making full use of the computing power of the multi-core CPU and significantly improving the throughput of enhancement processing.

[0090] In some possible implementations, in step S150, at least one of the following calculations can be performed based on a single indicator in the telemetry indicator data points in the query item list: summation, average, count, minimum, maximum, 95th percentile, and 99th percentile; and / or at least one of the following calculations can be performed based on multiple indicators in the telemetry indicator data points in the query item list: summation, difference, multiplication, and division.

[0091] In this embodiment, the enhancement processing node supports a total of 16 operations, including label enhancement, numerical transformation, label operations, single-indicator aggregation, and multi-indicator calculation. Label enhancement operations include four types: `set_label`, `set_label_force`, `fill`, and `replace`, used to set, force set, fill, and replace labels for telemetry indicator data points. Numerical transformation operations include three types: `set_value_numeric`, `inc_value`, and `mul_value`, used to set, increase, and multiply values. Label operations include two types: `rename_label` and `merge_labels`, used to rename and merge labels. Single-indicator aggregation operations include seven types: `sum`, `avg`, `count`, `min`, `max`, `p95`, and `p99`, used to calculate summation, average, count, minimum, maximum, 95th percentile, and 99th percentile for a single indicator. Multi-indicator calculation operations include four types: `sum`, `diff`, `mul`, and `div`, used to calculate summation, difference, multiplication, and division of multiple indicators, thereby generating derived indicators such as error rate, success rate, and resource utilization. The above operating system provides users with rich data processing capabilities, which can meet diverse telemetry indicator enhancement needs.

[0092] Please refer to Figure 2 , Figure 2 This embodiment provides a block diagram of an electronic device 100. The electronic device 100 includes a real-time telemetry indicator enhancement processing device 110, a machine-readable storage medium 120, and a processor 130. This electronic device 100 can be the enhancement processing node shown in this embodiment.

[0093] The machine-readable storage medium 120 and the processor 130 are electrically connected directly or indirectly to each other to enable data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses or signal lines. The real-time telemetry indicator enhancement processing device 110 includes at least one software function module that can be stored in the machine-readable storage medium 120 or embedded in the operating system (OS) of the electronic device 100 in the form of software or firmware. The processor 130 is used to execute executable modules stored in the machine-readable storage medium 120, such as the software function modules and computer programs included in the real-time telemetry indicator enhancement processing device 110.

[0094] The machine-readable storage medium 120 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc. The machine-readable storage medium 120 is used to store programs. After receiving execution instructions, the processor 130 executes the programs, which can realize the real-time telemetry indicator enhancement processing method provided in this embodiment.

[0095] Processor 130 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0096] Please refer to Figure 3This embodiment also provides a real-time telemetry indicator enhancement processing device 110, which includes at least one functional module that can be stored in a machine-readable storage medium 120 in software form. Functionally, the real-time telemetry indicator enhancement processing device 110 may include a data acquisition module 111, a first query module 112, a second query module 113, a third query module 114, and a data enhancement module 115.

[0097] The data acquisition module 111 is used to acquire a list of query items including multiple query items, each query item being a combination of a telemetry indicator data point and its corresponding metadata fields to be queried; In this embodiment, the data acquisition module 111 can be used to perform... Figure 1 For a detailed description of the data acquisition module 111, see the description of step S110 shown.

[0098] The first query module 112 is used to query the first-level local cache and obtain the first query result corresponding to the query item in the query item list that can be hit in the first-level local cache; In this embodiment, the first query module 112 can be used to execute... Figure 1 For a detailed description of the first query module 112, see the description of step S120 shown.

[0099] The second query module 113 is used to query the second-level distributed cache for the query item that is not found in the first-level local cache, obtain the second query result, and detect the remaining lifetime of the cache entry. If the remaining lifetime is less than the preset refresh threshold, the cache entry is updated. In this embodiment, the second query module 113 can be used to execute... Figure 1 For a detailed description of the second query module 113, see the description of step S130 shown.

[0100] The third query module 114 is used to call the metadata API in batches according to the preset block size for the query items that are not found in the first-level local cache and the second-level distributed cache, and write the obtained batch query results into the first-level local cache and the second-level distributed cache. In this embodiment, the third query module 114 can be used to execute... Figure 1 For a detailed description of step S140 shown, please refer to the description of step S140 for details regarding the third query module 114.

[0101] The data augmentation module 115 is used to augment the telemetry indicator data points in the list of query items based on the first query result, the second query result, and the batch query result, and output the augmented indicator data.

[0102] In this embodiment, the data enhancement module 115 can be used to perform... Figure 1 For a detailed description of the data augmentation module, please refer to the description of step S150 shown.

[0103] In summary, the telemetry indicator real-time enhancement processing method and apparatus provided in this application, through a two-level caching architecture and step-by-step querying, enables most metadata query requests to be completed in microseconds, significantly reducing overall processing latency. Simultaneously, for query items that are not found in either level of cache, the metadata API is called in batches according to a preset block size for batch querying of field values, and the results are then backfilled into the two-level cache. This significantly reduces the number of calls to external services and provides a cache hit basis for subsequent queries of the same field, thus forming a virtuous cycle of cache query and backfilling. Furthermore, when querying the second-level distributed cache, the remaining lifetime of cache entries is detected, and an update is triggered when it is less than the refresh threshold, effectively avoiding query latency jitter after cache expiration, further ensuring the stability of system processing performance, and enabling efficient, low-latency real-time enhancement processing.

[0104] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0105] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0106] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0108] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for real-time enhancement processing of telemetry indicators, characterized in that, Includes the following steps: Obtain a list of query items, which includes multiple query items, where each query item is a combination of a telemetry indicator data point and its corresponding metadata fields to be queried; Query the first-level local cache to obtain the first query result corresponding to the query item in the query item list that can be hit in the first-level local cache; For the query item that is not found in the first-level local cache, the second-level distributed cache is queried to obtain the second query result, and the remaining lifetime of the cache entry is checked. If the remaining lifetime is less than a preset refresh threshold, the cache entry is updated. For the query item that is not found in either the first-level local cache or the second-level distributed cache, the corresponding field value is called in batches according to the preset block size to perform batch query on the metadata API, and the obtained batch query results are written to the first-level local cache and the second-level distributed cache. Based on the first query result, the second query result, and the batch query result, the telemetry indicator data points in the list of items to be queried are enhanced, and the enhanced indicator data is output.

2. The method according to claim 1, characterized in that, The step of obtaining a list of query items that includes multiple query items includes: Receive batches of raw indicator data, wherein the batches of raw indicator data contain at least one raw telemetry indicator data point. Perform at least one of the following data quality control processes on the original telemetry index data points: data deduplication, consistent hash sampling, and cardinality control, to obtain a list of data points to be enhanced. Iterate through all the telemetry index data points in the list of data points to be enhanced, extract the metadata fields to be queried, combine each telemetry index data point and its corresponding metadata fields to be queried into the query item, and perform deduplication and merging processing on all query items to generate the query item list.

3. The method according to claim 1, characterized in that, The step of updating the cache entry includes: If the current second query result is returned, an asynchronous task is started to attempt to acquire the distributed lock corresponding to the cached entry; If the distributed lock is successfully acquired, a cache update operation is performed, and the distributed lock is released after the update is completed. If the distributed lock cannot be acquired, the cache update operation will not be performed.

4. The method according to claim 3, characterized in that, The step of updating the cache entry further includes: During the cache update operation, the cache entry is checked again before updating the cache to see if it has been updated by other nodes. When a cache update operation fails, the existing old data in the cache entry is retained.

5. The method according to claim 3, characterized in that, The step of updating the cache entry further includes: A first time-to-live (TTL) is set for cached entries obtained through successful queries, and a second TTL is set for cached entries obtained through unsuccessful queries; the first TTL is longer than the second TTL.

6. The method according to claim 5, characterized in that, The step of setting a first time-to-live (TTL) for cached entries obtained through successful queries and setting a second TTL for cached entries obtained through unsuccessful queries includes: For a cached entry that is successfully retrieved, a time period is randomly determined within the first time range as the first lifetime. For cached entries obtained after a failed query, a random duration is determined within the second duration range as the second lifetime.

7. The method according to claim 1, characterized in that, The step of batch querying the corresponding field values ​​of the item that is not found in either the first-level local cache or the second-level distributed cache by calling the metadata API in batches according to a preset block size includes: The field values ​​corresponding to the query items that are not found in either the first-level local cache or the second-level distributed cache are aggregated into a query field set. The set of fields to be queried is divided into at least one subset of fields according to a preset block size; For each field subset, a batch query is performed by calling the metadata API, with each field subset corresponding to one API call. The query results obtained from each batch query are aggregated and written to the first-level local cache and the second-level distributed cache.

8. The method according to claim 1, characterized in that, The step of enhancing the telemetry index data points in the list of items to be queried includes: The number of worker threads in the worker thread pool is adaptively determined based on the central processing unit resources of the operating environment, and the telemetry index data points in the list of items to be queried are enhanced in parallel through the worker thread pool.

9. The method according to any one of claims 1 to 8, characterized in that, The step of enhancing the telemetry index data points in the list of items to be queried includes: The query can perform at least one of the following calculations based on a single indicator in the telemetry data points in the query list: summation, average, count, minimum, maximum, 95th percentile, and 99th percentile; and / or at least one of the following calculations based on multiple indicators in the telemetry data points in the query list: summation, difference, multiplication, and division.

10. A real-time enhancement processing device for telemetry indicators, characterized in that, The real-time enhancement processing device for telemetry indicators includes: The data acquisition module is used to acquire a list of query items including multiple query items, where each query item is a combination of a telemetry indicator data point and its corresponding metadata fields to be queried; The first query module is used to query the first-level local cache and obtain the first query result corresponding to the query item in the query item list that can be hit in the first-level local cache; The second query module is used to query the second-level distributed cache for the query item that is not hit in the first-level local cache, obtain the second query result, and detect the remaining lifetime of the cache entry. If the remaining lifetime is less than a preset refresh threshold, the cache entry is updated. The third query module is used to perform batch queries on the corresponding field values ​​of the query item that is not found in the first-level local cache and the second-level distributed cache by calling the metadata API in batches according to the preset block size, and write the obtained batch query results into the first-level local cache and the second-level distributed cache. The data enhancement module is used to enhance the telemetry indicator data points in the list of items to be queried based on the first query result, the second query result, and the batch query result, and output the enhanced indicator data.

Images