High-performance time sequence database storage system and method based on telecommunication network data

Abstract

The invention discloses a high-performance time sequence database storage system and method based on telecommunication network data, and belongs to the technical field of big data storage. The system comprises an interface layer, a processing layer and a storage layer, wherein the storage layer adopts a Redis cluster and an HBase cluster to store hotspot data and total data respectively; and the processing layer dynamically generates Rowkey of the HBase table through a configurable index tag mechanism, and completes aggregation operation and intelligently manages cache in a data writing stage in combination with preaggregation calculation and a Hermes-SDK dynamic identification hotspot data mechanism. According to the method, the problems of performance bottleneck, low cache static state efficiency and high real-time aggregation calculation pressure caused by Cartesian product scanning in massive time series data query in a telecommunication network are effectively solved, high-concurrency and low-delay data writing and query are realized, and the system throughput and the resource utilization rate are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of big data storage technology, and specifically to a high-performance time-series database storage system and method based on telecommunications network data. Background Technology

[0002] In the telecommunications network domain, there are multiple specialties, such as IP, IDC, environmental monitoring, PON, and cloud. The amount of performance data is closely related to the number of cloud network devices, the number of indicators, and the collection frequency. In fact, the amount of performance data generated every day in the telecommunications network domain of each province reaches tens of billions.

[0003] To meet the demands of various operational support scenarios, existing architectures typically use a time-series database as the primary storage, supplemented by a caching library to provide hot data query services. OpenTSDB is a distributed time-series database primarily used for processing large amounts of time-series data. It uses HBase as its underlying storage system and provides a simple HTTP API for data insertion and querying. Furthermore, through compressed and optimized time-series structures, OpenTSDB can efficiently store and query large volumes of time-series data. To mitigate the limitations of high-concurrency queries, it can be used in conjunction with the Redis caching library for hot data queries, trading server memory resources for improved query performance.

[0004] Due to the extremely rapid growth in data volume, the existing architecture faces the following urgent problems that need to be addressed as data volume continues to increase: The concurrent query performance of both open-source and commercial time-series database systems on the market cannot meet the high requirements of the telecommunications network field, necessitating the use of caching libraries. However, the limited memory resources of servers prevent unlimited expansion, resulting in significant performance issues related to query performance.

[0005] OpenTSDB is based on HBase storage, and the HBase Rowkey design includes metrics, time, performance values, and all tags. However, in the telecommunications network domain, data queries do not pass all tags as API parameters; only the primary key is provided. This results in an extremely large number of data rows being scanned by HBase when calling the query interface, leading to severe Cartesian product problems and significantly impacting data query performance. Furthermore, because the primary keys for queries differ across different specializations within the telecommunications network domain, using a fixed index such as IP address does not fundamentally solve the problem.

[0006] Whether it's a regular raw data query or an aggregation calculation scenario, the same data provision logic is used, namely scanning HBase data tables, performing analysis and calculations, and transforming the structure. This leads to a surge in database pressure during aggregation calculations, affecting the performance of both data writing and regular queries at that moment, resulting in a decline in user experience.

[0007] For scenarios where cache libraries provide hot data, current practices generally rely on experience-based strategies. Furthermore, the lifecycle management of hot data is typically handled by periodically eliminating it without dynamically linking it to the actual query frequency. This results in inaccurate cached hot data. Summary of the Invention

[0008] The purpose of this invention is to provide a high-performance time-series database storage system and method based on telecommunications network data, so as to solve the problems of insufficient system performance caused by Cartesian product scanning, static inefficiency of caching strategies, and high real-time pressure of aggregation calculation in the prior art.

[0009] To achieve the above objectives, the technical solution provided by the present invention is: a high-performance time-series database storage system based on telecommunications network data, comprising: an interface layer that provides external service interfaces, a processing layer for data processing and optimization of data from the interface layer, and a storage layer responsible for storing data processed by the processing layer. The interface layer is an HTTP interface that supports data writing, querying, and DSL filtering operations; The processing layer is used to perform pre-calculation processing during data writing to generate aggregation results; The storage layer includes a Redis cluster as a cache for hot data and an HBase cluster as the main database; The processing layer distinguishes between GET and SCAN queries based on traffic splitting conditions during data reading and stores time-series data in blocks based on the Rowkey structure. The processing layer has built-in aggregation functions and contains a configuration management module that can set the Rowkey structure and hot data caching strategies.

[0010] To optimize the above technical solution, the specific measures also include: The processing layer also includes a write engine and a read engine; The write engine is configured as follows: verify the written data; determine the index label and generate the Rowkey structure of the HBase table according to the mapping relationship between the metric and the index label; identify hot data based on Hermes-SDK and write the hot data to the Redis cluster; and query the pre-aggregation configuration table when writing data. If the metric of the currently written data is configured with a pre-aggregation operator, then execute the corresponding aggregation calculation synchronously and store the result in the HBase cluster. Furthermore, the read engine is configured as follows: based on Hermes-SDK, hot data is identified and hot data queries are routed to the Redis cluster; for non-hot data queries, Rowkey is generated based on index tags to obtain data from the HBase cluster, and tag matching through filters, as well as aggregation or downsampling calculations are supported.

[0011] The specific process of identifying hotspot data based on Hermes-SDK includes: Hermes-SDK is embedded in the write engine and read engine. When writing and querying data, it dynamically generates a cache query key based on the indicator name and device IP address in the written or queried data. Hermes-SDK reports the cache query key to the independent Hermes server cluster. Furthermore, the Hermes server cluster periodically analyzes the access frequency of all reported cache query keys, identifies keys with access frequencies exceeding a dynamic threshold as hot data keys, and updates the global hot data key list to each Hermes-SDK. The write engine or read engine determines whether the cached query key of the current operation is hot data by querying the list of hot data keys in the local Hermes-SDK, so as to determine the storage or query path of the data.

[0012] The configuration management module stores the mapping relationship between metrics and index tags, as well as a pre-aggregated configuration table; the Rowkey of the HBase table generated by the write engine is composed of the following three parts concatenated in sequence: the metric name in the written data, the base time after the timestamp of the written data is normalized according to the preset time granularity, and the index tag value corresponding to the metric obtained from the configuration management module.

[0013] Furthermore, the pre-aggregation configuration table predefines whether pre-aggregation is enabled and the types of aggregation operators enabled through different indicators; wherein, the aggregation operator types include maximum value, minimum value, average value, and summation.

[0014] The synchronous execution of aggregation calculations by the write engine and storage of the results into the HBase cluster specifically includes: in the data writing process, when it is determined that the metric of the currently written data has at least one aggregation operator configured in the pre-aggregation configuration table, the write engine reads the historical aggregation results of the corresponding dimension of the metric from the pre-aggregation result table of the HBase cluster; Furthermore, the performance value of the currently written data is calculated based on the configured aggregation operator type and the historical aggregation results to obtain a new aggregation result value. The new aggregation result value and the original written data are written to the corresponding pre-aggregation result table and source data table in the HBase cluster within the same transaction or batch.

[0015] The read engine also includes a filter processing unit configured to convert tag regular expression matching or complex filtering conditions in the query request into HBase database executable filters to complete the filtering during the data scanning phase.

[0016] As another important technical solution, the present invention also provides a time-series data storage method, comprising the following steps: S1: Perform format validation, type conversion, and tag extraction on the input source data; S2: Reads the data processed by S1, performs pre-calculation, and generates aggregate results; during data reading, GET queries or SCAN queries are distinguished by query splitting conditions. S3: Use Hermes-SDK to determine whether data is hot data based on a query frequency threshold; if so, write it to the Redis cluster. S4: Write storage data to both the Redis cluster (used as a cache for hot data) and the HBase cluster (used as the main database).

[0017] When writing data to storage, if it is hot data, the source data is parsed into structured fields, and the index labels of the data metrics are determined. The hot data is then used to construct a Rowkey structure based on the structured fields, time dimension, and index labels, and then converted into an ordered set structure to be written to the Redis cluster.

[0018] Furthermore, if the source data metrics are configured for pre-aggregation, the pre-aggregation results are calculated and updated while the source data is being written, and the source data and pre-aggregation results are then written to the HBase cluster in batches.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a storage system architecture that includes an interface layer, a processing layer, and a storage layer, and adopts a layered storage strategy of Redis cluster and HBase cluster. This enables intelligent management of hot and cold data splitting for massive time-series data, significantly reducing the direct load on the main database in high-concurrency query scenarios. It effectively avoids the risk of data write blocking or even system crash due to excessive query pressure, thereby comprehensively ensuring the high availability and stability of the database system.

[0020] This invention innovatively designs a configurable HBase Rowkey structure based on metrics, benchmark time, and index tags, and stores non-index tags in the column family root to adapt to the query mode in the telecommunications field that uses a few key tags as primary keys. This allows most queries to locate data through precise Get operations, greatly reducing the scope of full table scans and effectively alleviating the Cartesian product problem caused by tag combinations in traditional time-series databases, thereby significantly improving query performance and concurrency processing capabilities.

[0021] This invention integrates a pre-aggregation mechanism into the data writing process. Based on a predefined configuration table, the system synchronously calculates and updates common aggregation results such as maximum, minimum, average, and summation when data is written to the database, and persists these results. This allows the system to directly return pre-calculated results when responding to aggregation query requests, avoiding real-time scanning and calculation of massive amounts of raw data during queries. This significantly reduces the real-time computing pressure on the database and substantially improves the response speed of aggregation queries and the overall system throughput.

[0022] This invention achieves dynamic discovery and lifecycle management of hot data through the integrated Hermes-SDK and server cluster. The system can analyze the access frequency of data items in real time, automatically identify and cache hot data, and dynamically evict old caches based on changes in popularity. This intelligent caching strategy based on actual access patterns significantly improves cache hit rate and memory resource utilization efficiency compared to static strategies based on fixed experience or periodic eviction, ensuring that limited cache resources always serve the most frequent query needs.

[0023] The processing layer of this invention has a rich set of aggregation functions, downsampling methods and flexible index calculation capabilities, providing users with ready-to-use support for complex data analysis without the need for additional secondary development and calculation at the application layer. This not only reduces the user's threshold and development cost, but also ensures computational efficiency and system performance, meeting the diverse data consumption scenarios in telecommunications network operation and maintenance.

[0024] The filter processing unit of this invention can convert complex conditions in user query requests into HBase-based executable filters, enabling the data filtering process to be completed during the database scanning phase. This avoids transmitting a large amount of unnecessary data to the application layer for filtering, reducing network overhead and application layer processing pressure, further optimizing the query path, and improving query efficiency.

[0025] This invention provides services to the outside world through a unified HTTP interface and provides a dedicated DSL to support complex query operations such as filtering, aggregation, and partitioning. It provides upper-layer applications with a simple, standardized, and powerful way to access and query data, improving the system's usability and integration convenience. At the same time, the data normalization processing at the interface layer ensures the consistency of the returned result structure, improving the user experience. Attached Figure Description Figure 1 : Schematic diagram of the overall architecture of the high-performance time-series database storage system of this invention.

[0026] Figure 2 The following is a flowchart of the timing data writing process of the write engine in this embodiment of the invention.

[0027] Figure 3: Flowchart of time-series data query for the read engine in this embodiment of the invention. Detailed Implementation The present invention will be further described in detail below through specific embodiments, but it should not be construed as limiting the scope of the subject matter of the present invention to the following embodiments. All technologies implemented based on the above content of the present invention fall within the scope of the present invention.

[0028] The following is an explanation of some of the terms used in this invention: Column Family: An HBase column family is a collection of columns in an HBase table. In HBase, each row in a table can contain multiple column families, and each column family can contain multiple columns. Column families are a logical organizational structure used to group related columns together for more efficient data access and processing. Each column family has a unique name used to identify and access the columns within that family in an HBase table.

[0029] Hermes-SDK: An open-source SDK wrapper for hot data discovery and local caching. Jedis-Client (the direct entry point for Java applications to interact with the caching server) integrates the corresponding capabilities by interacting with it. The technical solution of the present invention will be further described in detail below with reference to specific embodiments: In some implementations, such as Figure 1 As shown, the present invention provides a high-performance time-series database storage system based on telecommunications network data, comprising: an interface layer that provides external service interfaces, a processing layer for data processing and optimization of data from the interface layer, and a storage layer responsible for storing data processed by the processing layer. In some implementations, the interface layer is an HTTP interface that supports data writing, querying, and DSL filtering operations, including a data writing interface, a data query interface, and further filtering, aggregation, partitioning, etc., through DSL in the data query interface; The processing layer is used to perform pre-calculation processing when data is written, and to generate aggregation results; In some implementations, the storage layer includes a Redis cluster as a cache for hot data and an HBase cluster as the master database, both of which adopt a master-slave architecture. The Redis cluster uses 1 master and 2 slaves, and the HBase cluster has 7 nodes. In practice, the size can be adjusted according to the amount of data.

[0030] The processing layer distinguishes between GET and SCAN queries when reading data by using splitting conditions, and stores time-series data in blocks based on the Rowkey structure. The processing layer has built-in aggregation functions and contains a configuration management module that can set the Rowkey structure and hot data caching strategies.

[0031] In some implementations, the configuration management module is configured to: configure different rowkey structures according to different metrics to maximize data retrieval via GET queries, avoid SCAN scans, and improve performance; configure hot data caching, cache switch configuration, cache metric configuration, cache data expiration mechanism, retrieve hot data through Hermes-SDK, and evict expired data.

[0032] The processing layer also includes a write engine and a read engine; The write engine is configured as follows: perform data length, encoding, and type validation on the written data; determine the index label based on the mapping relationship between the metric and the index label and generate the Rowkey structure of the HBase table, and store data with the same rowkey in blocks; identify hot data based on Hermes-SDK and write the hot data to the Redis cluster; and query the pre-aggregation configuration table when writing data. If the metric of the currently written data is configured with a pre-aggregation operator, then execute the corresponding aggregation calculation synchronously and store the result in the HBase cluster. Preferably, the read engine is configured as follows: based on Hermes-SDK, hot data is identified and hot data queries are routed to the Redis cluster; for non-hot data queries, Rowkey is generated based on index tags to retrieve data from the HBase cluster, and tag matching through filters, as well as aggregation or downsampling calculations are supported.

[0033] The specific process of identifying hotspot data based on Hermes-SDK includes: Hermes-SDK is embedded in the write engine and read engine. When writing and querying data, it dynamically generates a cache query key based on the metric name and device IP address in the written or queried data. Hermes-SDK then reports the cache query key to the independent Hermes server cluster. Preferably, the Hermes server cluster periodically analyzes the access frequency of all reported cache query keys, identifies keys whose access frequency exceeds a dynamic threshold as hot data keys, and updates the global hot data key list to each Hermes-SDK. The write engine or read engine determines whether the cached query key of the current operation is hot data by querying the list of hot data keys in the local Hermes-SDK, in order to decide the data storage or query path.

[0034] The configuration management module stores the mapping relationship between metrics and index tags, as well as a pre-aggregated configuration table; the Rowkey of the HBase table generated by the write engine is composed of the following three parts concatenated in sequence: the metric name in the written data, the base time after the timestamp of the written data is normalized according to the preset time granularity, and the index tag value corresponding to the metric obtained from the configuration management module.

[0035] Preferably, the pre-aggregation configuration table predefines whether pre-aggregation is enabled and the types of aggregation operators enabled through different indicators; wherein, the aggregation operator types include maximum value, minimum value, average value and summation.

[0036] The synchronous execution of aggregation calculations by the write engine and storage of the results into the HBase cluster specifically includes: In the data writing process, when it is determined that the metric of the currently written data has at least one aggregation operator configured in the pre-aggregation configuration table, the write engine reads the historical aggregation results of the corresponding dimension of the metric from the pre-aggregation result table of the HBase cluster; The performance value of the currently written data is calculated based on the configured aggregation operator type and the historical aggregation results to obtain a new aggregation result value. The new aggregation result value and the original written data are written to the corresponding pre-aggregation result table and source data table in the HBase cluster within the same transaction or batch.

[0037] Preferably, the read engine also includes a filter processing unit configured to convert tag regular expression matching or complex filtering conditions in the query request into HBase database executable filters to complete the filtering during the data scanning phase.

[0038] In some implementations, aggregation methods are used to perform statistical calculations on time-series data over a period of time during a query, including maximum, minimum, summation, and average values. The specific implementation steps are as follows: First, the query interface receives the aggregation request and parses parameters such as aggregation type, time range, indicator name, and label conditions. Then, the system generates the corresponding Rowkey or Scan range based on the query conditions and retrieves the set of raw data points that meet the conditions from HBase. The raw data is filtered according to the label conditions and grouped by time window or device. Next, the corresponding aggregation operation is performed on the numerical sequences within each group: for maximum or minimum values, the system iterates through the numerical sequences and records the extreme values; for summation, all values ​​are accumulated; for average values, the values ​​are accumulated and divided by the number of data points. Finally, the aggregation results are encapsulated into structured JSON or time-series data format and returned to the querying party.

[0039] This sampling method is used to downsample frequently collected data, reducing the data volume and facilitating trend analysis. It supports sampling methods for maximum, minimum, summation, and average values. The specific implementation process is as follows: The query interface receives sampling requests and parses parameters such as the sampling interval (e.g., every 5 minutes), sampling method (e.g., taking the maximum value), and time range. The system obtains the original data sequence based on the time range and indicators, and divides the time range into multiple time windows according to the sampling interval. A specified aggregation operation (e.g., taking the maximum value) is performed on the data within each time window to obtain the representative value for that window. The representative values ​​of each window are arranged in chronological order to form the downsampled time-series data sequence, which is then returned. This method significantly reduces the data volume and is suitable for chart display or trend analysis.

[0040] Preferably, pre-aggregation involves calculating and storing aggregation results in advance during the data writing phase to improve query performance. The specific implementation process is as follows: The system configures whether pre-aggregation and aggregation operators are needed for different metrics in a configuration table. During the data writing process, if the metric currently being written has pre-aggregation configured, pre-aggregation calculation is triggered: The system reads the historical aggregation results (such as the current maximum value) of the metric within a specified time window (e.g., hourly) from the pre-aggregation result table in HBase, calculates and updates the newly written value with the historical results, and writes the updated pre-aggregation results and the original written data into the corresponding pre-aggregation result table and source data table in HBase within the same transaction or batch. When querying aggregated data, the system prioritizes obtaining results from the pre-aggregation table, eliminating the need to scan the original data table in real time, thereby significantly reducing query latency and master database pressure.

[0041] The metric calculation method supports expression calculations for multiple metrics, such as comparisons between devices and arithmetic operations between metrics. The specific implementation process is as follows: The system parses the expression in the query request to determine the list of involved metrics. Based on these metrics, it retrieves the raw data sequences of each metric in parallel from HBase. The data for multiple metrics are aligned by timestamps, and expression calculations are performed point-by-point, supporting arithmetic operations, comparison operations, and function calls. Finally, the system generates a new metric sequence with a structure consistent with the time-series data and returns it to the querying party.

[0042] The present invention also provides a time-series data storage method, comprising the following steps: S1: Perform format validation, type conversion, and tag extraction on the input source data; S2: Reads the data processed by S1, performs pre-calculation, and generates aggregate results; during data reading, GET queries or SCAN queries are distinguished by query splitting conditions. S3: Use Hermes-SDK to determine whether data is hot data based on a query frequency threshold; if so, write it to the Redis cluster. S4: Write storage data to both the Redis cluster (used as a cache for hot data) and the HBase cluster (used as the main database).

[0043] Preferably, when writing data to storage, if it is hot data, the source data is parsed into structured fields, and the index labels of the data metrics are determined. The hot data is then used to construct a Rowkey structure based on the structured fields, time dimension, and index labels, and converted into an ordered set structure before being written to the Redis cluster.

[0044] If the source data metrics are configured for pre-aggregation, the pre-aggregation results are calculated and updated while the source data is being written, and the source data and pre-aggregation results are then written to the HBase cluster in batches.

[0045] In some embodiments, the core workflow of the write engine and read engine in the time-series database system of the present invention is as follows: like Figure 2 As shown, the time-series data writing process of the write engine is as follows: When an external request calls the data write interface, the write engine first validates the requested data, including the format, required fields (such as metric, timestamp, value), data type, and length. After successful validation, the write engine dynamically generates a query key based on the metric name and device IP in the data, and calls the embedded Hermes-SDK to determine whether the key is hot data. If it is hot data, the cache write logic is entered first: the metric, tag, timestamp, and value of the data are parsed; using the metric and device IP as the key, the non-indexed tag set and timestamp are used as members and scores respectively, and an ordered set (ZSET) structure is constructed and written to the Redis cluster. Regardless of whether the data is hot or not, the write engine will execute the master database write logic: obtain the index tag (IndexTag) corresponding to the current metric from the configuration management; round the timestamp to the hourly granularity to obtain the base time (baseTime); and generate the row key (Rowkey) of the HBase table, with the structure metric + baseTime + indexTag value. Non-indexed tags and their corresponding performance values ​​are stored in the specified column family of the corresponding row in the HBase table, based on their timestamp offset relative to the base time. Simultaneously, the write engine queries the pre-aggregation configuration table; if the currently written metric is configured with a pre-aggregation operator, the aggregation result within that time window is calculated and updated synchronously. Finally, the raw performance data is written to the source data table in HBase, while the pre-aggregation results are written to a dedicated pre-aggregation result table.

[0046] In some implementations, such as Figure 3As shown, the time-series data query process of the read engine is as follows: When an external request calls the data query interface, the read engine first validates the query parameters, parsing out the required fields such as metrics, time ranges, and filter tags. Then, the read engine generates a query key based on the parsed metrics and device IP, and uses Hermes-SDK to determine if it is a hot data query. If it is a hot data query, the request will be directly routed to the Redis cluster, retrieving data in ordered set format based on the aforementioned key. If it is not a hot data query, the main database query logic is entered: First, it determines if it is an aggregation query. If so, it first attempts to retrieve the result directly from the pre-aggregation result table in HBase using the Get(Rowkey) method. If a match is found, the result is returned directly; otherwise, the original data query path continues. For original data queries, the read engine's filter processing unit converts the query conditions (such as regular expression matching of tags, complex filtering) into HBase executable Filters. The system generates an accurate Rowkey based on the index tags in the query conditions for a Get operation; otherwise, it generates a range for a Scan operation. After retrieving data blocks from HBase, the read engine determines whether aggregation or downsampling calculations are needed based on the user request. If so, it calls the corresponding aggregator or sampler for processing. Finally, the results are encapsulated by the data normalization module, including structure transformation, timestamp formatting, and time sorting, to provide a unified and readable data structure. The same data normalization process is performed on hot data retrieved from Redis to ensure consistency of the data returned by the interface. Throughout the process, the Hermes server cluster continuously collects access events reported by each SDK, dynamically analyzes and updates the hot data key list, achieving adaptive management of hot data.

[0047] Example 1 In the telecommunications O domain, there are multiple specializations, such as IP, IDC, environmental monitoring, PON, and cloud. The amount of performance data is closely related to the number of cloud network devices, the number of indicators, and the collection frequency. For example, if there are 10,000 network devices, each device collects 10 indicators, the collection cycle is once every 10 minutes, and each device has 100 ports, then the amount of data in one day is: 10,000 * 10 * 6 * 24 * 100 = 1,440,000,000.

[0048] In reality, the number of devices across all specialties in the Jiangsu Province telecommunications network reaches 100,000, so the data collection cycle is set to 5 minutes.

[0049] After the data platform obtains the data reported by the downstream collection platform, it calls the time series database writing interface of the present invention. First, it performs a preliminary verification to determine whether there are any required fields for the time series database in the source data, including collection time, indicators, and tags.

[0050] This includes data construction for both the Redis cache storage structure and the HBase storage structure. Only frequently accessed data needs to be stored in the cache; Hermes is used to determine if a Redis key is frequently accessed. Before data construction, the index tag for the metric is retrieved from the cache. For performance data collecting flow rate data from telecom IP professional equipment, there are generally two tags: device IP and device port number. The device IP is the index tag for the query, i.e., a mandatory query condition, and the metric is flow rate. The constructed data structure is as follows: In Redis, a zset structure corresponds to: ip && flux (key), timestamp (score), and flow rate value (value). Data is written to the cache library and the underlying library in batches of 100 records per second.

[0051] At this point, the performance data of the Telecom O Domain IP professional has been successfully written to the main storage component HBase and the cache library Redis. Under this system architecture, the write performance can reach 100,000 data points / second, effectively handling the huge volume of performance data from Telecom.

[0052] Example 2 After the IP-specific device performance data is written to the time-series database CT-TSDB, a data query service needs to be provided to support operational support scenarios such as device inspection. For device inspection, the upstream system will query performance data based on device IP, indicator code, and time range. Therefore, if the flow rate indicators of 1,000 devices need to be inspected, the CT-TSDB data query interface will be called 1,000 times, placing extremely high demands on the interface's concurrency.

[0053] When an external request calls the time-series database query interface for device inspection, the time-series database query program first performs preliminary data validation, parsing out required fields and format validation, including device IP, indicator code, and time range, and converting them into the underlying storage query structure. This includes the Redis cache key, and the HBase database Rowkey and columns. Only hot data needs to be stored in the cache; Hermes is used to determine if the Redis key is hot data. Before constructing the data, the index tag for that indicator is retrieved from the cache.

[0054] The query first determines whether it is a hot data point. It uses Hermes to determine whether the Redis key is hot data. Next, it checks whether the query conditions contain index tags. If they contain index tags, it quickly locates the row of the target data point using GET(Rowkey). If they do not contain index tags, it needs to scan multiple rows using Scan(). Finally, it obtains the final data block based on the column family offset.

[0055] The acquired data blocks are further filtered according to query conditions, such as returning data from a specified port; the timestamps are converted to YYYY-MM-DD HH:mm:SS to improve readability; and the returned data is sorted in ascending time order.

[0056] At this point, the performance data of the Telecom O domain IP can be provided to the upper-layer application system through the interface. Under this architecture, high-concurrency data device-level query scenarios can be realized to cope with scenarios such as large-scale device inspection. The cache library can reach 10,000 times / second, and the main database can reach 3,000 times / second.

[0057] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Any simple modifications, equivalent substitutions, and improvements made by those skilled in the art to the above embodiments without departing from the scope of the technical solution of the present invention, based on the technical essence of the present invention, shall still fall within the protection scope of the technical solution of the present invention.

Claims

1. A high-performance time-series database storage system based on telecommunications network data, characterized in that: It includes an interface layer that provides external service interfaces, a processing layer that processes and optimizes the data from the interface layer, and a storage layer that is responsible for storing the data processed by the processing layer. The interface layer is an HTTP interface that supports data writing, querying, and DSL filtering operations; The processing layer is used to perform pre-calculation processing during data writing to generate aggregation results; The storage layer includes a Redis cluster as a cache for hot data and an HBase cluster as the main database; The processing layer distinguishes between GET and SCAN queries based on traffic splitting conditions during data reading and stores time-series data in blocks based on the Rowkey structure. The processing layer has built-in aggregation functions and contains a configuration management module that can set the Rowkey structure and hot data caching strategies.

2. The high-performance time-series database storage system based on telecommunications network data according to claim 1, characterized in that: The processing layer also includes a write engine and a read engine; The write engine is configured as follows: verify the written data; determine the index label and generate the Rowkey structure of the HBase table according to the mapping relationship between the metric and the index label; identify hot data based on Hermes-SDK and write the hot data to the Redis cluster; and query the pre-aggregation configuration table when writing data. If the metric of the currently written data is configured with a pre-aggregation operator, then execute the corresponding aggregation calculation synchronously and store the result in the HBase cluster. The read engine is configured as follows: based on Hermes-SDK, hot data is identified and hot data queries are routed to the Redis cluster; for non-hot data queries, Rowkey is generated based on index tags to retrieve data from the HBase cluster, and tag matching through filters, as well as aggregation or downsampling calculations are supported.

3. The high-performance time-series database storage system based on telecommunications network data according to claim 2, characterized in that: The specific process of identifying hotspot data based on Hermes-SDK includes: Hermes-SDK is embedded in the write engine and read engine. When writing and querying data, it dynamically generates a cache query key based on the indicator name and device IP address in the written or queried data. Hermes-SDK reports the cache query key to the independent Hermes server cluster. The Hermes server cluster periodically analyzes the access frequency of all reported cached query keys, identifies keys with access frequencies exceeding a dynamic threshold as hot data keys, and updates the global hot data key list to each Hermes-SDK. The write engine or read engine determines whether the cached query key of the current operation is hot data by querying the list of hot data keys in the local Hermes-SDK, so as to determine the storage or query path of the data.

4. A high-performance time-series database storage system based on telecommunications network data according to claim 2, characterized in that: The configuration management module stores the mapping relationship between metrics and index tags, as well as a pre-aggregated configuration table; the Rowkey of the HBase table generated by the write engine is composed of the following three parts concatenated in sequence: the metric name in the written data, the base time after the timestamp of the written data is normalized according to the preset time granularity, and the index tag value corresponding to the metric obtained from the configuration management module.

5. A high-performance time-series database storage system based on telecommunications network data according to claim 2, characterized in that: The pre-aggregation configuration table predefines whether pre-aggregation is enabled and the types of aggregation operators enabled based on different indicators; wherein, the aggregation operator types include maximum value, minimum value, average value, and summation.

6. A high-performance time-series database storage system based on telecommunications network data according to claim 2, characterized in that: The synchronous execution of aggregation calculations by the write engine and storage of the results into the HBase cluster specifically includes: in the data writing process, when it is determined that the metric of the currently written data has at least one aggregation operator configured in the pre-aggregation configuration table, the write engine reads the historical aggregation results of the corresponding dimension of the metric from the pre-aggregation result table of the HBase cluster; The performance value of the currently written data is calculated based on the configured aggregation operator type and the historical aggregation results to obtain a new aggregation result value. The new aggregation result value and the original written data are written to the corresponding pre-aggregation result table and source data table in the HBase cluster within the same transaction or batch.

7. A high-performance time-series database storage system based on telecommunications network data according to claim 2, characterized in that: The read engine also includes a filter processing unit configured to convert tag regular expression matching or complex filtering conditions in the query request into HBase database executable filters to complete the filtering during the data scanning phase.

8. A time-series data storage method, characterized in that, Includes the following steps: S1: Perform format validation, type conversion, and label extraction on the input source data; S2: Read the data processed by S1, perform pre-calculation, and generate the aggregation result; During data reading, GET queries or SCAN queries are distinguished by query splitting conditions. S3: Use Hermes-SDK to determine whether data is hot data based on a query frequency threshold; if so, write it to the Redis cluster. S4: Write storage data to both the Redis cluster (used as a cache for hot data) and the HBase cluster (used as the main database).

9. The time-series data storage method according to claim 8, characterized in that: When writing data to storage, if it is hot data, the source data is parsed into structured fields, and the index labels of the data metrics are determined. The hot data is then used to construct a Rowkey structure based on the structured fields, time dimension, and index labels, and then converted into an ordered set structure to be written to the Redis cluster.

10. The time-series data storage method according to claim 8, characterized in that: If the source data metrics are configured for pre-aggregation, the pre-aggregation results are calculated and updated while the source data is being written, and the source data and pre-aggregation results are then written to the HBase cluster in batches.

Images