Data subscription method with flexible and controllable data granularity

Abstract

The invention discloses a data subscription method with flexible and controllable data granularity. The method comprises the following steps: setting a plurality of data nodes for storing time sequence data at a server; setting a management node for managing the plurality of data nodes at the server; when a target customer in a plurality of customers in the client needs to subscribe the time sequence data, the target customer sends a time sequence data subscription request containing a subscription theme to a management node of the server, and the subscription theme defines a data type and data granularity; the management node allocates a target data node with a data type to the target customer according to the data type defined by the subscription theme in the target customer subscription request, and establishes a point-to-point data channel between the target customer and the target data node; and the target data node sends the stored data to the target client through the data channel according to the data granularity defined by the subscription theme.

Description

Technical Field

[0001] This invention relates to the field of time-series data processing, and specifically to a data acquisition and subscription mechanism based on a time-series database that allows for flexible and controllable data granularity and facilitates data distribution, particularly a data subscription method with flexible and controllable data granularity. Background Technology

[0002] Different users have different data needs in time-series databases. These range from requiring the entire database to needing data from specific tables, or even just a single column within a table. Time-series databases need to carefully control this granularity, as this involves issues of secure data distribution and efficient data retrieval.

[0003] Furthermore, for data written in real time, an efficient method is needed to retrieve it in real time, especially for time-series databases, which have high requirements for real-time data retrieval.

[0004] Data subscription is an excellent way to acquire data. Therefore, implementing a data subscription mechanism based on time-series databases can effectively solve problems such as data distribution and efficient acquisition of real-time data in time-series databases.

[0005] Chinese patent document CN117668078A discloses a data subscription method, apparatus, electronic device, and storage medium. Through a subscription management unit in a dedicated data processor, it obtains subscription requests from any data service of a data node, identifies the subscription information of the requests to determine the subscription group, and pushes the subscription requests to at least one database or retrieves subscription data from at least one database. While this method provides data subscription management functionality, it still has shortcomings in terms of granular control over data subscriptions, making it difficult to achieve precise subscription to data in specific tables or columns.

[0006] Chinese patent document CN114722092A discloses a time-series data processing method, system, storage medium, and electronic patent technology. This method retrieves time-series data to be processed from a time-series database in response to client requests, stores the corresponding sharding results of the time-series data in a message queue, and writes the sharding results read from the message queue into memory. While this method provides storage and retrieval functions for time-series data, its real-time data acquisition capabilities still need improvement, particularly when the latest data is required, as it lacks an efficient mechanism for achieving this. Summary of the Invention

[0007] The purpose of this invention is to provide a time-series data (hereinafter referred to as "data") subscription method with flexible and controllable data granularity that can solve the above-mentioned problems of the prior art.

[0008] The present invention provides a data subscription method with flexible and controllable data granularity, comprising:

[0009] Set up multiple data nodes (Vnodes) on the server side to store time-series data;

[0010] On the server side, a management node Mnode is set up to manage the multiple data nodes Vnodes;

[0011] When a target customer Consumer1 among multiple customers in the client needs to subscribe to time series data, the target customer Consumer1 sends a time series data subscription request containing a subscription topic to the management node Mnode on the server, wherein the subscription topic defines the data type and data granularity.

[0012] The management node Mnode allocates a target data node with the data type defined in the subscription topic Topic in the subscription request of the target customer Consumer1, and establishes a point-to-point data channel between the target customer Consumer1 and the target data node.

[0013] The target data node sends the stored data to the target customer Consumer1 through the data channel according to the data granularity defined by the subscription topic.

[0014] Preferably, the management node Mnode assigns a target data node with the data type defined in the subscription topic Topic in the subscription request of the target customer Consumer1. This includes: the management node Mnode searches for data nodes Vnode that store the data type in the subscription topic Topic in the subscription request of the target customer Consumer1, and assigns one or more data nodes from the found data nodes Vnode as target data nodes to the target customer Consumer1 according to the load balancing algorithm.

[0015] Preferably, the process by which the management node Mnode establishes a point-to-point data channel between the target customer Consumer1 and the target data node includes: the management node Mnode forwarding the subscription topic to the target data node and sending the address of the target data node to the target customer Consumer1, so that the target customer Consumer1 establishes a point-to-point data channel with the target data node through the target data node address.

[0016] Preferably, the process of the target data node sending the stored data to the target customer Consumer1 through the data channel according to the data granularity defined by the subscription topic includes: the target customer Consumer1 sending a consumption request to the target data node using the target data node address, establishing the data channel between the target customer Consumer1 and the target data node; the target data node reading the data stored in the database according to the data granularity defined by the subscription topic forwarded by the management node Mnode, and transmitting the read data to the target customer Consumer1 via the data channel.

[0017] Preferably, the subscription topics include query topics applicable to specific business fields or specific conditional data; super table topics applicable to monitoring a certain type of equipment; and database topics used for database migration, disaster recovery backup, or building a full data warehouse.

[0018] Preferably, the query topic is used to subscribe to or specify one or more data columns in the source table or original data table and set data column filtering conditions so that the target data node distributes the specified one or more data columns to the target customer Consumer1 according to the data column filtering conditions.

[0019] Preferably, the super table topic is used to subscribe to or specify changes to the entire super table system, sub-table metadata, or sub-tables that meet the tag filtering conditions, so that the target data node can distribute the specified changes to the entire super table system, sub-table metadata, or sub-tables that meet the tag filtering conditions to the target customer Consumer1.

[0020] Preferably, the database Topic is used to subscribe to or specify all data changes of all tables in the target database or cloud data operations such as creation and deletion of all tables in the target database, so that the target data node can distribute the subscribed entire super table system, sub-table metadata change information, or sub-tables that meet the tag filtering conditions to the target customer Consumer1.

[0021] Preferably, when the target data node receives the first consumption request from the target customer Consumer1, it immediately records a snapshot version number of the current database, uses the snapshot version number to divide the data stream into historical data and real-time data; and uses the storage engine tsdb to save the historical data, and uses the write-ahead log WAL to save the real-time time-series data.

[0022] Preferably, the target data node reads the data stored in the database according to the data granularity defined by the subscription topic forwarded by the management node Mnode, specifically as follows: the target data node reads historical data from the storage engine tsdb that meets the filtering conditions before the snapshot version number, and reads real-time data from the write-ahead log wal that meets the filtering conditions after the snapshot version number.

[0023] The above-described technical solution of the present invention has the following beneficial technical effects:

[0024] 1. Database-based data subscription mechanism: This invention deeply integrates a data subscription mechanism within the time-series database, transforming data subscription from a traditional external message middleware into a native capability of the database kernel. This mechanism combines centralized scheduling by the management node (Mnode) with distributed execution by the data nodes (Vnode), enabling direct and efficient interaction between consumers and storage nodes. This not only simplifies the architectural complexity of data distribution but also achieves higher throughput and lower data latency than external solutions through database kernel-level optimization, providing a unified, high-performance data distribution solution for time-series data scenarios.

[0025] 2. Flexible and controllable data acquisition mechanism with fine granularity; This invention achieves fine-grained control over data granularity by creating subscription topics using SQL-like semantics. Subscribing to topics allows for direct column projection (SELECT) and predicate filtering (WHERE) at the data source head (data nodes). This server-side filtering mechanism overturns the traditional model of "pulling all data and filtering on the client side," enabling subscription granularity to be precise from the entire database and supertable (including metadata) down to specific rows and columns. This significantly reduces the amount of data transmitted over the network and processed on the client side, fundamentally improving distribution efficiency and achieving security access control based on data content.

[0026] 3. Efficient Real-Time Data Acquisition Mechanism Based on WAL: This invention combines database snapshots and Write-Ahead Log (WAL) dual-engine real-time consumption technology. Data nodes record the snapshot version number upon receiving a subscription and use this as a boundary to collaboratively acquire data from two sources: the storage engine TSDB (historical storage) and the WAL (real-time log). The core of achieving efficient real-time performance lies in the direct consumption of WAL: it allows newly written data to be pushed to consumers with extremely low latency (equivalent to database write latency) before it is written to disk as a historical file. This achieves near-synchronous timing between data writing and data distribution within the database, perfectly meeting the stringent real-time requirements of scenarios such as the Internet of Things and monitoring. Attached Figure Description

[0027] Figure 1This is a schematic diagram illustrating the principle of a data subscription method with flexible and controllable data granularity as described in this invention.

[0028] Figure 2 This is a schematic diagram of the load balancing algorithm and process of the management node Mnode in this invention;

[0029] Figure 3 This is a schematic diagram illustrating how the target data node of the present invention distributes subscribed data to the target customer. Detailed Implementation

[0030] Introduction to basic concepts

[0031] Time-series databases: A type of database system optimized for processing time-series data. Time-series data refers to a series of data points recorded in chronological order, such as sensor readings, application logs, or financial market data. These databases typically feature efficient time-range querying, data compression, and lifecycle management capabilities.

[0032] Management Node (Mnode): In the distributed time-series database system of this solution, the management node is a centralized coordination service component. It does not directly store user data, but is responsible for managing the cluster's metadata, coordinating the allocation relationship between data nodes and consumers, and executing load balancing strategies. It is the "brain" or "scheduling center" of the entire data subscription mechanism.

[0033] A data node (Vnode) is the physical or logical unit in a distributed time-series database that actually stores and processes data. Each data node is responsible for one or more data shards, directly handling data subscription requests from consumers and performing data reading and filtering operations.

[0034] A Topic is a core logical abstraction introduced in this invention. A topic represents a named data subscription stream, the content of which is defined through SQL-like statements. It clarifies "who (consumer) consumes data from which source (from which table, supertable, or database) at what granularity (which columns to select, what conditions to meet)." Topics are key to achieving flexible and controllable granularity in data subscriptions.

[0035] TSDB: Refers to the storage engine of a time-series database, responsible for persistently storing historical data on disk in optimized formats such as columnar and compressed formats. It is the primary source for consuming historical data.

[0036] WAL (Write-Ahead Log) is a common "write-ahead log" technology in database systems. Any data write operation is sequentially and quickly recorded in the WAL file before being successfully committed and stored in the TSDB. While primarily used for fault recovery, in this invention, it is innovatively used as a high-efficiency data source for real-time data consumption; its append-only nature naturally makes it a high-performance message queue.

[0037] Consumer: Refers to a client application or service that requests and receives subscribed data. Consumers declare their data needs by subscribing to topics from the management node and then pull or receive data streams directly from the data nodes according to the instructions.

[0038] See Figure 1 The present invention sets up multiple data nodes Vnode for storing time-series data and management nodes Mnode for managing the multiple data nodes Vnode on the server side; consumers or clients (hereinafter referred to as "clients") obtain data from the server side.

[0039] See Figure 1 The present invention provides a data subscription method with flexible and controllable data granularity, comprising: when a target customer Consumer1 among multiple customers in a client needs to subscribe to time-series data, the target customer Consumer1 sends a time-series data subscription request containing a subscription topic (Topic) to a management node Mnode on the server side, wherein the subscription topic defines the data type and data granularity; the management node Mnode allocates a target data node with the data type defined in the subscription topic defined in the target customer Consumer1's subscription request to the target customer Consumer1, and establishes a point-to-point data channel between the target customer Consumer1 and the target data node; the target data node Vnode sends the stored data to the target customer Consumer1 through the data channel according to the data granularity defined in the subscription topic.

[0040] The management node Mnode allocates target data nodes with the data type defined in the subscription topic Topic in the subscription request of the target customer Consumer1. This includes: the management node Mnode searches for data nodes Vnode that store the data type defined in the subscription topic Topic in the subscription request of the target customer Consumer1, and allocates one or more data nodes from the found data nodes Vnode as target data nodes to the target customer Consumer1 according to the load balancing algorithm.

[0041] The management node Mnode establishes a point-to-point data channel between the target customer Consumer1 and the target data node by: forwarding the subscription topic to the target data node, sending the target data node address to the target customer Consumer1, and enabling the target customer Consumer1 to establish a point-to-point data channel with the target data node through the target data node address.

[0042] The process of the target data node Vnode sending the stored data to the target customer Consumer1 through the data channel according to the data granularity defined by the subscription topic includes: after the target customer Consumer1 sends a consumption request or a data retrieval request to the target data node according to the target data node address, the target data node Vnode transmits the time-series data to the target customer Consumer1 according to the data granularity defined by the subscription topic forwarded by the management node Mnode.

[0043] Subscription topics include query topics applicable to specific business fields or specific conditions; super table topics applicable to monitoring a certain type of device; and database topics used for database migration, disaster recovery backup, or building a full data warehouse. Target customers (Consumer1) obtain data at different granularities from target data nodes (Vnodes) by creating query topics, super table topics, or database topics.

[0044] A query topic is used to subscribe to or specify one or more data columns in a source table or original data table and set data column filtering conditions so that the target data node distributes the specified one or more data columns to the target customer Consumer1 according to the data column filtering conditions. Therefore, the data granularity defined by a query topic is the data column.

[0045] A supertable is a table template in a time-series database, with its specific data stored in multiple sub-tables with the same structure. The supertable topic is used to subscribe to or specify changes to the entire supertable system, sub-table metadata, or sub-tables that meet tag filtering conditions. This allows the target data node to distribute the specified supertable system, sub-table metadata changes, or sub-tables that meet tag filtering conditions to the target client, Consumer1. Therefore, the data granularity defined by a supertable topic is the entire supertable system, sub-table metadata changes, or sub-tables.

[0046] Database topics are used to subscribe to or specify all data changes to all tables in a target database, or cloud data operations such as the creation and deletion of all tables within the target database. This allows the target data node to distribute the subscribed super table system, sub-table metadata change information, or sub-tables that meet the tag filtering conditions to the target client, Consumer1. Therefore, the data granularity defined by a database topic is the data changes to the entire database or the metadata operations for the creation and deletion of all tables within the entire database.

[0047] When the target data node Vnode receives the first consumption request from the target client Consumer1, it immediately records a snapshot version number of the current database. Using the snapshot version number, it divides the data stream into historical data and real-time data (that is, time-series data written to the database before the snapshot version number is classified as historical data; time-series data written to the database after the snapshot version number is classified as real-time data). It then uses the storage engine tsdb to save the historical data and uses the write-ahead log WAL to save the real-time data.

[0048] The target data node Vnode reads the data stored in the database according to the data granularity defined by the subscription topic forwarded by the management node Mnode. Specifically, the target data node Vnode reads historical data that meets the filtering conditions before the snapshot version number from the storage engine tsdb, and reads real-time data that meets the filtering conditions after the snapshot version number from the write-ahead log wal.

[0049] The subscription mechanism, principle, and effects of this invention will be described in detail below.

[0050] This invention constructs a collaborative data subscription system based on a time-series database. The core operating mechanism of this system lies in the unified scheduling and coordination of the data consumption process through a centralized management node (Mnode), thereby achieving efficient and dynamic matching between data nodes and consumers.

[0051] The overall technical process of this invention can be summarized into the following three key stages:

[0052] Subscription Scheduling and Resource Allocation Phase: The process begins with a consumer initiating a subscription request to the management node (Mnode). The Mnode, acting as the system's "brain," does not directly process data but specializes in resource management and scheduling. Upon receiving requests from multiple consumers, the Mnode executes the core load balancing algorithm, fairly allocating data nodes (Vnodes) to each consumer and synchronizing this allocation decision to the corresponding data nodes, thus completing the initial mapping of consumption tasks.

[0053] Connection or data channel establishment and confirmation phase: After confirming receipt of the allocation instruction from the management node, the data node (Vnode) returns an acknowledgment signal to the Mnode. Subsequently, the Mnode sends a "subscription successful" notification along with the allocated data node address to the consumer. This is equivalent to providing each consumer with a "data navigation map," guiding them to establish a point-to-point consumption channel directly with the target data node.

[0054] Data retrieval and filtering execution phase: After receiving clear address guidance, consumers can directly send consumption requests to the designated data nodes (Vnodes) without going through Mnodes. Data nodes act as "warehouses and processing stations" for data. Based on the data granularity defined by the subscribed topic (such as specific columns and filtering conditions), they simultaneously read data from persistent storage (TSDB) and write-ahead logs (WAL), perform real-time filtering, and ultimately continuously deliver precisely matched data streams to consumers.

[0055] The data subscription mechanism of this invention is mainly implemented through the collaborative work of the following key technical modules:

[0056] Topic module: Flexible definition of data granularity

[0057] Consumers first create a topic using SQL-like (Structured Query Language) syntax to precisely define the scope of the required data. This is the cornerstone for achieving flexible and controllable data granularity. Topics can be created in the following three ways:

[0058] 1. Query topic: Suitable for fine-grained subscriptions that require specific business fields or data that meet specific conditions.

[0059]

[0060] Column projection: SELECT c1, c3 specifies that only columns c1 and c3 are retrieved from the source table tb_name. Other columns (such as c2) are filtered out before data distribution, reducing the amount of data transmitted over the network.

[0061] Predicate filtering: WHERE c1 > 10 is a filtering condition. After reading the original data, the data node will apply this condition to judge each row and only output the rows that satisfy the condition c1 > 10.

[0062] 2. Super Meter Topic: Suitable for monitoring a specific type of equipment (such as all sensors of the same model), and allows selection of whether to detect changes in the group of equipment.

[0063]

[0064] SuperTable (STable) Abstraction: A supertable is a table template in a time-series database, with its actual data stored in multiple sub-tables with the same structure. This Topic type allows users to subscribe to the entire supertable system.

[0065] Metadata (META) option: WITH META is a key option. When enabled, subscribers not only receive data records but also metadata change information for all child tables, such as the creation of new child tables or modifications to child table tags. This is crucial for downstream systems that need to maintain synchronization with the database metadata.

[0066] Tag filtering: `WHERE tag1='value'` filters the sub-table. The condition here is based on tags defined in the super-table, not on the data columns. The system will only distribute data from sub-tables whose tags satisfy `tag1='value'` to consumers.

[0067] 3. Database topic: Primarily used for migrating the entire database, disaster recovery backup, or building a full data warehouse.

[0068]

[0069] Database-level subscription: This is the coarsest-grained subscription method, which means that all data changes in all tables (including all sub-tables under ordinary tables and super tables) under the target database are subscribed to.

[0070] Synergy with META: Although not explicitly shown in the example, database topics can also be combined with the WITH META option to capture metadata operations such as creation and deletion of all tables in the entire database.

[0071] Management Node (Mnode) Module: The Coordination Center of the System

[0072] Mnode does not store actual user data; instead, it manages cluster metadata and coordinates resource allocation, acting as the "brain" of the entire subscription system. Load balancing (rebalance) is a crucial function of Mnode, ensuring fair workload distribution among multiple consumers. Mnode triggers rebalance when there are changes in the members of a consumer group (e.g., a new consumer joins, or an existing consumer goes offline abnormally). See also... Figure 2 The detailed steps of the algorithm are as follows:

[0073] Parameter calculation: Let M be the total number of Vnodes where the Topic data is distributed, and N be the total number of consumers.

[0074] Baseline allocation: Calculate the baseline number of Vnodes that should be allocated to each consumer, S = floor(M / N), and the remaining number of Vnodes that cannot be allocated equally, R = M % N.

[0075] Allocation strategy:

[0076] The system will identify R consumers, each of which will be allocated S + 1 Vnodes.

[0077] The remaining N - R consumers are each assigned S Vnodes.

[0078] Smooth migration: During reallocation, Mnodes will reuse the previous allocation results as much as possible, operating only on Vnodes that need to be migrated, in order to reduce data turbulence. For example, a Consumer that originally had S+2 Vnodes may only need to retain S+1 after rebalancing, with the extra Vnode being allocated to other Consumers.

[0079] Coordination Notification: The Mnode will notify all relevant Vnodes and all Consumers of the final allocation plan. Vnodes are informed which Consumer they need to serve, and Consumers receive a list of Vnode addresses they need to connect to.

[0080] Data Node (Vnode) Module: The unit for executing data.

[0081] A Vnode is the node that actually stores and processes data, responsible for responding to direct data requests from consumers. To balance the integrity of historical data consumption with the low latency of real-time data consumption, a Vnode consumption model combining TSDB and WAL is adopted.

[0082] 1. Snapshot Version Number

[0083] When to record: When a Vnode receives the first consumption request from a Consumer, it will immediately record a snapshot version number of the current database.

[0084] Core function: This version number serves as a watershed, clearly dividing the data stream into "historical" and "real-time" parts.

[0085] 2. TSDB Consumption (Historical Data)

[0086] Data source: TSDB is an optimized columnar storage engine responsible for persistently storing historical data.

[0087] Consumption method: Vnode generates a list of table IDs that are currently valid based on the snapshot version number, and then scans the data files and metadata files in TSDB in order according to this list.

[0088] Progress identifier: In TSDB, the consumption progress is identified by a tuple (Table_ID, Timestamp), which precisely records which table and which point in time has been consumed.

[0089] 3. WAL consumption (real-time data)

[0090] Data source: WAL (Write-Ahead Log) is a standard component of databases. All write requests are first sequentially appended to the WAL file before being written to the TSDB. It is essentially an append-only message queue.

[0091] Consumption method: Starting from the snapshot version number of the record, the Vnode continuously listens for and sequentially reads new records in the WAL.

[0092] Progress indicator: Consumption progress in WAL is indicated by a monotonically increasing version number.

[0093] Data filtering and transmission: Regardless of whether the data comes from TSDB or WAL, Vnode will execute the filtering logic (column projection, condition judgment) defined by the Topic in memory in real time, and encapsulate the final result into a data stream and push it to the Consumer directly connected to it.

[0094] Consumption progress management (Offset) module: Ensuring reliability

[0095] To ensure the "at least once" or "exact once" semantics of data consumption, the system provides a robust consumption progress management mechanism.

[0096] 1. Abstraction and representation of Offset

[0097] Offset is a standardized representation of consumption progress. However, its specific meaning varies depending on the data source.

[0098] TSDB Offset: It is represented as (Table_ID, Timestamp) and is multidimensional because it needs to track different consumption times of multiple tables.

[0099] WAL Offset: Represented as Version, it is one-dimensional because WAL is a globally ordered log stream.

[0100] 2. Progress Maintenance and Submission

[0101] Client-side logging: During consumption, the Consumer receives Offset information from the Vnode and maintains it in memory. This ensures that within the same consumption session, the next data fetch will continue directly from the previous position, avoiding duplication.

[0102] Server-side persistence: To achieve cross-session persistence (i.e., the consumer can continue even after restarting), the consumer needs to periodically or before exiting commit its current consumption progress to the server (Mnode or a dedicated coordinator). The committed offset will be persistently stored.

[0103] Submission method:

[0104] ▪ Auto-commit: This is executed automatically and periodically in the background by the client SDK. It is simple but may cause a small number of duplicates or losses in the event of a failure.

[0105] ▪ Manual commit: This is explicitly called by the application logic after successfully processing a batch of data, providing more precise control and enabling a higher level of semantics.

[0106] 3. Resume interrupted download

[0107] When a new Consumer instance is assigned a Vnode previously managed by another Consumer, or when the same Consumer restarts, it queries the server for the latest committed offset on that Vnode and starts consuming from there, thus achieving seamless breakpoint resumption.

[0108] Specific implementation examples: See Figure 3 Suppose a power company uses this time-series database system to store current data from tens of thousands of monitoring points across the country. Each monitoring point reports one data point per second, containing fields such as timestamp, device_id, current, and voltage.

[0109] Currently, the company's safety monitoring center needs to monitor all devices experiencing current overload (assuming a threshold of 220A) in real time to issue timely alerts. Due to the massive amount of data, the processing capacity of a single consumer is limited, necessitating the collaboration of two consumers.

[0110] Define a data subscription topic.

[0111] The administrator of the security monitoring center first creates a subscription topic named topic_high_current_alarm, defined as follows:

[0112]

[0113] This is a query topic.

[0114] The SELECT ... clause specifies the data granularity for subscription as four fields: timestamp, device_id, current, and voltage. Other fields (such as possible phase) will be filtered out.

[0115] The WHERE current >= 220 is the core filtering condition, ensuring that only abnormal data with a current value greater than or equal to 220A is distributed, while normal data is ignored. This greatly reduces the amount of data transmitted over the network and processed by consumers.

[0116] Management Node (Mnode) Coordination and Resource Allocation

[0117] 1. Subscription Request: Two consumer programs, consumer1 and consumer2, start up and send a request to the management node (Mnode) to subscribe to topic_high_current_alarm.

[0118] 2. Load balancing (Rebalance):

[0119] Mnode discovered that the data involved in the topic_high_current_alarm (i.e., all data in the current_measurements table with current >= 220) is actually distributed across two data nodes: vnode1 and vnode2.

[0120] Mnode executes the Rebalance algorithm: Total number of Vnodes M=2, number of consumers N=2. Calculation yields S = 2 / 2 = 1, R = 0.

[0121] Mnodes are allocated fairly: vnode1 is assigned to consumer1, and vnode2 is assigned to consumer2.

[0122] Routing notification: The Mnode notifies vnode1, vnode2, consumer1, and consumer2 of the allocation results respectively. At this point, consumer1 knows it needs to connect to vnode1, and consumer2 knows it needs to connect to vnode2.

[0123] Data processing and consumption of data nodes (Vnodes)

[0124] For consumer1 and vnode1:

[0125] 1. Establish connection: Based on the received address, consumer1 directly sends a consumption connection request to vnode1.

[0126] 2. Recording Snapshots: Upon receiving a request, vnode1 records the current database snapshot version number (e.g., version_1000). This version number serves as the dividing line between historical and real-time data.

[0127] 3. Dual-engine data acquisition and filtering:

[0128] TSDB Consumption (Historical Data): vnode1 reads all data belonging to the current_measurements table prior to version_1000 from the persistent storage file (TSDB). While reading each data entry, the Topic filter condition current >= 220 is applied in real-time. Assuming two historical records meeting the condition are filtered out, they are sent to consumer1.

[0129] WAL Consumption (Real-time Data): Simultaneously, vnode1 continuously monitors the Write-Ahead Log (WAL) starting from version_1000. All newly written data with a current value greater than or equal to 220A is immediately retrieved from the WAL by vnode1 and filtered again. Assume that two more real-time records meeting the criteria are subsequently filtered out and sent to consumer1.

[0130] 4. Result: Consumer1 ultimately received a total of 4 current overload alarm data from vnode1.

[0131] For consumer2 and vnode2:

[0132] The process is completely parallel and independent of the above:

[0133] Consumer2 is directly connected to vnode2. vnode2 records its own snapshot version number. vnode2 reads and filters data that satisfies current >= 220 from its own TSDB and WAL. Assuming 3 records are filtered from TSDB and 2 from WAL, consumer2 ultimately receives a total of 5 current overload alarm data from vnode2.

[0134] Results Summary and Consumption Progress Management

[0135] Data aggregation: The background service of the safety monitoring center aggregated the results of consumer1 and consumer2, obtaining a total of 9 current overload data, realizing the monitoring of all abnormal data.

[0136] Offset Management:

[0137] During the consumption process, consumer1 and consumer2 will record their consumption progress (Offset) on vnode1 and vnode2 respectively. For example, consumer1 will record: "I have consumed up to (t1,1735660802) in TSDB on vnode1 and up to version number 1500 in WAL."

[0138] They periodically commit this progress to the server. If a consumer program crashes and restarts, it can continue consuming from the last committed progress, without missing any alerts or processing duplicates.

[0139] Extension of the Implementation Examples: Flexibility of Granular Control

[0140] The flexibility of this mechanism doesn't stop there. For example, if the security monitoring center only wants to analyze overload conditions on New Year's Day, 2025, it only needs to modify the Topic definition to:

[0141]

[0142] The system will only distribute data related to current overload on January 1, 2025. This again demonstrates the core advantage of pre-filtering at the server (Vnode), enabling flexible control over data granularity and significantly reducing invalid data transmission.

[0143] Although the present invention has been described in detail above, it is not limited thereto, and those skilled in the art can make various modifications based on the principles of the present invention. Therefore, all modifications made in accordance with the principles of the present invention should be understood to fall within the protection scope of the present invention.

Claims

1. A data subscription method with flexible and controllable data granularity, comprising: setting a plurality of data nodes Vnode for storing time-series data on a server side; setting a management node Mnode for managing the plurality of data nodes Vnode on the server side; when a target client Consumer1 in a plurality of clients Consumers needs to subscribe to time-series data, the target client Consumer1 sends a time-series data subscription request containing a subscription topic Topic to the management node Mnode on the server side, wherein the subscription topic Topic defines a data type and a data granularity; the management node Mnode allocates a target data node with the data type defined by the subscription topic Topic in the target client Consumer1 subscription request to the target client Consumer1, and establishes a point-to-point data channel between the target client Consumer1 and the target data node; the target data node sends the saved time-series data to the target client Consumer1 through the data channel according to the data granularity defined by the subscription topic Topic. 2.The data subscription method with flexible and controllable data granularity according to claim 1, wherein the management node Mnode allocates a target data node with the data type defined by the subscription topic Topic in the target client Consumer1 subscription request to the target client Consumer1 comprises: the management node Mnode finds data nodes Vnode storing the data type defined by the subscription topic Topic in the target client Consumer1 subscription request, and allocates one or more data nodes Vnode found to the target client Consumer1 as target data nodes according to a load balancing algorithm. 3.The data subscription method with flexible and controllable data granularity according to claim 2, wherein the management node Mnode establishes a point-to-point data channel between the target client Consumer1 and the target data node comprises: the management node Mnode forwards the subscription topic Topic to the target data node, and sends the address of the target data node to the target client Consumer1, so that the target client Consumer1 establishes a point-to-point data channel with the target data node through the address of the target data node. 4.The data subscription method with flexible and controllable data granularity according to claim 3, wherein the target data node sends the saved time-series data to the target client Consumer1 through the data channel according to the data granularity defined by the subscription topic Topic comprises: the target client Consumer1 sends a consumption request to the target data node using the address of the target data node, so as to establish the data channel between the target client Consumer1 and the target data node. The target data node reads the time-series data stored in the database according to the data granularity defined by the subscription topic forwarded by the management node Mnode, and transmits the read time-series data to the target customer Consumer1 via the data channel.

5. The data subscription method with flexible and controllable data granularity according to claim 4, wherein the subscription topic includes query topic topics applicable to specific business fields or specific condition data; super table topic topics applicable to monitoring a certain type of equipment; and database topics used for database migration, disaster recovery backup, or building a full data warehouse.

6. The data subscription method with flexible and controllable data granularity according to claim 5, wherein the query topic is used to subscribe to or specify one or more data columns in the source table or the original data table and set data column filtering conditions, so that the target data node distributes one or more specified data columns to the target customer Consumer1 according to the data column filtering conditions.

7. The data subscription method with flexible and controllable data granularity according to claim 5, wherein the super table topic is used to subscribe to or specify the entire super table system, sub-table metadata change information, or sub-tables that meet the tag filtering conditions, so that the target data node distributes the specified entire super table system, sub-table metadata change information, or sub-tables that meet the tag filtering conditions to the target customer Consumer1.

8. The data subscription method with flexible and controllable data granularity according to claim 5, wherein the database Topic is used to subscribe to or specify all data changes of all tables in the target database or metadata operations of creation and deletion of all tables in the target database, so that the target data node will distribute the subscribed entire super table system, sub-table metadata change information or sub-tables that meet the tag filtering conditions to the target customer Consumer1.

9. In the data subscription method with flexible and controllable data granularity according to claim 4, when the target data node receives the first consumption request from the target customer Consumer1, it immediately records a snapshot version number of the current database, uses the snapshot version number to divide the data stream into historical data and real-time time-series data; and uses the storage engine tsdb to save the historical data, and uses the write-ahead log wal to save the real-time time-series data.

10. The data subscription method with flexible and controllable data granularity according to claim 9, wherein the target data node reads the data stored in the database according to the data granularity defined by the subscription topic forwarded by the management node Mnode, specifically: the target data node reads historical time-series data that meets the filtering conditions before the snapshot version number from the storage engine tsdb, and reads real-time time-series data that meets the filtering conditions after the snapshot version number from the write-ahead log wal.

Images