Real-time data subscription method and device based on time sequence database and medium

Abstract

The invention discloses a real-time data subscription method and device based on a time sequence database and a medium, belongs to the technical field of databases, and aims to solve the technical problems that the real-time performance is poor, the resource consumption is large and the time sequence data required by the current time sequence database in real time acquisition in mass data storage and processing is poor. According to the technical scheme, the method comprises the steps that a data subscription scheduling task is constructed through a predefined SQL statement, in combination with an efficient data change capture module based on a Raft log mechanism, data subscribed by a user is obtained in real time, and the data subscribed by the user is filtered according to a customization rule; then the screened data is automatically transmitted to a preset target sink component in a specified format, so that the whole data processing flow has the characteristics of real-time performance, accuracy and high availability; the method specifically comprises the following steps: constructing a data subscription scheduling task; executing the data subscription scheduling task; and capturing data change based on the raft log.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a real-time data subscription method, device and medium based on a time series database. Background Art

[0002] KaiwuDB is a distributed, multi-modal fusion database product designed specifically for AIoT scenarios, with built-in native AI support capabilities. The system can build both time series and relational databases in a single instance, and implement comprehensive processing of multiple data models. In addition, KaiwuDB has excellent time series data management performance, which can meet the needs of tens of millions of devices accessing, millions of data writing in seconds, and hundreds of millions of data reading in seconds.

[0003] In the field of AIoT, time series data often exhibits high-frequency generation and real-time characteristics. As the number of connected devices and the data streams they generate continue to grow, how to quickly and effectively obtain time series information has become a key technical challenge. Traditional time series databases usually rely on batch processing or periodic polling methods to obtain real-time data, but this method has the following shortcomings:

[0004] ① Lack of timeliness: Due to the traditional mechanism of data query at scheduled time intervals, the speed of data update is limited, which means that users must wait until the next scheduled query point to receive the latest information change notification, thereby adding unnecessary delays and failing to meet the requirements of real-time data analysis.

[0005] ② Low resource utilization efficiency: Frequent polling operations not only increase the workload of the database server, but also consume a lot of computing resources. Especially when facing large-scale data sets and highly concurrent access requests, the overall system performance will be significantly reduced. Summary of the invention

[0006] The technical task of the present invention is to provide a real-time data subscription method, device and medium based on a time series database to solve the problem of poor real-time performance and high resource consumption in the current time series database in real-time acquisition of required time series data in massive data storage and processing.

[0007] The technical task of the present invention is achieved in the following way: a real-time data subscription method based on a time series database, which constructs a data subscription scheduling task through a predefined SQL statement, combines an efficient data change capture module based on the Raft log mechanism, obtains the data subscribed by the user in real time, and filters the data subscribed by the user according to customized rules; then automatically transmits the filtered data in a specified format to a preset target sink component, thereby ensuring that the entire data processing process has the characteristics of real-time, accuracy and high availability; the details are as follows:

[0008] Build data subscription scheduling tasks;

[0009] Execute data subscription scheduling tasks;

[0010] Data change capture based on raft log.

[0011] As a preferred method, the data subscription scheduling task is constructed as follows:

[0012] Specify data subscription rules: users specify data subscription rules through specified SQL statements;

[0013] Record and manage task information: After the user submits the SQL statement, the database will store the parsed task information in the system table system.topics of the database;

[0014] Construct execution operators: The database constructs an execution plan based on the task information obtained through analysis, and constructs related execution operators through the execution plan. The execution operators include TOPIC execution operators and subquery execution operators. The TOPIC execution operator is used to process the capture of data changes, the transmission of data streams, and the output of final results. The subquery execution operator is used to filter and transform the captured data according to the subquery execution plan to ensure that the data meets user needs.

[0015] Preferably, the format of the SQL statement is as follows:

[0016] CREATE TOPIC[IF NOT EXISTS]<topic_name> INTO <sink>[WITH <option>＝ <value>]as <subquery>;

[0017] in, <sink>Used to specify the connection information of the data receiving component; sink is the HTTP URL of Webhooks or other supported receiving endpoints, through which the real-time captured data is pushed to the external system or component in the specified format for subsequent processing;

[0018] <option>＝ <value>Indicates the optional parameters in the data subscription rule. Users can specify the following parameters as needed:

[0019] ① Data format: specifies the transmission format of data, supporting json or xml formats; the data format determines the encoding method of data during transmission and is suitable for the parsing requirements of different recipients;

[0020] ②Operation type output (operate_type_out): determines whether to output the type of data operation, the output data operation type is insert (INSERT), update (UPDATE) or delete (DELETE), the default is false, that is, the operation type is not output;

[0021] ③Before_data_out: determines whether to output the original value before the data change. The default value is false. When the option of before_data_out is true, the value before the change will be output when the data change is captured, helping the receiving end understand the historical status of the data.

[0022] <subquery>Indicates specifying a query statement, which is a query on the time series table; users can filter and select data in the corresponding subquery according to actual needs, specify columns or conditions, and ensure that the subscribed data meets specific requirements.

[0023] Preferably, the task information includes the following:

[0024] ①Subscription topic name (topic_name): a unique identifier used to identify the current data subscription task;

[0025] ②Task status: indicates the current execution status of the data subscription task; the current execution status of the data subscription task includes pending, executing and completed;

[0026] ③Sink information: records the connection information of the data receiving component (such as the URL of Webhooks, database connection information, etc.);

[0027] ④Rule parameter information: including data format, operation type output options and pre-change data output options;

[0028] ⑤ Subquery SQL statement: record the query statement related to the corresponding subscription task for reference during execution;

[0029] ⑥ Subquery execution plan: The execution plan generated based on the subquery statement indicates how to execute the corresponding query to maximize performance;

[0030] ⑦Current watermark: used to track the progress of data flow and ensure the consistency of capturing and synchronizing data changes.

[0031] As a preferred method, the data subscription scheduling task is executed as follows:

[0032] Task execution initialization: After the TOPIC execution operator is initialized, the TOPIC execution operator builds a data change capture request based on the current waterline information and table information, and then sends the capture request to the big data change capture module; at the same time, the database sets a timeout timer (usually 100ms) for each captured object, waiting for the captured data change event; the timeout timer is used to monitor whether there is new change data within a certain period of time, so as to respond in time;

[0033] Processing event messages: Event messages include data event messages and end event messages;

[0034] Convert and output data format: The subquery execution operator processes the data according to the rules of the subscription task and the information of the output data column; when the processing is completed, the data is converted to the specified json or xml format, and the formatted data is sent to the specified sink component to realize data output and distribution.

[0035] Preferably, the capture request information is as follows:

[0036] ① Table ID (table_ID): identifies the specific table that needs to capture data changes;

[0037] ②Data change capture time range (start_time, end_time): defines the time range of captured data changes;

[0038] ③Event data receiving channel (channel): specifies the transmission channel for data change capture results.

[0039] Preferably, the data event message is as follows: when the data event type read from the channel is data, the subquery execution operator is called to further process the data. The subquery execution operator filters and processes the event data according to the user-defined filtering conditions and output column rules to ensure that only data that meets the conditions is further processed;

[0040] The end event message is as follows: When the data event type read from the channel is end, it indicates that the data change capture task has been completed. At this time, the watermark information is updated to ensure that the next task starts from the correct state, and the updated watermark information is written to the system.topics table.

[0041] As a preferred method, data change capture based on raft log is as follows:

[0042] In KaiwuDB, Raft logs record database status changes in real time, including all operations that affect the database status. Raft logs accurately identify each operation record through timestamps and indexes, and provide a reliable basis for subsequent data change capture.

[0043] The data change capture module uses the timestamp and index information in the Raft log to quickly locate the log entries that meet the conditions according to the table ID and time range information sent by the TOPIC execution operator, and extracts the relevant change data and operation type (such as INSERT, UPDATE, DELETE) from the Raft log, and encapsulates the change data into an event message (EventMsg) and sends it to the receiving channel of the TOPIC execution operator;

[0044] When the data change capture module completes the capture of all data changes within the specified time range, it sends an end type event message (EventMsg) to the result channel, marking the end of the current data change capture request task.

[0045] An electronic device comprising: a memory and at least one processor;

[0046] Wherein, the memory stores a computer program;

[0047] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the real-time data subscription method based on the time series database as described above.

[0048] A computer-readable storage medium stores a computer program, which can be executed by a processor to implement the real-time data subscription method based on a time series database as described above.

[0049] The real-time data subscription method, device and medium based on the time series database of the present invention have the following advantages:

[0050] (1) The present invention allows users to subscribe to specific types of data according to their own needs, and receive instant notifications when relevant data changes, thereby effectively overcoming the limitations of the traditional polling mode and solving the problems of high latency and high resource consumption commonly found in existing time series databases;

[0051] (ii) The present invention adopts an event-driven data push strategy to ensure that information updates can be quickly communicated to subscribers, greatly improving the timeliness of data processing and enhancing real-time response capabilities;

[0052] (III) The present invention optimizes resource allocation efficiency and reduces system load by streamlining unnecessary query requests, thereby significantly improving the overall resource utilization efficiency;

[0053] (iv) The task information of the present invention not only helps the system manage various data subscription tasks, but also provides a basis for real-time monitoring and scheduling; at the same time, through the management of the system table, users can view the execution status of subscription tasks at any time and perform management operations such as pausing and resuming tasks;

[0054] (V) The TOPIC execution operator and subquery execution operator of the present invention are the basis for subsequent task scheduling and execution, ensuring the efficiency and accuracy of data capture and processing;

[0055] (VI) The present invention dynamically captures and processes data changes in a time series database. It supports flexible data query methods and customized output formats, while ensuring the real-time, accuracy and overall system performance of the data capture process;

[0056] (VII) The present invention utilizes the log mechanism of the Raft consensus algorithm to keep the data capture operation synchronized with the state of the entire system. Even in the face of system failure or recovery, all changes that have occurred can be fully tracked through the Raft log, thereby ensuring data consistency and integrity. In addition, by combining the standard SQL query language with highly configurable subscription rules, users can adjust data collection strategies according to their specific needs, thereby optimizing database performance and simplifying data processing steps.

[0057] (VIII) In terms of design, considering the task scheduling problem in a high-concurrency environment and the need for real-time response, the present invention adopts a sophisticated watermark control technology and an efficient query execution scheme, which can continuously provide high-quality services in large-scale data flow scenarios, and manages the status information, progress tracking and change result reporting of each subscription activity through a specially established system table system.topics, further enhancing the monitoring capability and controllability of the execution of subscription tasks;

[0058] (IX) The present invention not only meets the user's high standards for data collection accuracy and timeliness, but also has excellent efficiency performance and good expansion capabilities; therefore, it is very suitable for application in the financial industry, the Internet of Things technology field, intelligent manufacturing and other aspects, for realizing functions such as real-time data monitoring, alarm notification and data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention is further described below in conjunction with the accompanying drawings.

[0060] Attached Figure 1 A flowchart for building data subscription scheduling tasks;

[0061] Attached Figure 2 A flowchart for executing data subscription scheduling tasks. DETAILED DESCRIPTION

[0062] The real-time data subscription method, device and medium based on a time series database of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments of the specification.

[0063] Embodiment 1:

[0064] This embodiment provides a real-time data subscription method based on a time series database. The method constructs a data subscription scheduling task through a predefined SQL statement, combines an efficient data change capture module based on the Raft log mechanism, obtains the data subscribed by the user in real time, and filters the data subscribed by the user according to customized rules; then automatically transmits the filtered data in a specified format to a preset target sink component, thereby ensuring that the entire data processing flow has the characteristics of real-time, accuracy and high availability; the details are as follows:

[0065] S1. Build data subscription scheduling tasks;

[0066] S2. Execute data subscription scheduling task;

[0067] S3. Data change capture based on raft log.

[0068] As attached Figure 1 As shown, the data subscription scheduling task in step S1 of this embodiment is specifically as follows:

[0069] S101, specifying data subscription rules: the user specifies data subscription rules through a specified SQL statement;

[0070] S102, recording and managing task information: after the user submits the SQL statement, the database stores the parsed task information in the system table system.topics of the database;

[0071] S103, constructing execution operators: the database constructs an execution plan according to the task information obtained through analysis, and constructs related execution operators through the execution plan; wherein, the execution operators include TOPIC execution operators and subquery execution operators; the TOPIC execution operator is used to process the capture of data changes, the transmission of data streams and the output of final results; the subquery execution operator is used to filter and transform the captured data according to the execution plan of the subquery to ensure that the data meets the user's needs.

[0072] The format of the SQL statement in this embodiment is as follows:

[0073] CREATE TOPIC[IF NOT EXISTS]<topic_name> INTO <sink>[WITH <option>＝ <value>]as <subquery>;

[0074] in, <sink>Used to specify the connection information of the data receiving component; sink is the HTTP URL of Webhooks or other supported receiving endpoints, through which the real-time captured data is pushed to the external system or component in the specified format for subsequent processing;

[0075] <option>＝ <value>Indicates the optional parameters in the data subscription rule. Users can specify the following parameters as needed:

[0076] ① Data format: specifies the transmission format of data, supporting json or xml formats; the data format determines the encoding method of data during transmission and is suitable for the parsing requirements of different recipients;

[0077] ②Operation type output (operate_type_out): determines whether to output the type of data operation, the output data operation type is insert (INSERT), update (UPDATE) or delete (DELETE), the default is false, that is, the operation type is not output;

[0078] ③Before_data_out: determines whether to output the original value before the data change. The default value is false. When the option of before_data_out is true, the value before the change will be output when the data change is captured, helping the receiving end understand the historical status of the data.

[0079] <subquery>Indicates specifying a query statement, which is a query on the time series table; users can filter and select data in the corresponding subquery according to actual needs, specify columns or conditions, and ensure that the subscribed data meets specific requirements.

[0080] The task information in step S102 of this embodiment includes the following contents:

[0081] ①Subscription topic name (topic_name): a unique identifier used to identify the current data subscription task;

[0082] ②Task status: indicates the current execution status of the data subscription task; the current execution status of the data subscription task includes pending, executing and completed;

[0083] ③Sink information: records the connection information of the data receiving component (such as the URL of Webhooks, database connection information, etc.);

[0084] ④Rule parameter information: including data format, operation type output options and pre-change data output options;

[0085] ⑤ Subquery SQL statement: record the query statement related to the corresponding subscription task for reference during execution;

[0086] ⑥ Subquery execution plan: The execution plan generated based on the subquery statement indicates how to execute the corresponding query to maximize performance;

[0087] ⑦Current watermark: used to track the progress of data flow and ensure the consistency of capturing and synchronizing data changes.

[0088] As attached Figure 2 As shown, the execution data subscription scheduling task in step S2 of this embodiment is specifically as follows:

[0089] S201, task execution initialization: After the TOPIC execution operator is initialized, the TOPIC execution operator constructs a data change capture request according to the current waterline information and table information, and then sends the capture request to the big data change capture module; at the same time, the database sets a timeout timer (usually 100ms) for each captured object, waiting for the captured data change event; wherein, the timeout timer is used to monitor whether there is new change data within a certain period of time, so as to respond in time;

[0090] S202, processing event messages: event messages include data event messages and end event messages;

[0091] S203, convert and output data format: the subquery execution operator processes the data according to the rules of the subscription task and the information of the output data column; when the processing is completed, the data is converted into the specified json or xml format, and the formatted data is sent to the specified sink component to realize data output and distribution.

[0092] The information of the capture request in step S201 of this embodiment is specifically as follows:

[0093] ① Table ID (table_ID): identifies the specific table that needs to capture data changes;

[0094] ②Data change capture time range (start_time, end_time): defines the time range of captured data changes;

[0095] ③Event data receiving channel (channel): specifies the transmission channel for data change capture results.

[0096] The data event message in step S202 of this embodiment is specifically as follows: when the data event type read from the channel is data, the subquery execution operator is called to further process the data. The subquery execution operator filters and processes the event data according to the user-defined filtering conditions and output column rules to ensure that only data that meets the conditions will be further processed.

[0097] The end event message in step S202 of this embodiment is specifically: when the data event type read from the channel is end, it means that the data change capture task has been completed; at this time, the watermark information is updated to ensure that the next task starts from the correct state, and the updated watermark information is written into the system.topics table.

[0098] The data change capture based on the raft log in step S3 of this embodiment is specifically as follows:

[0099] S301. In the KaiwuDB, the Raft log records the database status changes in real time, including all operations that affect the database status. The Raft log accurately identifies each operation record through timestamps and indexes, and provides a reliable basis for subsequent data change capture.

[0100] S302: The data change capture module uses the timestamp and index information in the Raft log to quickly locate the log entry that meets the conditions according to the table ID and time range information sent by the TOPIC execution operator, and extracts the relevant change data and operation type (such as INSERT, UPDATE, DELETE) from the Raft log, and encapsulates the change data into an event message (EventMsg) and sends it to the receiving channel of the TOPIC execution operator.

[0101] S303: After the data change capture module completes capturing all data changes within the specified time range, it sends an end type event message (EventMsg) to the result channel, marking the end of the current data change capture request task.

[0102] Embodiment 2:

[0103] This embodiment also provides an electronic device, including: a memory and a processor;

[0104] Wherein, the memory stores computer-executable instructions;

[0105] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the real-time data subscription method based on the time series database in any embodiment of the present invention.

[0106] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0107] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0108] Embodiment 3:

[0109] This embodiment also provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions are loaded by a processor, so that the processor executes the real-time data subscription method based on a time series database in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0110] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0111] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0112] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0113] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.< / subquery> < / value> < / option> < / sink> < / subquery> < / value> < / option> < / sink> < / subquery> < / value> < / option> < / sink> < / subquery> < / value> < / option> < / sink>

Claims

1. A real-time data subscription method based on a time series database, characterized in that: The method is to build a data subscription scheduling task through predefined SQL statements, combined with an efficient data change capture module based on the Raft log mechanism, to obtain the data subscribed by users in real time, and filter the data subscribed by users according to customized rules; The filtered data is then automatically transferred to the preset target sink component in the specified format, thereby ensuring that the entire data processing process has the characteristics of real-time, accuracy and high availability; the details are as follows: Build data subscription scheduling tasks; Execute data subscription scheduling tasks; Data change capture based on raft log.

2. The real-time data subscription method based on a time series database according to claim 1 is characterized in that: The specific tasks for building data subscription scheduling are as follows: Specify data subscription rules: users specify data subscription rules through specified SQL statements; Record and manage task information: After the user submits the SQL statement, the database will store the parsed task information in the system table system.topics of the database; Construct execution operators: The database constructs an execution plan based on the task information obtained through analysis, and constructs related execution operators through the execution plan. The execution operators include TOPIC execution operators and subquery execution operators. The TOPIC execution operator is used to process the capture of data changes, the transmission of data streams, and the output of final results; the subquery execution operator is used to filter and transform the captured data according to the subquery execution plan to ensure that the data meets user needs.

3. The real-time data subscription method based on a time series database according to claim 1 or 2, characterized in that: The format of the SQL statement is as follows: CREATE TOPIC[IF NOT EXISTS]<topic_name>INTO <sink>[WITH <option>＝ <value>]as <subquery> ；< / subquery> < / value> < / option> < / sink> in, <sink> Used to specify the connection information of the data receiving component; sink is the HTTP URL of Webhooks or other supported receiving endpoints, through which the real-time captured data is pushed to the external system or component in the specified format for subsequent processing;< / sink> <option>＝ <value> Indicates the optional parameters in the data subscription rule. Users can specify the following parameters as needed:< / value> < / option> ①Data format: specifies the data transmission format, supporting json or xml formats; the data format determines the encoding method of the data during transmission, which is suitable for the parsing requirements of different recipients; ② Operation type output: determines whether to output the type of data operation, the output data operation type is insert, update or delete, the default is false, that is, the operation type is not output; ③ Output data before change: determines whether to output the original value before the data change. The default value is false. When the option of outputting data before change is true, the value before change will be output at the same time when capturing data changes, helping the receiving end understand the historical status of the data. <subquery> Indicates specifying a query statement, which is a query on the time series table; users can filter and select data in the corresponding subquery according to actual needs, specify columns or conditions, and ensure that the subscribed data meets specific requirements.< / subquery> 4. The real-time data subscription method based on a time series database according to claim 2 is characterized in that: Task information includes the following: ①Subscription topic name: a unique identifier used to identify the current data subscription task; ②Task status: indicates the current execution status of the data subscription task; the current execution status of the data subscription task includes pending, executing and completed; ③Sink information: records the connection information of the data receiving component; ④Rule parameter information: including data format, operation type output options and pre-change data output options; ⑤ Subquery SQL statement: record the query statement related to the corresponding subscription task; ⑥ Subquery execution plan: The execution plan generated based on the subquery statement indicates how to execute the corresponding query to maximize performance; ⑦Current waterline: used to track the progress of data flow and ensure the consistency requirements of capturing and synchronizing data changes.

5. The real-time data subscription method based on a time series database according to claim 1 is characterized in that: The execution of data subscription scheduling tasks is as follows: Task execution initialization: After the TOPIC execution operator is initialized, the TOPIC execution operator builds a data change capture request based on the current waterline information and table information, and then sends the capture request to the big data change capture module; at the same time, the database sets a timeout timer for each captured object, waiting for the captured data change event; the timeout timer is used to monitor whether there is new change data within a certain period of time, so as to respond in time; Processing event messages: Event messages include data event messages and end event messages; Convert and output data format: The subquery execution operator processes the data according to the rules of the subscription task and the information of the output data column; when the processing is completed, the data is converted to the specified json or xml format, and the formatted data is sent to the specified sink component to realize data output and distribution.

6. The real-time data subscription method based on a time series database according to claim 5 is characterized in that: The capture request information is as follows: ① Table ID: identifies the specific table that needs to capture data changes; ②Data change capture time range: define the time range of captured data changes; ③Event data receiving channel: specifies the transmission channel for data change capture results.

7. The real-time data subscription method based on a time series database according to claim 5 is characterized in that: The data event message is as follows: when the data event type read from the channel is data, the subquery execution operator is called to further process the data. The subquery execution operator filters and processes the event data according to the user-defined filtering conditions and output column rules to ensure that only data that meets the conditions is further processed; The end event message is as follows: When the data event type read from the channel is end, it indicates that the data change capture task has been completed. At this time, the watermark information is updated to ensure that the next task starts from the correct state, and the updated watermark information is written to the system.topics table.

8. The real-time data subscription method based on a time series database according to claim 1 is characterized in that: The data change capture based on raft log is as follows: In the open service database, the Raft log records the database status changes in real time, including all operations that affect the database status. The Raft log accurately identifies each operation record through timestamps and indexes, and provides a reliable basis for subsequent data change capture. The data change capture module uses the timestamp and index information in the Raft log to quickly locate the log entries that meet the conditions based on the table ID and time range information sent by the TOPIC execution operator, and extracts the relevant change data and operation type from the Raft log, and encapsulates the change data into an event message and sends it to the receiving channel of the TOPIC execution operator; When the data change capture module completes the capture of all data changes within the specified time range, it sends an end type event message to the result channel, marking the end of the current data change capture request task.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the real-time data subscription method based on the time series database as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the real-time data subscription method based on a time series database as described in any one of claims 1 to 8.

Images