A stream computing implementation method
By splitting stream computing tasks into triggering and computing phases, and combining database persistence and WAL modules, efficient and flexible stream computing is achieved, solving the problems of insufficient flexibility and scalability in existing technologies, and improving system performance and reliability.
Patent Information
- Application Number
- CN202610079806.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-21
- Publication Date
- 2026-05-15
- Estimated Expiration
- 2046-01-21
AI Technical Summary
Existing stream computing engines lack flexibility and scalability, making it difficult to adapt to complex business scenarios and large-scale data processing needs. They are also difficult to develop, have limited computing flexibility, and their state management and checkpoint mechanisms increase system complexity and resource consumption.
The stream computing task is split into two stages: triggering and computation. By decoupling triggering and computation, a single SQL statement is used to define the task. The database persistent storage and WAL module are used for efficient data management. State caching is eliminated, and flexible windowing and grouping strategies are adopted to optimize the computation and output process.
It achieves efficient and flexible streaming computing, improves the system's processing power and resource utilization, reduces development difficulty and maintenance costs, supports rich data processing and analysis functions, and enhances the system's reliability and availability.
Smart Images
Figure CN121560498B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of database technology, and in particular to a method for implementing stream computing for databases, which can efficiently and flexibly realize various functions of stream computing. Background Technology
[0002] With the rapid development of technologies such as the Internet of Things (IoT) and the Industrial Internet, the volume of time-series data (hereinafter referred to as data) has exploded, placing higher demands on real-time data processing and analysis. Traditional database systems, when processing massive amounts of time-series data, often require waiting for the data to accumulate to a certain volume before batch processing, failing to meet the needs of real-time analysis. To address this issue, stream computing technology has emerged. By distributing large amounts of data across each time point, transmitting it in small batches, and performing real-time computation at each time point, real-time data processing is achieved.
[0003] Chinese patent document CN202310447137 discloses a lightweight solution for real-time data processing of time-series databases. Its core principle is to achieve efficient data processing through "SQL-defined rules + timed window calculations." This technology optimizes flexibility, resource utilization, and reliability based on existing stream computing, solving problems in more complex business scenarios.
[0004] Chinese patent document CN202510780484 discloses a "targeted optimization scheme for distributed scenarios," whose core advantage is solving the problems of "special data recalculation + window fragmentation." It boasts stable performance and is suitable for distributed time-series data scenarios with high recalculation accuracy requirements (such as large-scale IoT device monitoring). This technology is an "advanced enhancement solution for all scenarios," with core advantages of "flexibility + resource saving + high reliability." Through decoupling, stateless computation, and state reuse designs, it adapts to complex business scenarios (cross-table, dynamic grouping) and resource-sensitive scenarios, offering more comprehensive functional coverage.
[0005] However, existing stream computing techniques still have some problems:
[0006] Most existing stream computing engines exist in a way similar to materialized views, which lacks flexibility and scalability, making it difficult to adapt to complex business scenarios and large-scale data processing needs.
[0007] Existing stream computing functions often require complex API programming, which increases development difficulty and maintenance costs, limiting the widespread application and promotion of stream computing;
[0008] Existing stream computing engines often require binding complex window and group computations to triggering conditions, which limits the flexibility of computation and makes it difficult to achieve richer data processing and analysis functions.
[0009] Existing state management and checkpoint mechanisms often rely on large amounts of temporary storage, which increases system complexity and resource consumption, and is detrimental to system efficiency and scalability.
[0010] To address these issues, a new approach to streaming computing is urgently needed, capable of efficiently and flexibly implementing various streaming computing functions based on the infrastructure and capabilities of a database system. Summary of the Invention
[0011] The purpose of this invention is to provide a stream computing implementation method that can efficiently and flexibly realize various functions of stream computing, so as to overcome the above-mentioned defects in the prior art.
[0012] A stream computing implementation method of the present invention includes:
[0013] The user sends a stream computation request to the server (or database) defined by trigger, group, compute, and output statements;
[0014] The server parses the trigger, grouping, calculation, and output statements in the stream computing request sent by the user, and generates and persists the triggering information, grouping information, calculation information, and output information.
[0015] After persistently saving trigger information, grouping information, calculation information, and output information, the server begins to deploy stream computing tasks. Specifically, this includes deploying and executing trigger tasks in the trigger phase on the corresponding data nodes, and scheduling and deploying computing tasks according to the calculation information.
[0016] After the server completes the deployment of the stream computing task, it executes the trigger task according to the trigger information; it executes the computing task on the data obtained when executing the trigger task according to the computing information; and it outputs the data computing results completed when executing the computing task according to the output method defined by the output information.
[0017] Preferably, the trigger information persistently stored on the server side is obtained by parsing the trigger statement in the user stream calculation request, including the trigger method, trigger table, trigger event start and end conditions, and trigger filtering conditions; the grouping information persistently stored on the server side is obtained by parsing the grouping statement in the user stream calculation request, which is grouped according to the trigger table sub-table.
[0018] Preferably, the computation information persistently stored on the server side is obtained by parsing the computation statements in the user's stream computation request.
[0019] Preferably, the output information persistently stored on the server side is the output table and data columns obtained by parsing the output statements in the user stream calculation request, which output the data calculation results.
[0020] Preferably, the server executes the triggering task according to the triggering information, including: reading the data columns of all sub-tables of the triggering table from the pre-log WAL of the corresponding data node; determining whether to start the triggering based on the triggering filter conditions; if the triggering is started, determining the start and end times of the triggering method of the data columns of each sub-table according to the triggering event start and end conditions; and outputting the relevant information of the data column triggering method.
[0021] Preferably, the triggering method defined by the start time and end time is an event window defined by the start time and end time, and the relevant information of the triggering method includes the sub-table name to which the window data belongs, the window start time, and the window end time.
[0022] Preferably, the server performs calculation tasks on the data obtained when the trigger task is executed according to the calculation information, including:
[0023] The server starts calculation based on the event window generated by the triggering task. If the triggering task generates multiple event windows, it reads data for all window time periods in batches from the corresponding trigger table according to the start and end times of the multiple event windows, and calculates the read data according to the calculation method to obtain the data calculation results for multiple event windows. If the triggering task generates one event window, it reads data for the time period of the event window from the sub-table of the corresponding trigger table according to the start and end times of the event window, and calculates the read data to obtain the data calculation result for the event window.
[0024] Preferably, the server outputs the data calculation results completed during the execution of the calculation task according to the output method defined in the output information, including: creating an output super table for output; generating an output sub-table in the output super table after the calculation task is completed; and writing the calculation results of the event window into the generated output sub-table.
[0025] Preferably, the server has a trigger and a WAL module for executing triggering tasks; when executing a triggering task, the trigger periodically requests trigger data from the WAL module; the WAL module reads the trigger data from the WAL and performs data filtering, and then returns the filtered data to the trigger; the trigger makes trigger judgments based on triggering methods, conditions, control options, etc., until a triggering event that meets the triggering conditions is found.
[0026] Preferably, the server has one or more computing units for performing computing tasks; after the trigger sends event information that meets the triggering conditions to the computing unit in the form of a computing request, the computing unit starts to perform data computing according to the computing information; after the computing is completed, the computing result is written to the database or pushed to the websocket service; after the writing or pushing is completed, the computing unit returns a computing response.
[0027] Preferably, when performing stream computing tasks, the WAL version number is used as a marker of computing progress; when generating a checkpoint, the WAL version number of the data stream at each data source is recorded, and the computing progress of each group is recorded by timestamp, so that when recovering from a fault, the WAL version number and group computing progress can be obtained from the checkpoint and the stream computing can continue accordingly.
[0028] Preferably, when performing stream computing tasks, the data generated in each time period is stored in the database, and the stored data is read from the database after each time period ends. The read data is then processed, thereby eliminating the status information of the intermediate calculation results generated in each time period.
[0029] Preferably, for the status information that generates intermediate calculation results when performing a stream computing task, the status information is stored in a database when the user selects to apply the status information so that the status information can be used in subsequent calculations.
[0030] This invention breaks down a complete stream computing task into two stages: triggering and computation. The triggering stage is responsible for filtering input data, determining event triggers, generating trigger groups and output tables. Only data records that meet the conditions will enter the event determination stage, and only when the event determination is successful will the next computation stage begin. The computation stage is responsible for performing calculations based on the events generated by the triggers and then outputting the results to the output table.
[0031] By decoupling triggering and computation, the processing capacity of stream computing can be significantly expanded. Users can also define the triggering and computation parts separately, which is clearer in logic and easier to understand. At the same time, for ease of use, this invention uses a single SQL statement to define a complete stream computing task. The statement structure also follows the principle of decoupling triggering and computation, allowing users to naturally write complete statements following the idea of triggering first and then computation.
[0032] Other technical advantages of the present invention are described below.
[0033] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description
[0034] Figure 1 This is a schematic diagram illustrating the main principle of a stream computing implementation method according to the present invention;
[0035] Figure 2 This is a schematic diagram of the server executing the stream computing of this invention. Detailed Implementation
[0036] Figure 1This demonstrates the principle of a stream computing implementation method according to the present invention.
[0037] 1. Create a stream computing task, which includes:
[0038] The system parses user-defined SQL statements to extract trigger types or trigger methods, grouping information, calculation statements, and other information. Through precise parsing of the SQL statements, it extracts key statement elements for subsequent processing and optimization.
[0039] Memory is managed based on a fixed-length space allocation strategy, and a memory database is established to manage streaming events in a unified manner. Through effective memory management strategies, the stability and efficiency of the system when processing large-scale data are ensured.
[0040] The stream computing task is split into two stages: triggering and computation, with separate trigger threads and computation threads. By separating the threads, higher parallel computing capabilities and better resource utilization are achieved.
[0041] 2. Execute the trigger phase calculation, which specifically includes:
[0042] Data filtering, event trigger judgment, grouping, and output table generation are performed based on trigger type or trigger method; strict data filtering and grouping strategies ensure the accuracy and efficiency of data processing.
[0043] The system generates corresponding placeholder information based on the trigger information; the dynamic generation of placeholders improves the system's flexibility and adaptability.
[0044] Data that meets the criteria is stored in an in-memory database; through efficient data storage and management, data access latency and resource consumption are reduced.
[0045] 3. Determine whether batch calculation is required. If so, perform batch calculation; otherwise, perform the calculation phase.
[0046] 4. Perform batch calculations, specifically including:
[0047] The calculation statements are assembled based on trigger information and placeholder information; intelligent statement assembly reduces the possibility of calculation errors.
[0048] Data is grouped and windowed based on timestamp information; flexible time grouping and window configuration can adapt to different business needs and data processing modes.
[0049] Batch calculations are performed using a combination of timing and counting; through optimized calculation strategies, the overall performance and processing capacity of the system are improved.
[0050] The calculation results are stored in the output table; through efficient data writing and management, the integrity and consistency of the data are ensured.
[0051] The window types include timed windows, sliding windows, status windows, event windows, session windows, and counting windows, supporting multiple combinations. The size and steps of the windows can be flexibly configured according to business needs. Through diverse window selection and configuration, it can adapt to different data processing scenarios and business requirements.
[0052] The batch calculation size can be dynamically adjusted according to system resources and business needs, and supports single-threaded or multi-threaded parallel computing; through flexible computing configuration, the utilization rate of system resources and processing efficiency are optimized.
[0053] 5. Perform calculations during the calculation phase, specifically including:
[0054] Read the required data from the database; through an efficient data reading mechanism, the system's response time and resource consumption are reduced.
[0055] The calculation is based on the principle of decoupling triggering and computation; the decoupled computation method improves the system's flexibility and scalability.
[0056] The calculation results are output to the corresponding output table; through an efficient result output mechanism, the timeliness and accuracy of the data are ensured.
[0057] Checkpoint management is performed based on WAL version number and group calculation progress; through intelligent checkpoint mechanism, rapid fault recovery and data backtracking are supported, improving system reliability and availability.
[0058] The intermediate state information required during the calculation process is stored in the database, using a time-segmented storage method, which supports incremental and backtracking calculations. Through efficient intermediate state management, computational redundancy and resource waste are reduced.
[0059] The checkpoint mechanism uses the WAL version number as a marker, supports point-in-time recovery and fault recovery, and reduces reliance on external storage. Through the advanced checkpoint mechanism, it improves the reliability and availability of the system and reduces the risk of data loss.
[0060] Example
[0061] 1. Create a stream computing task
[0062]
[0063] The flow creation statement contains, from left to right, the definitions of the triggering part, the result output part, and the calculation part. The contents of each part are as follows:
[0064] • Trigger section: Trigger type or trigger method, trigger parameters, grouping information, control options;
[0065] • Output section: Result notification information, output table name, output table column definition, output table TAG definition;
[0066] • Calculation section: Standard SQL query calculation statements;
[0067] 1.1 Trigger type or trigger method
[0068] Built-in supported trigger types or trigger methods include:
[0069] •SESSION(ts_col, session_val): Triggered by the session window
[0070] •STATE_WINDOW(col) [TRUE_FOR(duration_time)]: Status window triggered.
[0071] • [INTERVAL(interval_val[, interval_offset])] SLIDING(sliding_val[,offset_time]): Triggered by sliding.
[0072] •EVENT_WINDOW(START WITH start_condition END WITH end_condition)[TRUE_FOR(duration_time)]: Event window triggered.
[0073] •COUNT_WINDOW(count_val[, sliding_val][, col1[, ...]]): Counting window triggered.
[0074] •PERIOD(period_time[, offset_time]): Triggers at regular intervals.
[0075] Triggering Groups
[0076] • [PARTITION BY col1[, ...]]: Specifies the grouping column to trigger. Triggers support grouping, and each group is processed as a separate stream computation task, with the results stored in a separate output table.
[0077] 1.2 Control Options
[0078] The control options for a specified stream are used to control the triggering behavior of the stream. Currently supported control options include:
[0079] •WATERMARK(duration_time): Specifies the tolerance period for out-of-order data.
[0080] • EXPIRED_TIME(exp_time): Defines the duration of expired data for calculations that ignore expired data.
[0081] •IGNORE_DISORDER: Used to control the ignoring of out-of-order data triggering in the trigger table.
[0082] • DELETE_RECALC: Specifies that the table should be automatically recalculated when data is deleted.
[0083] •FILL_HISTORY[(start_time)]: Specifies that historical data is required for calculation. Data that already exists in the database when the stream is created is treated as historical data. By specifying this option, the calculation results of batch processing can also be included in the results of stream processing, achieving the effect of batch and stream integration.
[0084] •CALC_NOTIFY_ONLY: Specifies that the calculation results are only sent as a notification and are not saved to the output table.
[0085] •LOW_LATENCY_CALC: Specifies that low-latency computation or notification is required after triggering. Low-latency computation or notification will ensure the timeliness of real-time stream computing tasks, but it will also reduce processing efficiency. With this option, users can balance latency and throughput.
[0086] •PRE_FILTER(expr): Specifies that the trigger table is filtered before the trigger is executed, and only data that meets the conditions will be included in the trigger judgment.
[0087] •FORCE_OUTPUT: Specifies the option to force output of calculation results, used to fill in results when no calculation results are generated.
[0088] •MAX_DELAY(delay_time): Specifies the maximum trigger wait time when the window is not closed, which is used in scenarios where window calculation results need to be output in a timely manner.
[0089] •EVENT_TYPE(event_types): Specifies the type of event triggered by the window, such as when the window is opened or closed.
[0090] •IGNORE_NODATA_TRIGGER: Specifies to ignore triggers when the table has no input data, to avoid invalid triggers when there is no data.
[0091] 1.3 Placeholders
[0092] To facilitate the use of triggering information generated during the triggering phase in the calculation phase, this invention defines placeholders to represent this triggering information. This information appears in the SQL statements during the calculation phase and is replaced as constants in the SQL statements during each actual count. Through the use of these placeholders, triggering and calculation are organically combined, enabling richer functionality. Below are some examples of placeholders:
[0093]
[0094] 1.4 Example of Flow Construction
[0095]
[0096] Figure 2 The specific process of the server performing the stream computing of this invention is shown, including:
[0097] A. The trigger periodically requests trigger data from the WAL module;
[0098] B. The WAL module reads the trigger data from the WAL and filters the data, then returns the filtered data to the trigger;
[0099] C. The trigger determines whether to trigger based on the trigger type, conditions, control options, etc.
[0100] The trigger executes the above process sequentially until an event that meets the triggering condition is found, and then the calculation phase begins:
[0101] D. The trigger sends event information that meets the conditions to multiple computing units as a computing request;
[0102] E. The calculation unit begins data calculation based on the calculation information;
[0103] F. After the calculation is completed, the result is written to the database or pushed to the WebSocket service according to the flow creation statement;
[0104] G. The computing unit returns a computing response after the write or push is completed;
[0105] At this point, one computation task is complete. It's important to note that while the computation phase is underway, the trigger is still repeating the AC (Accumulation and Computation) process; therefore, triggering and computation are running simultaneously in parallel. When a trigger generates multiple computation events, they can be sent as a single computation task to a single computation unit, or they can be split into multiple tasks and computed in parallel across multiple computation units, depending on the strategy.
[0106] The key technologies of this invention are: an implementation method that allows for the separation or combination of triggering and computation; an efficient triggering method based on WAL in the database without data caching; the elimination of state management for event-based stream computing tasks; an implementation method for user-visible and reusable state information; and an efficient and minimalist checkpoint mechanism.
[0107] Compared with the prior art, the technical advantages of the present invention are as follows:
[0108] 1. Decoupling of triggering and computation
[0109] A complete stream computing task is divided into two phases: triggering and computation. The triggering phase is responsible for filtering the input data, judging the event trigger, generating the trigger group and output table. Only data records that meet the conditions will enter the event judgment, and only when the event judgment is successful will it enter the next step of the computation phase. The computation phase is responsible for performing calculations based on the events generated by the trigger, and then outputting the results to the output table.
[0110] By decoupling triggering and computation, the processing capacity of stream computing can be significantly expanded. Users can also define the triggering and computation parts separately, which is clearer in logic and easier to understand. At the same time, for ease of use, this invention uses a single SQL statement to define a complete stream computing task. The statement structure also follows the principle of decoupling triggering and computation, allowing users to naturally write complete statements following the idea of triggering first and then computation.
[0111] 2. Expansion and enrichment of computing functions
[0112] The triggering phase can generate a series of events, groups, windows, and other related attribute information. Calculation statements can be defined based on this information. The organic combination of these two phases can achieve richer functionality than a single statement, for example:
[0113] • The table objects that trigger the calculation can be different, so a cross-table join operation can be achieved without a Join statement;
[0114] • The calculation object can be a group generated by the trigger, or it can be not limited to the group generated by the trigger. It can perform calculations on other groups, and it can also generate its own groups and regroup based on the group generated by the trigger. The methods are flexible and diverse.
[0115] • It also has great flexibility in terms of the computation time of time series data. The object of window computation is the window that is triggered, or the window that is generated by itself, or the window that is generated again within the window. The types of windows can also be combined arbitrarily, and even computation can be performed without being based on windows.
[0116] 3. Controllable and flexible batch calculation function
[0117] The calculation phase requires completing the calculation tasks of all groups and windows according to the calculation statement. In the time series data scenario, windows and groups correspond to a time interval and a device, respectively. Therefore, their number, especially the number after combination, may be enormous.
[0118] To address this scenario, the present invention employs a batch computation method under acceptable latency conditions to achieve optimal system throughput. After a batch of computational demands is triggered, the present invention can perform computations either by using a single thread to batch compute a group of windows, or by using a single thread to batch compute one or more (all) windows of all groups. Therefore, the processing method for computational functions can be flexibly selected based on available computing resources and the adaptability of computational statements to achieve optimal resource utilization and throughput.
[0119] 4. Expanding the scope of application of stateless computation
[0120] In traditional stream computing engines, apart from some simple tasks such as filtering and data transformation that do not require state information, other tasks heavily rely on state information. The use of state information requires corresponding state information storage, backup, recovery, and management functions. These functions bring huge storage and hardware requirements, as well as implementation and performance problems such as complex implementation and long backup and recovery times.
[0121] Stream computing engines within database systems have a natural advantage in solving these types of problems because database systems have persistent storage capabilities for input data, thus eliminating the need for input data caching. Combined with a strategy of decoupling triggering from computation, this invention can retrieve the necessary data from the database storage engine for computation only after the event triggering conditions are met, thus eliminating the need to cache intermediate computation result state information. A similar strategy can be used to handle out-of-order data: first calculate its impact range, then recalculate the affected intervals, thereby avoiding the need for state information. After removing all these limitations, this invention can extend stateless computation to almost all computational tasks, including aggregation, windowing, and associative computations. The only constraint is the latency of the computation result; as long as the immediate computation result meets the user's latency requirements, stateless computation can be used.
[0122] For example, calculations can be performed only after the triggering conditions are fully met. When calculating the hourly average temperature of IoT devices, it is not necessary to perform the calculation immediately after the start of an hour. Otherwise, the calculation state will continuously generate intermediate results. The storage and maintenance costs of this state information are very high. It is only necessary to read the temperature data of that hour from the database once after the end of each hour to perform the calculation, thus achieving the best resource utilization.
[0123] 5. Optimization and reuse of state information
[0124] In traditional stream computing engines, state information exists as temporary information, and its scope is limited to the result calculation of a specified calculation statement. It is usually temporarily stored in the form of checkpoints. It is a waste that state information, as a high-cost information carrier, cannot be used more.
[0125] For example, the calculation results generated can be either the final result or an intermediate result state, depending on the business needs. Taking the calculation of the hourly average temperature (avg) of IoT devices as an example, the average temperature value is the final calculation result, while the temperature and the number of records are intermediate results. Depending on the user's selection of the output intermediate result state, the state information generated by the calculation will be stored in the database and can be used later when calculating the hourly, daily, or even monthly average temperature of devices, so that the calculation results of stream computing can be reused to the greatest extent.
[0126] In this invention, state information is no longer stored as temporary information, but rather stored in the storage engine of a database system to achieve reusability. In a time-series database, this invention generates state information in time-segmentation format, with intermediate states of the calculation results for each time segment stored as state information. The scope of use for state information can be extended to any calculation statement containing one or more time segments. In this form, whether to generate state information is determined by the user, and the generation of state information is incremental, while its removal can be done on demand. Therefore, state information becomes visible, manageable, and usable information for the user, allowing it to play a greater role in the user's business.
[0127] 6. A highly efficient and minimalist checkpoint mechanism
[0128] After separating state information from the checkpoint, the checkpoint's role becomes extremely simple: it only needs to be able to resume computation progress from the checkpoint. Because a large amount of state information is removed, there is no information that needs to be saved to the checkpoint during the computation phase. Therefore, checkpoint generation, synchronization, management, and recovery become highly efficient. It does not require complex distributed snapshot algorithms, nor does it even require incremental checkpointing mechanisms, and it does not rely on high-performance storage hardware or components such as RocksDB.
[0129] In the database system, this invention uses the version number of the committed record in WAL as a marker of the computation progress. When generating a checkpoint, the record stream records the WAL version number of each data source (distributed) and the computation progress (timestamp) of each group. During recovery, the WAL version number and group computation progress are obtained from the checkpoint and the stream computation continues accordingly.
[0130] In summary, this invention provides a stream computing implementation method, including:
[0131] The user sends a stream computation request to the server, which is defined by trigger, group, compute, and output statements;
[0132] The server parses the trigger, grouping, calculation, and output statements in the stream computing request sent by the user, and generates and persists the triggering information, grouping information, calculation information, and output information.
[0133] After persistently saving trigger information, grouping information, calculation information, and output information, the server begins to deploy stream computing tasks. Specifically, this includes deploying and executing trigger tasks in the trigger phase on the corresponding data nodes, and scheduling and deploying computing tasks according to the calculation information.
[0134] After the server completes the deployment of the stream computing task, it executes the trigger task according to the trigger information; it executes the computing task on the data obtained when executing the trigger task according to the computing information; and it outputs the data computing results completed when executing the computing task according to the output method defined by the output information.
[0135] Among them, the trigger information persistently stored on the server side is obtained by parsing the trigger statement in the user stream computing request, which includes the trigger method, trigger table, trigger event start and end conditions, and trigger filtering conditions; the grouping information persistently stored on the server side is obtained by parsing the grouping statement in the user stream computing request, which includes grouping by trigger table sub-tables.
[0136] The computation information persistently stored on the server side is obtained by parsing the computation statements in the user's stream computation request.
[0137] The output information persistently stored on the server side consists of an output table and data columns that output the data calculation results obtained by parsing the output statements in the user's stream calculation request.
[0138] The server-side execution of the triggering task according to the triggering information includes: reading the data columns of all sub-tables of the triggering table from the pre-log WAL of the corresponding data node; determining whether to start the triggering based on the triggering filter conditions; if the triggering is started, determining the start and end times of the triggering method of the data columns of each sub-table according to the triggering event start and end conditions; and outputting the relevant information of the data column triggering method.
[0139] The triggering method defined by the start and end times is an event window defined by the start and end times. The relevant information of the triggering method includes the sub-table name to which the window data belongs, the window start time, and the window end time.
[0140] The server-side calculation tasks, which involve performing calculations on the data obtained when the trigger task is executed based on the calculation information, include:
[0141] The server starts calculation based on the event window generated by the triggering task. If the triggering task generates multiple event windows, it reads data for all window time periods in batches from the corresponding trigger table according to the start and end times of the multiple event windows, and calculates the read data according to the calculation method to obtain the data calculation results for multiple event windows. If the triggering task generates one event window, it reads data for the time period of the event window from the sub-table of the corresponding trigger table according to the start and end times of the event window, and calculates the read data to obtain the data calculation result for the event window.
[0142] The server outputs the data calculation results completed during the execution of the calculation task according to the output method defined in the output information, including: creating an output super table for output; generating an output sub-table in the output super table after the calculation task is completed; and writing the calculation results of the event window into the generated output sub-table.
[0143] The server has a trigger and a WAL module for executing triggering tasks. When executing a triggering task, the trigger periodically requests trigger data from the WAL module. The WAL module reads the trigger data from the WAL and filters the data, and then returns the filtered data to the trigger. The trigger makes a triggering judgment based on the triggering method, conditions, control options, etc., until it finds a triggering event that meets the triggering conditions.
[0144] The server is equipped with one or more computing units for performing computing tasks. After the trigger sends event information that meets the triggering conditions to the computing unit in the form of a computing request, the computing unit starts to perform data computing based on the computing information. After the computing is completed, the computing result is written to the database or pushed to the WebSocket service. After the writing or pushing is completed, the computing unit returns a computing response.
[0145] When performing stream computing tasks, the WAL version number can be used as a marker of computing progress. When generating a checkpoint, the WAL version number of the data stream at each data source is recorded, and the computing progress of each group is recorded by timestamp. This allows the WAL version number and group computing progress to be retrieved from the snapshot checkpoint and the stream computing to continue when the fault is recovered.
[0146] In this process, when executing stream computing tasks, the data generated in each time period is stored in the database. After each time period ends, the stored data is read from the database and the read data is processed, thereby eliminating the status information of the intermediate calculation results generated in each time period.
[0147] Specifically, for the status information that generates intermediate calculation results when performing stream computing tasks, the status information is stored in the database when the user selects to apply the status information so that the status information can be used in subsequent calculations.
[0148] The following example, using data collection from common devices in the Industrial Internet of Things (IIoT), illustrates the above method in detail. Assume a group of devices collects temperature and pressure information via sensors, and data modeling is performed using a supertable:
[0149]
[0150] If you need to calculate the average and maximum pressure values of each device in real time when the temperature is too high (temperature >= 200°C) using stream computing, you can create a stream using the following statement:
[0151]
[0152] The execution process of this statement is explained as follows:
[0153] 1. Create a stream computing task
[0154] The syntax parser analyzes this stream creation statement, verifies its syntax, and then generates various types of information by combining the metadata information from the super table `device`:
[0155] • Trigger information:
[0156] • The trigger table is device
[0157] • Triggered via event window.
[0158] • The trigger condition for the event is a temperature greater than or equal to 200, and the termination condition is a temperature less than 100.
[0159] • The tolerance period for triggering out-of-order data is 1 second.
[0160] • The trigger filtering condition is pressure > 1; data with a pressure value less than or equal to 1 will not trigger.
[0161] • Grouping information:
[0162] • Determine to group according to tbname, which is the sub-table.
[0163] • Calculation information:
[0164] • The calculation statement `select _twstart, avg(pressure), max(pressure) from %%tbname where ts >= _twstart and ts <= _twend` is parsed and optimized as a normal query, ultimately generating a complete execution plan.
[0165] • Output information:
[0166] The output table is determined to be the super table stm1_res. The output columns of the calculation statement are the columns of the super table. The tag column information of the super table is generated according to the default rules, and then the super table stm1_res is created. 。
[0167] After all the above information is generated and categorized, it is persistently saved so that the stream computing task can continue after a system restart. After saving, the deployment phase of the stream computing task begins. First, trigger tasks are deployed on each node in the super table `device`. Then, the execution computations generated by the computation information are scheduled and deployed, but not started. After the entire deployment is complete, the trigger tasks are notified to start execution, and the trigger execution phase begins. At this point, the creation of the stream computing task is complete.
[0168] 2. Triggering the execution phase
[0169] Once the task is initiated, it enters the trigger execution phase, which proceeds as follows:
[0170] a. The trigger task reads the timestamp, temperature, and pressure columns of all sub-tables of the super table device from the node's input WAL. If pressure < 1, the corresponding data will be discarded and will not be included in the subsequent trigger event condition judgment.
[0171] b. For each sub-table's temperature column, determine the trigger condition for the event. That is, check if the temperature value at each moment is greater than or equal to 200. Once a temperature value that meets the condition is found, it means that the event has been started. Record the timestamp at this time (e.g., 2025-08-31 10:00:00) as the start time of the event window (_twstart), and record the name of the sub-table to which this record belongs (e.g., device1).
[0172] c. After the event is started, the event termination condition is determined, that is, whether there is a record of a temperature value less than 100. Once it appears, it means that an event window has been generated. The timestamp at this time (e.g., 2025-08-31 10:05:00) is recorded as the end time (_twend) of the event window.
[0173] d. After finding the event window that meets the criteria, output the relevant information about the window, including the name of the sub-table to which the data in the window belongs (%%tbname), the window start time (_twstart), the window end time (_twend), etc.
[0174] At this point, the single-trigger execution phase is complete. The triggering task will continue to execute the above steps in a loop, continuously outputting window information that meets the conditions.
[0175] 3. Calculation and execution phase
[0176] The calculation task performs calculations based on the window information generated by the triggering task. If multiple windows are generated in a single instance, batch window calculations can be performed; if only one window is generated, only a single window calculation is performed. In this example, the calculation task reads data from the corresponding sub-table (device1) based on the start and end times of the obtained window (2025-08-31 10:00:00 to 2025-08-31 10:05:00), then calculates the average pressure (avg) and maximum pressure (max), and finally produces the calculation result for this window.
[0177] 4. Results Output
[0178] After the calculation results are generated in the device1 table, the corresponding output sub-table's name and tag column need to be generated based on the device1 table name and tag column information. Then, this output sub-table is created in the output super table stm1_res, and finally, the window's calculation results are written to this sub-table. In addition, the triggered task will also periodically output calculation progress information to the checkpoint file.
[0179] Although the present invention has been described in detail above, it is not limited thereto, and those skilled in the art can make various modifications based on the principles of the present invention. Therefore, all modifications made in accordance with the principles of the present invention should be understood to fall within the protection scope of the present invention.
Claims
1. A stream computing implementation method, comprising: The user sends a stream computation request to the server, which is defined by trigger, group, compute, and output statements; The server parses the trigger, grouping, calculation, and output statements in the stream computing request sent by the user, and generates and persists the triggering information, grouping information, calculation information, and output information. After persistently saving trigger information, grouping information, calculation information, and output information, the server begins to deploy stream computing tasks, including deploying and executing trigger tasks in the trigger phase on the corresponding data nodes, and scheduling and deploying computing tasks according to the calculation information. After the server completes the deployment of the stream computing task, it first executes the trigger task according to the trigger information; Then, a calculation task is performed on the time-series data obtained when the triggering task is executed, according to the calculation information; and the calculation results of the time-series data completed when the calculation task is executed are output according to the output method defined by the output information. Among them, the trigger information that the server persists to store is obtained by parsing the trigger statement in the user stream computing request, which includes the trigger type, trigger table, trigger method, trigger event start and end conditions, and trigger filtering conditions; the grouping information that the server generates and persists to store is obtained by parsing the grouping statement in the user stream computing request, which includes grouping by trigger table sub-tables. The server-side execution of the triggering task according to the triggering information includes: reading the data columns of all sub-tables of the triggering table from the pre-log WAL of the corresponding data node; determining whether to start the triggering based on the triggering filter conditions; if the triggering is started, determining the start and end times of the triggering method of the data columns of each sub-table according to the triggering event start and end conditions; and outputting the relevant information of the triggering method of the data columns.
2. In the stream computing implementation method according to claim 1, the computing information persistently stored on the server side is the computing method of computing data obtained by parsing the computing statements in the user's stream computing request.
3. In the stream computing implementation method according to claim 1, the output information persistently stored on the server side is the output table and data columns obtained by parsing the output statement in the user's stream computing request, which outputs the time-series data calculation results.
4. In the stream computing implementation method according to claim 2, the data column triggering method defined by the start time and end time is an event window defined by the start time and end time, and the relevant information of the data column triggering method includes the sub-table name to which the window data belongs, the window start time, and the window end time.
5. The stream computing implementation method according to claim 4, wherein the server performs a computing task on the time-series data obtained when the trigger task is executed according to the computing information, including: The server begins calculations based on the event window generated by the triggered task. If the triggered task generates multiple event windows, then based on the start and end times of the multiple event windows, data for all window time periods are read in batches from the corresponding trigger table, and the read data is calculated according to the calculation method described above to obtain the data calculation results for multiple event windows; If a task generates an event window, the data for that time period is read from the sub-table of the corresponding trigger table based on the start and end times of the event window. The read data is then used to perform calculations to obtain the data calculation result for that event window.
6. The stream computing implementation method according to claim 5, wherein the server outputs the data calculation results completed when executing the computing task according to the output method defined by the output information, including: Create an output supertable for output; After the computation task is completed, an output sub-table is generated in the output super-table; Write the calculation results of the event window into the generated output subtable.
7. The stream computing implementation method according to claim 6, wherein the server has a trigger and a WAL module for executing triggering tasks; when executing a triggering task, the trigger periodically requests trigger data from the WAL module; the WAL module reads the trigger data from the WAL and performs data filtering, and then returns the filtered data to the trigger; the trigger makes a trigger judgment based on the trigger type, condition, and control options until a triggering event that meets the triggering conditions is found.
8. The stream computing implementation method according to claim 7, wherein the server is provided with one or more computing units for performing computing tasks; After the trigger sends a triggering event that meets the triggering conditions to the computing unit as a calculation request, the computing unit starts to perform data calculation based on the calculation information; after the calculation is completed, the calculation result is written to the database or pushed to the WebSocket service; after the writing or pushing is completed, the computing unit returns a calculation response.
9. The stream computing implementation method according to claim 1, wherein when executing the stream computing task, the WAL version number is used as a marker of the computing progress; when generating a checkpoint, the WAL version number of the data stream at each data source is recorded, and the computing progress of each group is recorded by timestamp, so that when the fault is recovered, the WAL version number and the group computing progress can be obtained from the checkpoint and the stream computing can continue accordingly.
10. The stream computing implementation method according to claim 1, wherein when executing the stream computing task, the data generated in each time period is stored in the database, the stored data is read from the database after each time period ends, and the read data is calculated, thereby eliminating the state information of the intermediate calculation results generated in each time period.
11. The stream computing implementation method according to claim 1, wherein for the state information that generates intermediate computing results when executing a stream computing task, when the user selects to apply the state information, the state information is stored in a database so that the state information can be used in subsequent calculations.