Streaming data processing method, sparse response type state engine, device and medium

Abstract

The invention relates to a streaming data processing method, a sparse response type state engine, a device and a medium, and the method comprises the steps: defining a rule table which comprises an input identification key column, a calculation expression key column and an output index key column; the rule table is analyzed to construct routing mapping between the input data and the output indexes, a state table and a response type state object are generated, the state table serves as a state container for storing all calculation processes of the output indexes, and all the calculation processes associated with the output indexes are packaged by the response type state object; and acquiring input data in real time, and mapping and matching to the state table of the output index and the response type state object by utilizing a route to calculate to obtain a stream data processing result. According to the method and the device, accurate routing based on the identification key is realized, so that only stream data calculation of the association index is triggered, and the calculation complexity is simplified by the reusability of association packaging of the response type state object and the calculation process, and the stream data processing efficiency is effectively improved.

Description

Technical Field

[0001] This application relates to the field of graph database technology, and in particular to a streaming data processing method, a sparse reactive state engine, an apparatus, and a medium. Background Technology

[0002] In time-series database systems, streaming data computing engines are key to supporting business decisions and system functions. However, existing streaming computing engines have many problems that need to be solved. For example, the traditional streaming computing engine combination of DolphinDB requires recalculating all metrics for each calculation, resulting in serious computational redundancy. It is difficult to implement fine-grained dependency triggering mechanisms, has low resource utilization, high response latency, complex configuration, and new metrics depend on user-described complex rules, etc.

[0003] Currently, no effective solution has been proposed for improving the processing efficiency of streaming data in databases within related technologies. Summary of the Invention

[0004] This application provides a streaming data processing method, a sparse reactive state engine, an apparatus, and a medium to at least address the problem of how to improve the processing performance of streaming data in databases in related technologies.

[0005] In a first aspect, embodiments of this application provide a streaming data processing method, the method comprising: Define a rule table for normalizing input data, wherein the rule table includes an input identifier key column, a calculation expression key column, and an output index key column; The rule table is parsed to construct a routing mapping between input data and output metrics; The rule table is parsed to generate a state table and a reactive state object. The state table serves as a state container for storing the various calculation processes of the output metric, and the reactive state object encapsulates and associates the various calculation processes of the output metric. Real-time acquisition of input data, and matching of the corresponding target output metrics using the routing mapping; The streaming data processing result of the target output indicator is calculated using the state table and reactive state object of the target output indicator.

[0006] In some embodiments, parsing the rule table to construct a routing mapping between input data and output metrics includes: The rule table is parsed to extract the associated input identifier key values ​​and output indicator key values ​​row by row; and the input route name is constructed based on the input identifier key values. Establish a mapping relationship between the input route name and the associated output indicator key value to obtain the route mapping between the associated input data and the output indicator.

[0007] In some embodiments, the input route name constructed based on the input identifier key value includes: Based on the input identifier key value, the length and content of the input identifier key value are concatenated to construct the input route name, wherein the input identifier key value corresponds to the device identifier data in the input data.

[0008] In some embodiments, parsing the rule table to generate a state table and a reactive state object includes: The calculation expression key column in the rule table is parsed row by row to construct each calculation process corresponding to the calculation expression key value, and a reactive state object is generated that is associated with each calculation process. The reactive state object includes a transition state object, a movement state object, and an iteration state object. The rule table is parsed row by row to generate a status table, wherein the status table includes an input identifier key column, an extra key column, and a status key column; the input identifier key column corresponds to the input identifier key column in the rule table; the extra key column is used to store context data in the input data; the status key column is used to store device status data in the input data, as well as calculation status data generated in each calculation process; The reactive state object is responsible for updating the calculated state key values ​​in the state key column of the state table.

[0009] In some embodiments, after generating the corresponding reactive state object, the method includes: Based on the dependencies between the various computation processes of the computation expression key value, a dependency graph is constructed between the reactive state objects, wherein the structure of the dependency graph is a tree structure.

[0010] In some embodiments, acquiring input data in real time and matching the corresponding target output metric using the routing mapping includes: Based on the key column index of the rule table, extract the key column data corresponding to each key column of the rule table from the real-time acquired input data; For the device identifier data corresponding to the input identifier key column, the length and content of the device identifier data are first concatenated to obtain the input name. Then, the corresponding input route name is matched in the route mapping to obtain the target output indicator corresponding to the input data.

[0011] In some embodiments, the streaming data processing result of the target output indicator is calculated using the state table and reactive state object of the target output indicator, including: Based on the correspondence between the input identifier key column of the status table and the input identifier key column of the rule table, context data and device status data are extracted from the input data and written into the corresponding additional key column and status key column, respectively. Then, based on the device status data, the calculation process of each calculation expression is executed through the responsive status object, and the calculation results are synchronously updated to the corresponding status key column; Finally, the calculated state data from the state key column of the state table are concatenated to obtain the stream data processing result of the target output index.

[0012] Secondly, embodiments of this application provide a sparse reactive state engine, which is used to execute the method described in the first aspect above. The engine is deployed in the DolphinDB time-series database and includes an engine parser and a computation executor. The engine parser is used to parse the predefined rule table to construct a routing mapping between input data and output metrics; The engine parser is used to parse the rule table to generate a state table and a reactive state object. The state table serves as a state container for storing the various calculation processes of the output metric, and the reactive state object encapsulates and associates the various calculation processes of the output metric. The computational executor is used to acquire input data in real time and use the routing mapping to match the corresponding target output metric. The computational executor is used to calculate the streaming data processing result of the target output indicator through the state table and reactive state object of the target output indicator.

[0013] Thirdly, embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0014] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0015] Compared to related technologies, this application provides a streaming data processing method, a sparse reactive state engine, an apparatus, and a medium. This method defines a rule table for standardizing input data, containing input identifier key columns, calculation expression key columns, and output metric key columns. The rule table is parsed to construct a routing mapping between input data and output metrics. The rule table is further parsed to generate a state table and a reactive state object. The state table serves as a state container storing the various calculation processes of the output metrics, and the reactive state object encapsulates the various calculation processes associated with the output metrics. Input data is acquired in real-time, and the routing mapping is used to match the corresponding target output metrics. The streaming data processing result of the target output metrics is calculated using the target output metrics' state table and the reactive state object. This achieves precise routing based on identifier keys, thereby triggering only the streaming data calculation of associated metrics. Furthermore, the reusability of the reactive state object and the encapsulation of the calculation process simplifies computational complexity, effectively improving streaming data processing performance and solving the problem of how to improve the processing performance of streaming data in a database. Attached Figure Description

[0016] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of the steps of a streaming data processing method according to an embodiment of this application; Figure 2 This is a flowchart illustrating the generation of a state table according to an embodiment of this application; Figure 3 This is a schematic diagram of the process for calculating the output index according to an embodiment of this application; Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0018] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0019] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0020] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0021] This application provides a streaming data processing method. Figure 1 This is a flowchart of the steps of a streaming data processing method according to an embodiment of this application, as follows: Figure 1 As shown, the method includes the following steps: Step S102: Define a rule table for normalizing the input data, wherein the rule table includes an input identifier key column, a calculation expression key column, and an output index key column; In step S102, the user defines a rule table in the form of a table or dictionary. Table 1 is an example of a rule table according to an embodiment of this application. As shown in Table 1, deviceID is the device identifier key column (input identifier key column) for input data, Formula is the calculation expression key column for output metrics, and outputMetricKey is the output metric key column. The output metric is associated with the user-defined calculation expression and is used to perform real-time calculation on the input data and output the result.

[0022] Table 1

[0023] It should be noted that the execution entity of this streaming data processing method is the Sparse Reactive State Engine, a streaming data processing engine capable of triggering calculations in real time based on changes in input data and maintaining historical states. The engine's architecture consists of: a Sparse Reactive State Engine Parser, responsible for parsing metric definitions and building dependency graphs; a computation executor, performing real-time calculations and managing data stream processing; a Reactive State, a state object encapsulating individual computational logic, derived from the calculation expressions of output metrics; and a State Table, maintaining all reactive states and calculation results for each output metric. This engine features sparse triggering, meaning it calculates only the output metrics associated with the input data's identifier, rather than triggering the calculation mode for all output metrics (see steps S104 and S108 below). Furthermore, it also features dynamic rule management, supporting the dynamic addition, deletion, and querying of output metrics during engine runtime without stopping or restarting the engine.

[0024] Step S104: Parse the rule table to construct a routing mapping between input data and output metrics; It should be noted that the core functions of the Sparse Reactive State Engine Parser include rule table parsing, dependency analysis, state object creation, and state table generation. Specifically, in steps S104 and S106, the rule table is received as input, and an independent state table and a sequence of reactive objects are generated for each output metric in the rule table, thus realizing the technical basis for metric isolation and sparse triggering.

[0025] Step S104 specifically includes the following steps: Step S1041: Parse the rule table to extract the associated input identifier key value and output indicator key value row by row; and construct the input route name based on the input identifier key value. Specifically, step S1041 involves concatenating the length and content of the input identifier key value to construct the input route name, where the input identifier key value corresponds to the device identifier data in the input data.

[0026] Step S1042: Establish a mapping relationship between the input route name and the associated output indicator key value to obtain the route mapping between the associated input data and the output indicator.

[0027] It should be noted that the engine parser processes the user-provided rule table (containing input identifier key columns, calculation expression key columns, and output metric key columns). First, it verifies the structural integrity of the rule table to ensure that the number of columns matches the number of key columns. Then, as in steps S1041 and S1042 above, it extracts the output metric key values ​​(metric definitions) row by row. For each metric, it extracts the associated input identifier key value and constructs an input route name. This input route name is formed by concatenating the length and content of the input identifier key, ensuring uniqueness and avoiding separator conflicts. Step S106: Parse the rule table to generate a state table and a reactive state object. The state table serves as a state container for storing the various calculation processes of the output metric, and the reactive state object encapsulates and associates the various calculation processes of the output metric. It should be noted that, Figure 2 This is a flowchart illustrating the state table generated according to an embodiment of this application, such as... Figure 2 As shown, the state table generation is the core process by which the engine parser creates an independent storage space for each output metric. The generation process begins with the initialization of the state table, which serves as the state container for the calculation of the output metrics.

[0028] Step S106 specifically includes the following steps: Step S1061: Parse the calculation expression key column in the rule table row by row to construct each calculation process corresponding to the calculation expression key value, and generate a reactive state object associated with each calculation process. The reactive state object includes a transition state object, a movement state object, and an iteration state object. It should be noted that, based on the in-depth parsing of the expression key values ​​in the rule table, corresponding reactive state objects are generated. Specifically, reactive state objects include transition state objects, moving state objects, and iterative state objects. Transition state objects are responsible for general expression calculations (such as arithmetic operations and function calls). They read the values ​​of the input columns from the state table and perform expression evaluation. Moving state objects implement moving window functions, maintain historical data windows and cumulative states, and update the window and calculate new window statistics (such as moving average and moving sum) with each append call. Iterative state objects support state iteration functions (such as conditional iteration and state iteration). The output columns of these state objects are also input columns, forming a self-looping update of the state. Iterative state objects maintain an iteration counter and termination conditions, and perform one or more iterations with each append call until the termination condition is met or the maximum number of iterations is reached.

[0029] Step S1062: Based on the dependency relationships between the various calculation processes of the expression key value, a dependency graph between reactive state objects is constructed, wherein the structure of the dependency graph is a tree structure; It should be noted that, as Figure 2 As shown, dependency analysis requires recursively traversing the object tree that computes expression key values. Specifically, the recursive traversal starts from the top level of the expression, distinguishing different object types: for function objects, all parameters are recursively analyzed; for formula objects, all operands are traversed; for column reference objects, column indices are recorded, and so on. Then, a dependency graph between reactive state objects is created, recording which input state key columns the reactive state objects depend on and which output state key columns in the state table. This directed graph structure clarifies the data flow.

[0030] Furthermore, dependency analysis also handles the mapping from state key columns in the state table to reactive state objects. This mapping records the indexes of the reactive state objects corresponding to the state key columns. When a state key column in the state table is referenced by multiple reactive state objects, the producer can be quickly located. This bidirectional indexing mechanism enables accurate triggering of affected computational chains during data updates. Further, such as... Figure 2 As shown, for complex expressions, the engine parser identifies common subexpressions and reuses the calculation results. That is, it checks whether the expression script has been calculated. If it has, it directly references the corresponding status key column to avoid repeated calculations. This optimization significantly improves performance when handling multiple output metrics that share the same subexpression.

[0031] Step S1063: Parse the rule table row by row to generate a status table. The status table includes an input identifier key column, an extra key column, and a status key column. The input identifier key column corresponds to the input identifier key column in the rule table. The extra key column is used to store the context data in the input data. The status key column is used to store the device status data in the input data, as well as the calculation status data generated in each calculation process. Step S1064: The reactive state object is responsible for updating the calculated state key value in the state key column of the state table.

[0032] It should be noted that, as Figure 2 As shown, in steps S1063 and S1064, for each output metric, the engine parser parses the rule table to generate the corresponding state table until each output metric is parsed (recursion ends). Specifically, the engine parser first adds input identifier key columns to the state table, which are user-defined input identifier key columns from the rule table; then it adds additional key columns and state key columns. The additional key columns are used to store context data (such as timestamps) in the input data, and the state key columns are used to store device status data in the input data. As the user-defined calculation expressions in the rule table are parsed more deeply, when intermediate calculation results need to be stored, state key columns are dynamically added. Each dynamically added state key column is associated with a reactive state object, which is responsible for updating the state of that state key column.

[0033] Furthermore, the key columns in the state table can be divided into three categories: input columns—recording the key columns that need to be updated from input data (including key columns and additional columns, which can be dynamically expanded according to the calculation expression); state columns—recording all state key columns, which is the complete set of columns for state calculation; and output columns—recording the columns that need to be output (including input identifier key columns, additional key columns, and state key columns). This classification makes the data flow path clear and the update and output operations executed efficiently.

[0034] Step S108: Acquire input data in real time and use routing mapping to match the corresponding target output metric; Specifically, in step S108, Figure 3 This is a schematic diagram illustrating the calculation of output metrics according to an embodiment of this application, such as... Figure 3 As shown, based on the key column index of the rule table, the key column data corresponding to each key column of the rule table is extracted from the real-time acquired input data; for the device identifier data corresponding to the input identifier key column, the length and content of the device identifier data are first concatenated to obtain the input name, and then the corresponding input route name is matched in the route mapping to obtain the target output metric corresponding to the input data.

[0035] It's important to note that in the Sparse Reactive State Engine, the `append` method is the core entry point for processing real-time data streams, responsible for receiving, validating, and distributing input data. Specifically, it first performs strict column count validation to ensure the number of columns in the input data exactly matches the number configured during engine initialization. If the column counts don't match, an error message is immediately returned, specifying the expected and actual number of columns received, preventing subsequent errors due to data structure mismatches. After successful validation, key column data is extracted from the input data based on the key column indexes in the rule table, constructing a key column vector. Different data types of key columns correspond to different extraction methods. For example, for string key columns, the system directly obtains the string reference; for symbolic key columns, the system converts the integer index into the actual string value using a symbol table; for numeric key columns, the system serializes the numeric value into a fixed-length byte sequence, and so on. Then, the key column data is converted into byte sequences of indicator identifiers. These byte sequences will be used for subsequent sparse route lookups. The entire input processing process is protected by mutex locks, ensuring thread safety for dynamic indicator management and data input.

[0036] It's important to further explain that sparse routing is a core mechanism that distinguishes it from traditional full-scale computation engines. The system uses input routing mapping to achieve efficient metric lookup. This mapping uses identifier encoding as the key. For each input identifier key column, the length of the key value is recorded first, and then the actual content is concatenated. This length prefix encoding method ensures that different key-value combinations produce unique input names, avoiding ambiguity issues that may arise from using separators. For example, the key-value combination "A001" + "A002" will be encoded as "4A0014A002", where 4 and 4 are the lengths of the two key values, respectively. An unordered hash table is used to achieve constant-time lookup.

[0037] For each line of input data, the system first constructs an input name based on the key column value, and then searches for that input name in the input routing map. There are three possible results: no matching metric found, a single matching metric found, or multiple matching metrics found. If no matching metric is found, the system skips that line of data without performing any calculations. This is key to the sparse engine's zero computational overhead. Therefore, the mapping and matching process for output metrics demonstrates the core advantage of the sparse engine. Traditional full-scale computation engines need to trigger the calculation of all metrics for each input data, while the sparse engine only triggers the calculation of related metrics.

[0038] Step S110: Calculate the streaming data processing result of the target output indicator using the state table and responsive state object of the target output indicator.

[0039] Specifically, step S110 is as follows: Figure 3As shown, based on the correspondence between the input identifier key column of the status table and the input identifier key column of the rule table, context data and device status data are extracted from the input data and written into the corresponding extra key column and status key column, respectively. Then, based on the device status data, the various calculation processes of the calculation expression are executed through the reactive status object, and the calculation results are synchronously updated to the corresponding status key column. Finally, the calculated status data of the status key column in the status table are concatenated to obtain the streaming data processing result of the target output indicator.

[0040] It's important to note that once the target output metric is matched to the key column, that key is mapped to the corresponding state table. Specifically, first, a hash map is used to find the row number of the key in the state table. If the key appears for the first time, the number of rows in the state table is expanded, and the input identifier key column, the additional key column, and the state key column are written sequentially. Then, data is copied in batches from the column vectors in the input data to the state table; for both vector and scalar inputs, updates are performed within the same row to ensure the state table always holds the latest key context. If the data length does not match or the write fails, the process immediately throws an exception and rolls back the update to avoid dirty state residue. Second, all associated reactive state objects in the state table are calculated sequentially. A reactive state object encapsulates the metric calculation logic; each object is responsible for calculating and updating one or more columns in the state table. The state object sequence is generated by the parser during metric parsing and ordered by dependencies to ensure the correctness of the calculation order. The execution process calls the methods of each state object, and the state object performs the corresponding calculation logic based on its type (transition state, move state, iterate state, etc.). Transition state objects are responsible for general expression calculations (such as arithmetic operations and function calls). It reads the values ​​of the input columns from the state table and evaluates the expressions. Moving state objects implement moving window functions, maintaining historical data windows and cumulative states. Each append call updates the window and calculates new window statistics (such as moving average and moving sum). Iterating state objects support state iteration functions (such as conditional iteration and state iteration). The output columns of these state objects are also input columns, forming a self-looping update of the state. Iterating state objects maintain an iteration counter and termination conditions, executing one or more iterations each time they are appended, until the termination condition is met or the maximum number of iterations is reached. The execution of state objects is the core of metric calculation. By encapsulating complex calculation logic in independent state objects, the system achieves modularity and reusability of the calculation logic. Different metrics can share the same type of state object, only configuring different parameters and input / output columns, greatly simplifying the implementation complexity of the engine.

[0041] It should be further explained that after all state objects have been executed, the system extracts the calculation results from the state table. The output record contains the input identifier key, additional key, output metric name, and calculation result value. The input identifier key and additional key are directly extracted from the corresponding rows of the state table, the output metric name is obtained from the column-to-name mapping, and the calculation result value is extracted from the output columns. This structured output format facilitates data consumption by downstream systems. After calculation, the system outputs the updated metric value to the configured output table. The output table uses a narrow table format, containing a device identifier key column, additional columns, an output metric name column, and an output value column. The system extracts the calculation results from the state table, obtains the output metric name according to the column-to-name mapping, and assembles them into standardized output records. For identifier keys matching multiple metrics, a separate result row will be output for each metric. The output table, like the input table, is in a narrow table format. The output process uses a batch submission strategy; the system first collects all output columns for the same metric and then appends them to the output table all at once. For shared table output, the system uses table locks to protect concurrent writes; for distributed file system tables, the system automatically handles table expiration and domain refresh; for downstream streaming engines, the system directly transmits data to achieve zero-copy processing. This batch processing method reduces frequent access to the output table, significantly improving overall throughput.

[0042] The method provided in this application embodiment achieves precise routing based on the identifier key, thereby triggering only the calculation of streaming data of associated indicators. Furthermore, the reusability of the responsive state object and the encapsulation of the calculation process simplifies the computational complexity and effectively improves the efficiency of streaming data processing, thus solving the problem of how to improve the processing efficiency of streaming data in the database.

[0043] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0044] In some embodiments, the method also has a dynamic indicator management mechanism.

[0045] The engine supports runtime dynamic management of metrics, including adding new metrics and deleting existing metrics. When adding a new metric, the system verifies the validity of the metric definition, checks the completeness of the metric table structure (input identifier key column, calculated expression column, output metric key column), and then calls the parser to generate an independent state table and a sequence of reactive state objects for the new metric.

[0046] Adding a new metric first generates a state table and state objects, then appends them to the state table and state key column. The system synchronously updates the input routing mapping and output metric mapping, establishing a mapping relationship between input names, output metric names, and metric indices. This incremental addition method ensures that new metrics take effect immediately without requiring an engine restart.

[0047] When deleting an indicator, the system searches for the indicator index to be deleted in the output indicator mapping and then adds that index to the set of deleted indexes. A strategy of marking for deletion rather than physical deletion is used to avoid index rearrangement causing existing mapping relationships to become invalid. Indicators marked for deletion will be skipped in subsequent data processing and will not trigger calculations again.

[0048] In some embodiments, the method also includes error handling and state management mechanisms.

[0049] The engine employs a robust error handling mechanism, particularly a complete transaction rollback mechanism during state table updates and metric management. When adding a new metric and a new key, the current row number in the state table is recorded first. If an exception occurs during the addition process, the system performs a full rollback: removing the added state table row, cleaning up the keys in the reactive state objects, and deleting the new key from the key index mapping. When performing state calculations, the calculation context is set first, and then all reactive state objects for that metric are executed sequentially. If a state object calculation fails, the error message is captured, and the engine state is updated to an error state. Detailed error information is returned when output fails.

[0050] This application provides a sparse reactive state engine for executing the methods provided in the above embodiments. The engine is deployed in the DolphinDB time-series database and includes an engine parser and a computation executor. The engine parser is used to parse the predefined rule table to build a routing mapping between input data and output metrics; The engine parser is used to parse the rule table to generate a state table and a reactive state object. The state table serves as a state container to store the various calculation processes of the output metric, and the reactive state object encapsulates and associates the various calculation processes of the output metric. The computational executor is used to acquire input data in real time and match the corresponding target output metric using routing mapping; The computation executor is used to calculate the streaming data processing results of the target output metric using the state table and reactive state object of the target output metric.

[0051] This embodiment provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0052] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0053] Optionally, the electronic device may further include a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a streaming data processing method. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0054] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0055] Furthermore, in conjunction with the streaming data processing methods in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the streaming data processing methods in the above embodiments.

[0056] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operation of the operating system and computer programs, the computer programs are executed by the processor to implement a streaming data processing method, and the database stores data.

[0057] Those skilled in the art will understand that Figure 4The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. A specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0058] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0059] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0060] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A streaming data processing method, characterized in that, The method includes: Define a rule table for normalizing input data, wherein the rule table includes an input identifier key column, a calculation expression key column, and an output index key column; The rule table is parsed to construct a routing mapping between input data and output metrics; The rule table is parsed to generate a state table and a reactive state object. The state table serves as a state container for storing the various calculation processes of the output metric, and the reactive state object encapsulates and associates the various calculation processes of the output metric. Real-time acquisition of input data, and matching of the corresponding target output metrics using the routing mapping; The streaming data processing result of the target output indicator is calculated using the state table and reactive state object of the target output indicator.

2. The method according to claim 1, characterized in that, Parsing the rule table to construct a routing mapping between input data and output metrics includes: The rule table is parsed to extract the associated input identifier key values ​​and output indicator key values ​​row by row; and the input route name is constructed based on the input identifier key values. Establish a mapping relationship between the input route name and the associated output indicator key value to obtain the route mapping between the associated input data and the output indicator.

3. The method according to claim 2, characterized in that, Based on the input identifier key value, the constructed input route name includes: Based on the input identifier key value, the length and content of the input identifier key value are concatenated to construct the input route name, wherein the input identifier key value corresponds to the device identifier data in the input data.

4. The method according to claim 1, characterized in that, Parsing the rule table to generate a state table and reactive state objects includes: The calculation expression key column in the rule table is parsed row by row to construct each calculation process corresponding to the calculation expression key value, and a reactive state object is generated that is associated with each calculation process. The reactive state object includes a transition state object, a movement state object, and an iteration state object. The rule table is parsed row by row to generate a status table, wherein the status table includes an input identifier key column, an extra key column, and a status key column; the input identifier key column corresponds to the input identifier key column in the rule table; the extra key column is used to store context data in the input data; the status key column is used to store device status data in the input data, as well as calculation status data generated in each calculation process; The reactive state object is responsible for updating the calculated state key values ​​in the state key column of the state table.

5. The method according to claim 4, characterized in that, After generating the corresponding reactive state object, the method includes: Based on the dependencies between the various computation processes of the computation expression key value, a dependency graph is constructed between the reactive state objects, wherein the structure of the dependency graph is a tree structure.

6. The method according to claim 5, characterized in that, Real-time acquisition of input data, and matching the corresponding target output metrics using the routing mapping, includes: Based on the key column index of the rule table, extract the key column data corresponding to each key column of the rule table from the real-time acquired input data; For the device identifier data corresponding to the input identifier key column, the length and content of the device identifier data are first concatenated to obtain the input name. Then, the corresponding input route name is matched in the route mapping to obtain the target output metric corresponding to the input data.

7. The method according to claim 6, characterized in that, The streaming data processing results of the target output metric are calculated using the state table and reactive state object of the target output metric, including: Based on the correspondence between the input identifier key column of the status table and the input identifier key column of the rule table, context data and device status data are extracted from the input data and written into the corresponding additional key column and status key column, respectively. Then, based on the device status data, the calculation process of each calculation expression is executed through the responsive status object, and the calculation results are synchronously updated to the corresponding status key column; Finally, the calculated state data from the state key column of the state table are concatenated to obtain the stream data processing result of the target output index.

8. A sparse reactive state engine, characterized in that, The engine is used to execute the method according to any one of claims 1 to 7, the engine is deployed in the DolphinDB time series database, and the engine includes an engine parser and a computation executor; The engine parser is used to parse the predefined rule table to construct a routing mapping between input data and output metrics; The engine parser is used to parse the rule table to generate a state table and a reactive state object. The state table serves as a state container for storing the various calculation processes of the output metric, and the reactive state object encapsulates and associates the various calculation processes of the output metric. The computational executor is used to acquire input data in real time and use the routing mapping to match the corresponding target output metric. The computational executor is used to calculate the streaming data processing result of the target output indicator through the state table and reactive state object of the target output indicator.

9. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images