Streaming data processing method and system based on responsive stateless computing engine

Abstract

The invention relates to a streaming data processing method and system based on a response stateless calculation engine, and the method comprises the steps: introducing an engine parser, parsing an index definition, automatically constructing a dependency graph, and generating an optimized suffix expression, thereby achieving a precise triggering mechanism based on a dependency, and improving the reliability of streaming data processing. Cascade calculation of a specific path is triggered only when input data is updated, and invalid operation is avoided. Meanwhile, a stateless calculation model and a value index pre-distribution strategy are adopted, so that memory occupation is only related to the number of indexes and is decoupled from the data volume, and the updating efficiency of O (1) complexity is achieved. In addition, dynamic addition, deletion and cyclic detection of indexes during operation are supported, and the performance and flexibility of large-scale real-time calculation are remarkably improved while logic correctness is ensured.

Description

Technical Field

[0001] This application relates to the field of streaming data processing technology, and in particular to a streaming data processing method, system, computer device, and computer-readable storage medium based on a reactive stateless computing engine. Background Technology

[0002] In the field of data processing and real-time computing, streaming data processing technology faces severe challenges as scenarios such as financial transactions and IoT monitoring increasingly demand faster data delivery. The core of streaming data processing lies in how to efficiently manage complex computational metrics and their interdependencies, and how to quickly output computational results when data arrives in real time.

[0003] In related technologies, handling such computationally dependent tasks presents the following main challenges: Performance bottlenecks and resource waste: Existing solutions (such as automatic recalculation mechanisms in spreadsheet software) perform reasonably well when processing small amounts of data, but their performance drops sharply when dealing with large-scale data. For streaming computing scenarios, traditional computing engines often employ full recalculation or batch processing modes based on time windows.

[0004] For example, in Dolphin DB, traditional stream computing engine combinations (such as a reactive state engine combined with a cross-sectional engine) often require nesting multiple engines when handling complex calculations. When the underlying data undergoes minor changes, the system may need to recalculate the entire dependency chain or even unrelated metrics, resulting in severe computational redundancy, low resource utilization, high response latency, and an inability to meet millisecond-level real-time requirements.

[0005] Dependency management is complex and opaque: In complex business scenarios, there are often deep nested dependencies between calculated metrics. In existing technologies, such dependencies are usually hard-coded in scripts or programs, lacking a unified graph structure management. This makes maintenance extremely difficult when business logic changes (such as adding or modifying metrics), and easily introduces logical errors such as circular dependencies. In addition, traditional stateful stream computing engines need to maintain a large amount of historical state data, resulting in high memory consumption and difficulty in handling high-concurrency real-time data streams.

[0006] Insufficient dynamic scalability: In real-world production environments, business requirements are dynamic, often necessitating the addition of new computational metrics or the deletion of old ones without service interruption. Most existing pre-compiled or statically configured computational engines lack flexible runtime dynamic management capabilities, requiring service restarts or recompilation for each metric change, severely impacting system availability. Summary of the Invention

[0007] This application provides a streaming data processing method, system, computer device, and computer-readable storage medium based on a reactive stateless computing engine, to at least solve the problem in related technologies that streaming data cannot be processed efficiently with low latency.

[0008] In a first aspect, embodiments of this application provide a streaming data processing method based on a reactive stateless computing engine, the method comprising: The engine parser parses user-defined computation metrics, constructs a dependency graph describing the dependencies between metrics, and generates a corresponding sequence of computation operators based on the dependency graph. The computation metrics include basic metrics and computation metrics. In response to the received real-time streaming data, the storage location of the basic indicator in the pre-allocated numerical vector is located through a pre-established index mapping, the current value of the basic indicator is updated according to the storage location, and the usage flag corresponding to the basic indicator is set to the updated state. Based on the dependency graph, the metrics to be calculated that directly depend on the updated basic metrics are retrieved, and the metrics to be calculated are processed by a preset recursive processing logic according to the sequence of calculation operators to perform streaming data processing.

[0009] In some embodiments, the preset recursive processing logic includes: Check whether all input dependencies of the metric to be calculated are in a valid state within the current processing cycle; If so, call the operator in the sequence of calculation operators that corresponds to the index to be calculated to perform the calculation; The calculation result is updated to the corresponding position in the numerical vector, and the usage flag corresponding to the indicator to be calculated is set to the updated state to trigger the calculation of other calculation indicators at the next level that depend on the indicator to be calculated.

[0010] In some embodiments, the engine parser parses user-defined computation metrics, constructs a dependency graph describing the dependencies between metrics, and generates corresponding computation operators based on the dependency graph, including: The expression tree of the computation metric is recursively traversed to identify the input dependencies referenced by the computation metric, and the dependency graph is established based on the input dependencies through a bidirectional linking mechanism. Perform topological sorting on the dependency graph to construct a computation update sequence that includes the execution hierarchy of computation metrics; The infix expression of the calculated index is converted into a postfix expression using the scheduling field algorithm; The postfix expression generates computation operators containing input source indices and output target indices, and the computation operators are organized according to the computation update sequence to form the computation operator sequence.

[0011] In some embodiments, the method further includes a circular dependency detection step during the construction of the dependency graph, specifically including: Pre-maintain the set of dependency paths and the set of nodes that have been detected; The nodes of the dependency graph are traversed using a depth-first search algorithm. During the traversal, if the current node is found to already exist in the dependency path set, a circular dependency is determined and the construction process is terminated. If the current node does not exist in the dependency path set, add it to the dependency path for detection, until all paths have been traversed.

[0012] In some embodiments, the method further includes, prior to responding to the received real-time streaming data: Parse the definitions of all calculated metrics and calculate the required storage space. The numerical vector is pre-allocated in memory to store the current values ​​of all basic and calculated indicators. The index mapping is constructed based on the correspondence between the unique identifier of each indicator and the array index in the numerical vector.

[0013] In some embodiments, the method further includes: A pre-defined marker vector of the same length as the numerical vector is used; The used tag vector is reset before each batch of real-time streaming data processing begins; When the value of the basic indicator or calculated indicator is updated, it is marked as valid at the corresponding subscript position of the used marker vector; By querying whether the corresponding values ​​of all input dependencies of the metric to be calculated in the usage tag vector are all valid tags, it is checked whether all input dependencies are in the valid state.

[0014] In some embodiments, the method further includes: In response to a metric change request, determine whether the change type is adding or deleting a metric; If an indicator is added, after verifying its legality, the node of the indicator to be processed is inserted into the dependency graph, and the sequence of computational operators is updated incrementally. If deleting a metric, check if the metric to be deleted has any dependencies; if not, remove the relevant nodes. The system monitors the cumulative amount of changes in monitoring metrics. If the number of deleted or changed metrics exceeds a preset reconstruction threshold, it triggers the reconstruction of the dependency graph and the sequence of computational operators. If not, it performs incremental updates on the dependency graph and the sequence of computational operators.

[0015] In some embodiments, after performing calculations by invoking the corresponding operators in the sequence of computational operators, the method further includes: Read pre-configured filter conditions or trigger conditions; Determine whether the calculation result meets the filtering condition, or whether the update behavior of the indicator to be calculated meets the triggering condition; If the conditions are met, extract the product identifier, indicator name, and value corresponding to the calculation result, and assemble them into an output record; The output records are written to a temporary buffer and submitted in batches to the final output table at the end of the processing cycle.

[0016] Secondly, embodiments of this application provide a streaming data processing system based on a reactive stateless computing engine, the system comprising: The building module is used to parse user-defined computation metrics through the engine parser, construct a dependency graph describing the dependencies between metrics, and generate a corresponding sequence of computation operators based on the dependency graph. The computation metrics include basic metrics and computation metrics. The preprocessing module is used to respond to the received real-time streaming data, locate the storage position of the basic indicator in the pre-allocated numerical vector through a pre-established index mapping, update the current value of the basic indicator according to the storage position, and set the usage flag corresponding to the basic indicator to the updated state. The computation module is used to retrieve, based on the dependency graph, the indicators to be calculated that are directly dependent on the updated basic indicators, and to perform preset recursive processing logic on the indicators to be calculated according to the sequence of computation operators to process the streaming data.

[0017] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0018] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0019] Compared to related technologies, the streaming data processing method and system based on a reactive stateless computing engine provided in this application introduces an engine parser to parse metric definitions and automatically construct dependency graphs, generating optimized postfix expressions to achieve a precise triggering mechanism based on dependencies. This triggers cascading computations of specific paths only when input data is updated, avoiding invalid computations. Simultaneously, by employing a stateless computing model and a value index pre-allocation strategy, memory usage is decoupled from the amount of data and only related to the number of metrics, achieving an update efficiency of O(1) complexity. Furthermore, it supports dynamic addition and deletion of metrics at runtime and loop detection, significantly improving the performance and flexibility of large-scale real-time computing while ensuring logical correctness. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the overall system architecture according to an embodiment of this application; Figure 2 This is a flowchart of a stream data processing method based on a reactive stateless computing engine according to an embodiment of this application; Figure 3 This is a flowchart of the cascading trigger calculation according to an embodiment of this application; Figure 4 This is a flowchart of the index dependency resolution according to an embodiment of this application; Figure 5 This is a structural block diagram of a streaming data processing system based on a reactive stateless computing engine according to an embodiment of this application; Figure 6 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0022] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0023] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0024] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0025] This application provides a streaming data processing method based on a reactive stateless computing engine. This method is applied to computing systems capable of high-performance streaming data analysis (e.g., the Dolphin DB database system). In this method, the reactive stateless computing engine serves as the core component, responsible for receiving real-time streaming data, parsing user-defined computing metrics, and performing real-time calculations based on the dependencies between metrics.

[0026] Figure 1 This is a schematic diagram of the overall system architecture according to an embodiment of this application. Figure 1 As shown, the system architecture mainly consists of a data input layer, a reactive stateless engine core, and a result output layer. The data input layer is responsible for receiving real-time data streams in standardized formats (such as narrow tables or tuples); the engine core includes a dependency parser (used to build dependency graphs and perform topological sorting) and a cascading trigger computation engine (used to perform accurate dependency identification and computation); the result output layer is responsible for formatting the computation results and outputting them to the target table.

[0027] Figure 2 This is a flowchart of a stream data processing method based on a reactive stateless computing engine according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps: S201 parses user-defined computation metrics through the engine parser, constructs a dependency graph describing the dependencies between metrics, and generates corresponding sequences of computation operators based on the dependency graph.

[0028] In this embodiment, the Reactive Stateless Engine Parser is the core of the initialization. Users can define the calculated metrics in either tabular or dictionary form. If using a tabular format, the input typically includes columns for calculation formulas, output metric identifiers, trigger conditions, and filtering conditions; if using a dictionary format, the output name and calculation formula are defined through key-value pair mapping.

[0029] Example table is as follows: formula| outputDeviceID| triggerOn |filter D00001_R03_OUT>-10 and D00001_R03_OUT<42 and D00001_R01_OUT andD00001_R02_OUT| MULTIPLY_Alert| D00001_R03_OUT |NULL D00001_R01_OUT + D00001_R02_OUT | R01_R02_Plus |NULL |NULL Specifically, the parsing process first involves indicator parsing and dependency identification. The parser recursively traverses the expression tree of each calculated indicator, distinguishing between the output indicator (Result) on the left and the input dependencies (Operand) on the right. An independent set of indicators is maintained for each logical product (e.g., a stock code), and a unique identifier is generated through a combination of "product name + indicator name". Based on this, a dependency graph is established through a bidirectional linking mechanism: for each node, both its dependent input nodes (incoming edges) and subsequent calculated nodes that depend on that node (outgoing edges) are recorded. This bidirectional indexing structure provides the topological foundation for subsequent data update propagation.

[0030] Furthermore, considering that the feasibility of the computational logic must be guaranteed during the construction of the dependency graph, a circular dependency detection step is included.

[0031] This step traverses the graph nodes using a depth-first search (DFS) algorithm. Two sets are maintained beforehand: the "dependency path set" currently being checked and the "set of nodes that have been checked." During traversal, if a node being visited is found to already exist in the dependency path set, a circular dependency is determined (e.g., A depends on B, B depends on A), the construction is terminated, and an exception is thrown. If a node is not in the path, it is added to the path, and the checking continues until the traversal is complete, ensuring that the generated dependency graph is a directed acyclic graph (DAG).

[0032] Subsequently, a topological sort is performed on the dependency graph. This step aims to determine the linear execution order of computations, ensuring that when a given metric is computed, all its dependent input metrics have been computed. The sorting employs a modified breadth-first search algorithm: first, nodes with an in-degree of zero (i.e., basic metrics that do not depend on other computed metrics) are identified as the first level; then, the output edges of these nodes are traversed, reducing the in-degree count of the corresponding target nodes; when the in-degree of a target node drops to zero, it is added to the next level. The resulting computation update sequence strictly follows the data flow.

[0033] Furthermore, to optimize machine execution efficiency, the parser utilizes the shunting-yard algorithm to convert user-defined infix expressions into postfix expressions (Reverse Polish Notation). During this process, an operator stack is maintained to handle priorities and ensure the correct order of operations. Based on the postfix expression, a specific sequence of computational operators is generated. Each operator object encapsulates the specific computational logic (such as addition, subtraction, multiplication, division, logical judgments, and function calls) and contains an input source index (pointing to its position in the numerical vector) and an output target index. In addition, the parser implements common subexpression elimination optimization, i.e., identifying and reusing identical intermediate computation results to avoid redundant calculations.

[0034] It should also be noted that memory pre-allocation and index construction are required before responding to real-time data. Specifically, the parser counts the number of all basic and calculated metrics, pre-allocates a contiguous vector of values ​​in memory to store the current values ​​of all metrics. Simultaneously, an index map is constructed, mapping the unique identifier of each metric to the array index of the value vector. This pre-allocation strategy avoids the overhead of dynamic memory allocation at runtime and reduces the time complexity of data access to O(1).

[0035] This step transforms complex business logic into a standardized sequence of postfix expressions and pre-allocates contiguous memory through the engine parser, eliminating the overhead of runtime script interpretation and the jitter caused by dynamic memory allocation, thus laying a solid and efficient underlying foundation for high-throughput streaming computing.

[0036] S202, in response to the received real-time streaming data, locate the storage location of the basic indicator in the pre-allocated numerical vector through the pre-established index mapping, update the current value of the basic indicator according to the storage location, and set the usage flag corresponding to the basic indicator to the updated state.

[0037] In this embodiment, this step is the entry point for data into the engine. The engine supports various input formats (such as narrow tables and tuples) and verifies whether the column structure of the input data (key columns, numeric columns, and timestamp columns) matches the configuration. The key column data is used for routing, locating the specific product processing unit based on the key value (single-key or double-key mode).

[0038] When new real-time data (usually basic indicators such as the latest price and trading volume) is received, complex calculations are not performed immediately. Instead, the basic data is updated first. Using the index mapping established in step S201, the physical address of the indicator in the value vector is directly located, and the new value is written.

[0039] Specifically, in the data input interface and format adaptation stage, the validity of the input parameters is first verified, supporting two main data input formats: single-table format and tuple format. The single-table format requires the input to be a complete table object, whose column structure is consistent with the dummyTable configured during engine initialization. The tuple format allows data to be provided in the form of independent vectors, which are internally assembled into a unified processing structure.

[0040] The number of columns in the input data must strictly match the number of input columns expected by the engine, including key columns, numeric columns, and optional timestamp columns. The system performs rigorous structure validation to ensure the correctness of the data format and avoid calculation anomalies caused by incorrect data format.

[0041] In the data parsing and structure extraction stage, after confirming the input format is valid, the data parsing phase begins. For tabular input, the engine directly extracts the corresponding column data from the table. For tuple-formatted input, the system assembles the vectors into a temporary table structure for easier unified processing.

[0042] The parsing process focuses on three core data components: key column data, numeric column data, and timestamp data. Key column data identifies the product and metric to which a data record belongs, supporting both single-key and double-key modes. Single-key mode is suitable for simple product identification, while double-key mode supports a two-level classification of product and metric.

[0043] The numeric column contains the raw data values ​​that need to be processed. The system ensures that the numeric type is compatible with the engine configuration and is typically required to support mathematical operations. Timestamp data is an optional component; when a time column is configured, the system extracts and records the time information for each data point.

[0044] During the parsing process, the system also handles the special case of scalar inputs. When the input data is a scalar, the system automatically packages it into a single-element vector to ensure the consistency of subsequent processing logic.

[0045] In the key column processing and data routing stages, different data processing strategies are adopted based on the number of configured key columns. Single-key column processing is suitable for simple data models, where the key column directly identifies the product, and the indicator information is determined through other methods. Double-key column processing supports more complex data models, where the first key column identifies the product, and the second key column identifies the specific indicator type.

[0046] In single-key column mode, the system assumes all input data corresponds to different metrics within the same product. The system routes data to the corresponding product processing unit based on the key value, updating the base value of the relevant metric. Double-key column mode provides more granular data routing capabilities. The system precisely locates the specific metric where data should be updated based on the combination of the product key and the metric key. This mode supports complex dependency calculations across products, such as relative and aggregate metrics.

[0047] Furthermore, the core task of data input is to update the current values ​​of basic indicators. Basic indicators refer to direct input data that does not rely on other indicators, such as the opening price, closing price, and trading volume of a stock.

[0048] To achieve precise dependency triggering, this implementation also maintains a usage flag vector (DirtyFlags) of the same length as the numerical vector. This vector is reset (e.g., all flags are set to False) before each batch of streaming data processing begins. When a fundamental metric is updated, its corresponding index in the flag vector is marked as "valid" or "updated" (True). This flag is the sole criterion for determining whether subsequent computations should be triggered, ensuring that unchanged data does not lead to invalid computations.

[0049] This flag plays a crucial role in subsequent calculation triggering; only metrics marked as updated will participate in the calculation triggering process. Furthermore, the flagging mechanism also supports some calculation functions. When certain dependent data is unavailable, the relevant calculated metrics will not be triggered, avoiding erroneous results based on incomplete data. This design improves the system's robustness and allows for graceful handling of data loss situations.

[0050] Figure 3 This is a flowchart of the cascading trigger calculation according to an embodiment of this application. For example... Figure 3 As shown, it receives narrow table data, iterates through each row of input, identifies the updated basic metrics, finds the directly dependent computed metrics if there are updates, checks if all input dependencies are ready, executes the computation and updates the results if ready, then recursively triggers the next level of dependency computation, and finally outputs the results to the target table; if the dependencies are not ready, it skips the computation.

[0051] Step S202 utilizes index mapping and tag vectors to perform in-situ updates and state marking of the basic data, avoiding full data scanning and copying, thus ensuring extremely low latency and extremely high concurrency processing capabilities when massive real-time data is accessed. S203, based on the dependency graph, retrieves the metrics to be calculated that directly depend on the updated basic metrics, and performs pre-defined recursive processing logic on the metrics to be calculated according to the sequence of calculation operators to process the streaming data.

[0052] After the data is updated, the cascading trigger calculation phase begins. Utilizing the outgoing edge information of the dependency graph, all next-level nodes (indicators to be calculated) that directly depend on the currently updated basic indicator are quickly retrieved.

[0053] For each metric to be calculated, the following pre-defined recursive processing logic is executed: In this embodiment, not all dependency updates trigger computation. By querying the state of all input dependencies of the metric to be calculated in the label vector, it is determined whether they are all in a valid state (or, according to specific logic, whether the necessary dependencies are valid). For example, if a metric depends on A and B, the input is considered ready only if A and B are both valid in the current period (or have historical valid values ​​and meet the triggering conditions).

[0054] Furthermore, if the validity check passes, the execution engine invokes the corresponding postfix expression operator sequence. The calculation process utilizes the operand stack to sequentially read the input values ​​from the numerical vector and perform arithmetic or logical operations.

[0055] Finally, the calculation result is written back to the corresponding position of the indicator in the numerical vector. After the calculation is completed, the state of the calculated indicator itself in "Use Tag Vector" is also set to "Updated". This action will trigger the calculation of other high-level indicators that depend on this calculated indicator at the next level, thus forming a recursive or cascading computation propagation until all affected path nodes have been updated.

[0056] This step, based on a dependency graph and a cascading triggering mechanism of flags, locates and recalculates only the metrics affected by data changes, avoiding the invalid and redundant calculations in the traditional full recalculation mode, and maximizing the utilization of computing resources while ensuring strong data consistency.

[0057] Specifically, the core value of data updates in this embodiment lies in triggering dependency computations. When a basic metric is updated, all computational metrics that depend on these basic metrics are automatically recalculated, and this may further cascade to trigger deeper dependency computations. The triggering process is based on a pre-built dependency graph. It iterates through the output dependency links of each updated basic metric to find all computational metrics that directly depend on it. For each such computational metric, it checks whether all its input dependencies are ready (i.e., all have been updated).

[0058] Once all input dependencies for a computational metric are ready, the computation operation for that metric is executed. The computation operation uses pre-generated postfix expressions and computation operators to ensure efficient and accurate numerical calculations. The computation result is directly updated to the corresponding position in the numerical vector, and the computational metric is marked as updated. Cascading execution is a key feature of the triggering process. After a computational metric is updated, it is recursively checked whether other metrics depend on this newly updated computational metric, forming a chain of computational propagation. This cascading mechanism ensures that all affected metrics in the entire dependency network are updated in a timely manner.

[0059] also, Figure 4 This is a flowchart of the index dependency resolution according to an embodiment of this application, such as... Figure 4 As shown, the metric dependency resolution process begins with the parser receiving user-defined metric data. First, it identifies the input format, supporting the parsing of tabular formula columns and trigger conditions, or dictionary-style variable mappings, and converts them uniformly into internal standard expression objects. Then, in the dependency graph construction phase, the parser extracts metric indices, analyzes reference relationships, and establishes bidirectional dependency links between metrics.

[0060] To ensure logical closure, circular dependency checks are performed, and deadlock paths are terminated upon detection. After successful verification, the computation graph optimization phase begins: a hierarchical computation update sequence is generated through topological sorting to ensure dependencies are prioritized; then, infix expressions are converted into efficient postfix expressions, and numerical indices are pre-allocated; finally, specific computation operators are generated, completing the initialization transformation from logical definition to execution engine.

[0061] Through steps S101 to S103 above, an engine parser is introduced to parse the indicator definitions and automatically construct a dependency graph, generating optimized postfix expressions to achieve a precise triggering mechanism based on dependencies. Cascaded computation of specific paths is triggered only when the input data is updated, avoiding invalid computations. Simultaneously, a stateless computation model and a value index pre-allocation strategy are adopted, decoupling memory usage from the amount of data to the number of indicators, achieving an update efficiency of O(1) complexity.

[0062] In the preferred embodiment provided in this example: The calculation process also involves filtering and output control. After the calculation is completed, pre-configured filtering conditions (such as numerical ranges) or trigger conditions (such as outputting only when a certain reference indicator changes) are read. If the conditions are met, the product identifier, indicator name, and value corresponding to the calculation result are extracted and assembled into an output record. To improve throughput, the output records are usually first written to a temporary buffer and then submitted in batches to the final output table at the end of the processing cycle.

[0063] In addition to numerical filtering, it also supports trigger conditions based on the status of other indicators. Trigger conditions allow the calculated indicator to be updated only when specific conditions are met, such as recalculating only when a reference indicator is updated. Trigger conditions are implemented through trigger indexes. The system configures trigger indicators in the calculation operator, and the calculation of the current indicator is only executed when the trigger indicator is marked as updated. This mechanism supports complex calculation timing control.

[0064] The combined use of triggering and filtering conditions provides fine-grained control over the computational process. Users can optimize computational performance and ensure the correctness of business logic by properly configuring triggering and filtering conditions.

[0065] Furthermore, the method in this embodiment also supports dynamic metric management. Upon responding to a metric change request, it determines whether the operation is an addition or deletion. If adding a metric, after verifying its validity (e.g., no circular dependencies), the new node is inserted into the dependency graph, and the computation operator sequence and extended numerical vector are incrementally updated. If deleting a metric, it checks whether the metric has any dependencies (i.e., whether there are outgoing edges). If dependencies exist, deletion is rejected to maintain the integrity of the graph; if no dependencies exist, the node is removed. To balance performance, the cumulative number of changes is monitored. When the number of changes exceeds a preset reconstruction threshold, a complete reconstruction of the entire dependency graph and operator sequence is triggered; otherwise, only incremental updates are performed. This mechanism ensures high processing performance while adapting to business changes.

[0066] Furthermore, the parser in this application employs a robust error handling mechanism, with exception handling implemented at critical operation points. Particularly during dynamic indicator management, the current state is saved before any modifications are executed, allowing for a rollback to a previous consistent state should an error occur during the operation.

[0067] State management includes maintaining product sets, dependencies, computation sequences, and value indexes. The system ensures that the computation graph is in a consistent state at all times, avoiding computational errors caused by partial updates. For memory resources, the system employs a strategy combining pre-allocation and dynamic adjustment to balance memory usage and performance requirements.

[0068] The parser in this embodiment implements several performance optimization techniques. The value index pre-allocation system allocates appropriate storage locations for all calculated metrics during the parsing phase, avoiding runtime memory allocation overhead. Dependency caching enables rapid identification of affected computational paths during data updates.

[0069] Expression optimization techniques include constant folding, common subexpression elimination, and dead code elimination. These optimizations are applied during the build phase to reduce runtime computation. The system also optimizes memory access patterns for streaming processing, improving cache utilization.

[0070] Secondly, this application also provides a stream data processing system based on a reactive stateless computing engine, which is implemented based on a reactive stateless computing engine. Figure 5 This is a structural block diagram of a streaming data processing system based on a reactive stateless computing engine according to an embodiment of this application, as shown below. Figure 5 As shown, the system includes: a construction module 50, a preprocessing module 51, and a calculation module 52, wherein: The construction module 50 is used to parse user-defined computation metrics through the engine parser, construct a dependency graph describing the dependencies between metrics, and generate a corresponding sequence of computation operators based on the dependency graph. The computation metrics include basic metrics and computation metrics. The preprocessing module 52 is used to respond to the received real-time streaming data, locate the storage position of the basic indicator in the pre-allocated numerical vector through a pre-established index mapping, update the current value of the basic indicator according to the storage position, and set the usage flag corresponding to the basic indicator to the updated state. The calculation module 53 is used to retrieve the indicators to be calculated that directly depend on the updated basic indicators based on the dependency graph, and to perform preset recursive processing logic on the indicators to be calculated according to the sequence of calculation operators to process the streaming data.

[0071] Through the above system, an engine parser is introduced to parse the indicator definitions and automatically construct the dependency graph, generating optimized postfix expressions to achieve a precise triggering mechanism based on dependencies. Cascading calculations of specific paths are triggered only when the input data is updated, avoiding invalid calculations. At the same time, a stateless computing model and a value index pre-allocation strategy are adopted, so that memory usage is only related to the number of indicators and decoupled from the amount of data, achieving an update efficiency of O(1) complexity.

[0072] In one embodiment, Figure 6 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 6 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 6 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network connection, the internal memory provides the environment for the operating system to run, the computer programs are executed by the processor to implement a streaming data processing method based on a reactive stateless computing engine, and the database stores data.

[0073] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0074] Furthermore, in conjunction with the information interaction method of the large-screen terminal in the above embodiments, this application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when the computer program is executed by a processor, it implements any of the streaming data processing methods based on a reactive stateless computing engine in the above embodiments.

[0075] In one embodiment, a computer program product is provided, comprising a computer program, characterized in that, when executed by a processor, the computer program implements any of the streaming data processing methods based on a reactive stateless computing engine in the above embodiments.

[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0077] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0078] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A streaming data processing method based on a reactive stateless computing engine, characterized in that, Based on a reactive stateless computing engine, the method includes: The engine parser parses user-defined computation metrics, constructs a dependency graph describing the dependencies between metrics, and generates a corresponding sequence of computation operators based on the dependency graph. The computation metrics include basic metrics and computation metrics. In response to the received real-time streaming data, the computation executor locates the storage location of the associated basic indicator in the pre-allocated numerical vector through a pre-established index mapping, updates the current value of the basic indicator according to the storage location, and sets the usage flag corresponding to the basic indicator to the updated state. Furthermore, based on the dependency graph, the metrics to be calculated that directly depend on the updated basic metrics are retrieved, and a preset recursive processing logic is executed on the metrics to be calculated according to the sequence of calculation operators to perform streaming data processing.

2. The method according to claim 1, characterized in that, The preset recursive processing logic includes: Check whether all input dependencies of the metric to be calculated are in a valid state within the current processing cycle; If so, call the operator in the sequence of calculation operators that corresponds to the index to be calculated to perform the calculation; The calculation result is updated to the corresponding position in the numerical vector, and the usage flag corresponding to the indicator to be calculated is set to the updated state to trigger the calculation of other calculation indicators at the next level that depend on the indicator to be calculated.

3. The method according to claim 1, characterized in that, The engine parser parses user-defined computation metrics, constructs a dependency graph describing the relationships between metrics, and generates corresponding computation operators based on the dependency graph, including: The expression tree of the calculated metric is recursively traversed to identify the input dependencies referenced by the calculated metric, and the dependency graph is established based on the input dependencies through a bidirectional linking mechanism. Perform topological sorting on the dependency graph to construct a computation update sequence that includes the execution hierarchy of computation metrics; The infix expression of the calculated index is converted into a postfix expression using the scheduling field algorithm; The postfix expression generates computation operators containing input source indices and output target indices, and the computation operators are organized according to the computation update sequence to form the computation operator sequence.

4. The method according to claim 3, characterized in that, In constructing the dependency graph, the method further includes a circular dependency detection step, specifically including: Pre-maintain the set of dependency paths and the set of nodes that have been detected; The nodes of the dependency graph are traversed using a depth-first search algorithm. During the traversal, if the current node is found to already exist in the dependency path set, a circular dependency is determined and the construction process is terminated. If the current node does not exist in the dependency path set, add it to the dependency path for detection, until all paths have been traversed.

5. The method according to claim 2, characterized in that, Prior to responding to the received real-time streaming data, the method further includes: Parse the definitions of all calculation indicators and calculate the storage space required for all calculation indicators; Based on the statistical results, the numerical vector is pre-allocated in memory, and the numerical vector is used to store the current values ​​of all basic indicators and calculated indicators. The index mapping is constructed based on the correspondence between the unique identifier of each indicator and the array index in the numerical vector.

6. The method according to claim 2, characterized in that, The method further includes: A pre-defined marker vector of the same length as the numerical vector is used; The used tag vector is reset before each batch of real-time streaming data processing begins; When the value of the basic indicator or calculated indicator is updated, it is marked as valid at the corresponding subscript position of the used marker vector; By querying whether the corresponding values ​​of all input dependencies of the metric to be calculated in the usage tag vector are all valid tags, it is checked whether all input dependencies are in the valid state.

7. The method according to claim 1, characterized in that, The method further includes: In response to a metric change request, determine whether the change type is adding or deleting a metric; If an indicator is added, the node of the indicator to be processed is inserted into the dependency graph, and the sequence of computational operators is updated incrementally. If deleting a metric, check if the metric to be deleted has any dependencies; if not, remove the relevant nodes. The system monitors the cumulative amount of changes in monitoring metrics. If the number of deleted or changed metrics exceeds a preset reconstruction threshold, it triggers the reconstruction of the dependency graph and the sequence of computational operators. If not, it performs incremental updates on the dependency graph and the sequence of computational operators.

8. The method according to claim 2, characterized in that, After performing calculations by invoking the corresponding operators in the sequence of computational operators, the method further includes: Read pre-configured filter conditions or trigger conditions; Determine whether the calculation result meets the filtering condition, or whether the update behavior of the indicator to be calculated meets the triggering condition; If the conditions are met, extract the product identifier, indicator name, and value corresponding to the calculation result, and assemble them into an output record; The output records are written to a temporary buffer and submitted in batches to the final output table at the end of the processing cycle.

9. A stream data processing system based on a reactive stateless computing engine, characterized in that, The system, implemented using a reactive stateless computing engine, includes: The building module is used to parse user-defined computation metrics through the engine parser, construct a dependency graph describing the dependencies between metrics, and generate a corresponding sequence of computation operators based on the dependency graph. The computation metrics include basic metrics and computation metrics. The preprocessing module is used to respond to the received real-time streaming data, locate the storage position of the basic indicator in the pre-allocated numerical vector through a pre-established index mapping, update the current value of the basic indicator according to the storage position, and set the usage flag corresponding to the basic indicator to the updated state. The computation module is used to retrieve, based on the dependency graph, the indicators to be calculated that directly depend on the updated basic indicators, and to perform preset recursive processing logic on the indicators to be calculated according to the sequence of computation operators to perform streaming data processing.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 8.

Images