Stateful function abstraction based financial factor streaming computation system, method and stream processing system

Abstract

The application discloses a financial factor streaming calculation system and method based on a stateful function abstraction and a stream processing system, and relates to the technical field of financial quantitative transaction systems.The financial factor streaming calculation system comprises an expression definition layer, a domain-specific language DSL for providing a domain-specific language DSL for representing business logic and acquiring a declarative DSL expression, an abstract syntax tree AST intermediate layer for parsing the acquired DSL expression into a standardized AST node tree, and each node of the AST node tree being configured with a unique calculation signature, and an execution engine layer for performing calculation and state management according to the parsed AST.The application can be constructed on a general stream calculation framework as an application layer, and by introducing DSL parsing, AST unified expression and state automatic hosting mechanisms, the underlying streaming details are shielded, and users can develop industrial-level stream-batch integrated factors by using declarative expressions.

Description

Technical Field

[0001] This invention relates to the field of financial quantitative trading system technology, and in particular to a financial factor streaming calculation system, method and streaming processing system based on state function abstraction. Background Technology

[0002] In quantitative financial trading, factor calculation is a core step. A financial factor expression may involve a series of complex requirements such as sliding window calculation, state maintenance, and unified batch and stream execution.

[0003] Currently, the mainstream factor calculation systems / frameworks in the industry can be mainly divided into the following four categories: Option 1: Dedicated time-series database solutions (such as DolphinDB, KDB+), which use self-developed proprietary languages ​​and storage engines, with tight coupling between computation and storage. However, this solution has a high learning curve, a closed ecosystem, and requires the introduction of a cumbersome database system, making lightweight deployment difficult.

[0004] Option 2: General-purpose stream computing frameworks (such as Apache Flink) that use SQL or the DataStream API to provide basic window operators. However, this option lacks financial-specific operators (such as TS_RANK and TS_ARGMAX), has limited ability to express complex logic in SQL, and is based on the JVM, resulting in weak computational performance and excessive resource consumption.

[0005] Option 3: Python data analysis frameworks (such as Pandas). Based on the Python language, they employ vectorized computation and have a rich API, adhering to data science ecosystem standards. However, this option is a traditional batch processing framework, supporting only batch processing and cannot be directly used for streaming live trading; moreover, state management requires manual maintenance and cannot be automatically persisted.

[0006] Option 4: A basic stream processing library (such as Bytewax) provides basic dataflow construction capabilities and stateful operator interfaces. However, this option is positioned as a general-purpose low-level runtime, rather than a financial computing engine for end users. Furthermore, directly using Bytewax for factor development faces the following problems: First, there is no financial DSL support: users still need to write cumbersome Python functions to process data, and cannot declare factors like formulas. Second, state logic is repetitive; common logic such as "sliding windows" requires rewriting state maintenance code in each factor, making it unreusable. Third, the development threshold is high, requiring users to deeply understand the state snapshot and recovery mechanisms of stream processing, rather than focusing on the financial logic itself.

[0007] In summary, existing financial factor calculation schemes either rely on large and comprehensive database systems (such as Scheme 1) or on underlying streaming frameworks (such as Scheme 2 and Scheme 4), which have the following defects: (1) Coupling problem between expression calculation and state management: In traditional factor calculation, state management of state calculations such as sliding window and cumulative statistics is mixed with business logic, which makes it difficult to reuse code, test difficult, and state persistence complex. (2) Insufficient expressive power of financial DSL: Existing financial calculation frameworks lack dedicated domain-specific languages ​​(DSLs), and users need to write a lot of imperative code to express declarative calculation logic (such as "20-day moving average"), resulting in low development efficiency. (3) Separation of batch processing and stream processing logic: Batch processing frameworks (such as Pandas) are used for backtesting of historical data, while different code implementations are used for live streaming calculations, resulting in inconsistencies between backtesting and live results and the risk of strategy drift. (4) High complexity of data source integration: Connecting to time series databases (such as ClickHouse) and message queues (such as Kafka) requires explicit schema definition, handling type conversion, and managing connection lifecycle, which increases the development burden. This forces developers to reinvent the wheel when building live trading systems, and to manually implement infrastructure such as sliding windows, state management, and serialization. Furthermore, inconsistencies between backtesting and live trading can easily arise due to flawed logic. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a financial factor streaming computation system, method, and stream processing system based on stateful function abstraction. The financial factor streaming computation system provided by this invention can be built on top of a general stream computing framework (such as Bytewax) as an application layer. By introducing DSL parsing, unified AST expression, and automatic state management mechanisms, it shields the underlying streaming details, allowing users to develop industrial-grade integrated stream and batch factor processing using declarative expressions.

[0009] To achieve the above objectives, the present invention provides the following technical solution: A streaming financial factor calculation system based on stateful function abstraction, the system comprising: The expression definition layer is used to provide a domain-specific language (DSL) for representing business logic, as well as to obtain declarative DSL expressions; The Abstract Syntax Tree (AST) intermediate layer is used to parse the acquired DSL expression into a standardized AST node tree, where each node of the AST node tree is configured with a unique computational signature. The execution engine layer is used to perform calculations and state management based on the parsed AST.

[0010] Furthermore, the execution engine layer is configured with a context manager, a stateful function library, and an execution engine; The context manager serves as the system's state management center and is configured with a state hosting pool for unified storage and management of all state objects with state functions. The stateful function library contains a set of computation functions configured with a standard interface. The interface physically separates the computation logic from the state data. The computation functions obtain the state object through the context manager and perform incremental computation. The execution engine is used to traverse the AST, call the stateful function library according to the driving mode, and use the context manager to cache and reuse the results of subexpressions.

[0011] Furthermore, the DSL is a language specific to the financial field, used to represent business logic in the financial sector; The DSL expression is a financial factor expression, which supports both string expressions and Python operator overloading.

[0012] Furthermore, the context manager is configured to: perform state management, including maintaining state instances of all stateful functions and shielding the creation and updating of states from the outside; perform cache management, including caching the computation results of subexpressions based on AST signatures for reuse; and perform instance management, including managing the lifecycle of function instances.

[0013] Furthermore, during state management, when the execution engine traverses to a stateful function node, it requests the state object corresponding to the stateful function from the context manager. At this point, the context manager is configured to: search the state management pool for the state object of the function; if the state object of the function is not found, automatically create a new state; if the state object of the function already exists, return the existing state.

[0014] Furthermore, the steps for performing calculations through the execution engine layer are as follows: The execution engine issues a computation request; After the context manager receives the computation request, it checks the subexpression cache; if the subexpression is cached, it returns the cached result; otherwise, it performs the following steps: Get or create a state object; Send the computation logic Compute to the stateful function; After receiving the aforementioned computation logic Compute, the state function reads the historical data of the state from the state object, concatenates the historical data with the new data to obtain concatenated data, executes the core algorithm on the concatenated data, that is, performs computation, obtains updated data after computation, writes the updated data back to the state object, and returns the computation result to the context manager. After receiving the calculation result, the context manager stores it in the cache and returns the result to the execution engine.

[0015] Furthermore, the driving mode includes two types: batch processing mode and stream processing mode. When performing calculations, the execution engine layer uses the same AST to automatically adapt to the two driving modes to ensure logical consistency. In the batch processing mode, the context manager temporarily maintains the state in memory, processes the entire historical data DataFrame at once, and destroys the state after the calculation is completed; In the stream processing mode, the AST is encapsulated as an operator of a preset stream computing framework; and, using the checkpointing mechanism of the stream computing framework, all state objects in the context manager are serialized and persistently stored.

[0016] Furthermore, the computation steps in stream processing mode are as follows: Data ingestion steps: After receiving the real-time data stream, the data is assembled into micro-batches; State recovery step: The operator recovers the context manager's state object from the checkpoint; Incremental computation steps: Call the stateful function to process new data and update the state in the context manager; State snapshot step: After the calculation is completed, the context manager's state object is serialized and saved to the checkpoint, waiting for the next trigger.

[0017] Furthermore, it also includes a schema-free connector (Schema-Free) for guiding full historical data (DataFrame), the schema-free connector (Schema-Free) being configured to: It performs automatic inference, which can automatically infer the column type of the target system based on the data type of the full historical data; it also performs automatic table creation, which can automatically generate data definition language (DDL) statements and create the table when the target table does not exist; and it has a unified interface to shield the upper layer from the differences between clients of different underlying systems.

[0018] This invention also provides a streaming calculation method for financial factors based on stateful function abstraction, comprising the following steps: Declarative DSL expressions are obtained through the expression definition layer; the expression definition layer provides a domain-specific language DSL for representing business logic; The DSL expression is parsed by the DSL parser in the intermediate layer of the Abstract Syntax Tree (AST), and a standardized AST node tree is generated based on the parsing result. Each node in the AST node tree is configured with a unique computation signature. The execution engine layer performs calculations and manages the state based on the parsed AST.

[0019] The present invention also provides a stream processing system, including a stream computing framework, wherein the stream computing framework is used as a bottom layer system, and the aforementioned system is connected on the bottom layer system as an upper layer application system.

[0020] Compared with the prior art, the present invention, by adopting the above technical solutions, has the following advantages and positive effects: The financial factor streaming computing system provided by the present invention can be built on a general streaming computing framework (such as Bytewax) as an application layer. By introducing DSL parsing, AST unified expression and automatic state management mechanism, the underlying streaming details are shielded, allowing users to develop industrial-grade integrated streaming and batch factoring using declarative expressions.

[0021] Compared to existing dedicated time-series databases such as DolphinDB / KDB+, this invention is characterized by its openness and lightweight nature. On the one hand, it is based on the Python ecosystem and can be seamlessly integrated with AI / ML libraries, eliminating the need for developers to learn dedicated scripting languages. On the other hand, this invention can be embedded as a library into applications, eliminating the need to deploy cumbersome database services.

[0022] Compared to existing solutions such as Apache Flink / Bytewax, this invention has a higher level of abstraction, providing a dedicated financial DSL and encapsulation of stateful functions, eliminating the need for users to write low-level state management code (such as ValueState operations). Furthermore, it enables transparent state management, automatically handling state maintenance for common financial logic such as sliding windows, significantly lowering the development threshold. Simultaneously, it employs a unified batch and streaming execution mechanism, ensuring complete consistency between batch and streaming, truly achieving a single codebase running in two places, eliminating logical differences between backtesting and live trading.

[0023] Compared to existing batch processing frameworks such as Pandas, this invention empowers stream computing, allowing the Pandas-style development experience to be used in real-time stream computing scenarios; at the same time, it optimizes performance by significantly reducing the overhead of repeated calculations through AST analysis and subexpression caching. Attached Figure Description

[0024] Figure 1 The logical structure diagram of a financial factor streaming calculation system based on state function abstraction provided in this embodiment of the invention.

[0025] Figure 2 A flowchart illustrating the execution calculation and state management of the execution engine layer provided in this embodiment of the invention.

[0026] Figure 3The logical structure diagram of the batch-stream unified execution mechanism provided in the embodiments of the present invention is shown. Detailed Implementation

[0027] The following detailed description, in conjunction with the accompanying drawings and specific embodiments, provides a further detailed account of the financial factor streaming calculation system, method, and streaming processing system based on stateful function abstraction disclosed in this invention. It should be noted that techniques (including methods and apparatus) known to those skilled in the art may not be discussed in detail, but where appropriate, such known techniques are considered part of the specification. Furthermore, other examples of exemplary embodiments may have different values. The structures, proportions, sizes, etc., depicted in the accompanying drawings are merely illustrative of the content disclosed in this specification for the understanding and reading of those skilled in the art, and are not intended to limit the conditions under which the invention can be implemented.

[0028] In the description of the embodiments of this application, " / " means "or", and "and / or" is used to describe the relationship between related objects, indicating that there can be three relationships. For example, "A and / or B" means: A and B exist alone, B exists alone, and A and B exist simultaneously. In the description of the embodiments of this application, "multiple" refers to two or more.

[0029] The technical concept and solution of the present invention will be described below based on exemplary application scenarios.

[0030] The relevant technical terms are explained as follows: 1) DSL, short for Domain Specific Language: This refers to a language specific to a particular domain. In the financial field, a domain-specific language is a language similar to Excel formulas, specifically designed for writing financial logic. It is simpler and more intuitive than general-purpose programming languages ​​(such as Python / C++).

[0031] 2) AST, short for Abstract Syntax Tree: Computers translate human-written formulas (such as a+b) into a tree structure (+ is the root, and a and b are the leaves), making it easier for programs to understand and execute.

[0032] 3) Stateful Computing: This refers to computation results that depend not only on the current input but also on previous inputs. For example, "today's cumulative increase" = "yesterday's cumulative increase" (state) + "today's increase".

[0033] 4) Checkpoint: Just like a game's "save". The system periodically saves all current states to the hard drive. In case of a power outage, the system can load the save and continue calculating after restarting, and the data will not be lost. Example

[0034] This invention provides a streaming financial factor calculation system based on stateful function abstraction. The system adopts a three-layer architecture: expression-abstract syntax tree (AST)-execution engine. Specifically, the system may include an expression definition layer, an AST intermediate layer, and an execution engine layer.

[0035] The Expression Definition Layer is used to provide a Domain-Specific Language (DSL) for representing business logic, and to obtain declarative DSL expressions.

[0036] See Figure 1 As shown, based on the expression definition layer, users can configure DSL expressions that describe business logic.

[0037] Preferably, the DSL is a financial domain-specific language used to represent business logic in the financial field. In this case, the DSL expression is a financial factor expression.

[0038] In this embodiment, the financial DSL expression can support both string expressions and Python operator overloading. As an example, and not a limitation, the string expression could be m_avg(close, 20), m_stddev(close, 20), etc.

[0039] The Abstract Syntax Tree Layer (AST) is used to parse the acquired DSL expression into a standardized AST node tree. Each node in the AST node tree (such as FunctionCallNode, BinaryOpNode) is configured with a unique computation signature for subsequent cache optimization.

[0040] For details, see Figure 1 As shown, the intermediate layer of the AST may include a construction layer module, a DSL parser, and an AST module. The construction layer module is used to perform syntactic analysis on the expression to obtain a hierarchical expression structure; the DSL parser is used to parse the hierarchical expression to obtain the combinational logic and operational intent of the expression; the AST module is used to generate an Abstract Syntax Tree (AST) based on the parsing results.

[0041] When parsing DSL expressions, you can use existing Lark (supports Python), ANTLR (supports Java), PLY (supports Python), or a handwritten recursive descent parser, etc. There are no restrictions here, and they will not affect the system architecture.

[0042] The Execution Engine Layer is the core component of the system, used to perform calculations and state management based on the parsed AST, i.e., it is responsible for the specific calculation execution and state management.

[0043] In practice, the execution engine layer can be configured with a context manager, a stateful function library, and an execution engine.

[0044] The context manager, as the system's state management center, is configured with a state hosting pool for the unified storage and management of all state objects with state functions.

[0045] The stateful function library contains a set of computation functions configured with standard interfaces; that is, it configures standard function interfaces. These interfaces physically separate computation logic (Compute) from state data (State). The computation functions obtain state objects through a context manager for incremental computation. See also... Figure 1 As shown, a function registry is configured for the stateful function library to register functions.

[0046] The execution engine is used to traverse the AST, call the aforementioned stateful function library according to the driving mode (batch processing or stream processing), and use the aforementioned context manager to cache and reuse the results of subexpressions.

[0047] Preferably, in this embodiment, the context manager is configured to: perform state management, including maintaining state instances of all stateful functions and shielding the creation and updating of states from the outside; and perform cache management, including caching the calculation results of subexpressions based on AST signatures for reuse, thereby minimizing redundant calculations; and perform instance management, including managing the lifecycle of function instances.

[0048] In this embodiment, when performing state management, when the execution engine traverses to a stateful function node, it requests the state object corresponding to the stateful function from the context manager.

[0049] At this point, the context manager is configured to: search the state management pool for the state object of the function; if the state object of the function is not found, automatically create a new state; if the state object of the function already exists, return the existing state.

[0050] See Figure 2The following diagram illustrates the typical steps involved in performing computation at the execution engine layer: First, the execution engine issues a calculation request, such as requesting to calculate m_avg(date, window=20).

[0051] After receiving the computation request, the context manager checks the sub-expression cache.

[0052] If a cache exists, return the cached result.

[0053] When there is no cache, perform the following steps: obtain or create a state object (state); send the computation logic (Compute) to a stateful function—for example... Figure 2 The compute(state, date, window) function is sent to the stateful function m_avg. The compute logic involves multiple business operations such as sliding window calculation, state maintenance, and unified batch execution of date.

[0054] After the stateful function receives the aforementioned computation logic `Compute`, it reads the state's historical data from the state object, concatenates the historical data `history` with the new data `new`, and obtains the concatenated data. The core algorithm, i.e., computation, is then executed on the concatenated data. After computation, updated data is obtained, written back to the state object, and the computation result is returned to the context manager.

[0055] After receiving the calculation result, the context manager stores it in the cache and returns the result to the execution engine.

[0056] In this embodiment, the system adopts a unified batch and stream execution mechanism. Specifically, the driving mode can include both batch processing mode and stream processing mode. When performing calculations, the execution engine layer uses the same AST to automatically adapt to the two driving modes to ensure logical consistency. See [link to documentation]. Figure 3 As shown.

[0057] In the batch processing mode, the context manager temporarily maintains the state in memory, processes the entire historical data DataFrame at once, and destroys the state after the calculation is complete. This mode is suitable for strategy development and historical backtesting operations.

[0058] In the stream processing mode, the AST is encapsulated as an operator of a predefined stream computing framework (for example, the Bytewax framework). Furthermore, in stream processing mode, the checkpointing mechanism of the stream computing framework can be utilized to serialize and persistently store all state objects in the context manager (i.e., achieve state persistence) for fault recovery. This mode is suitable for live trading operations.

[0059] Specifically, the computation in the streaming mode may include data ingestion, state recovery, incremental computation, and incremental computation steps.

[0060] Data ingestion steps: After receiving the real-time data stream, the data is assembled into micro-batches.

[0061] State recovery step: The operator recovers the context manager's state object from the checkpoint.

[0062] Incremental calculation steps: Call the stateful function to process new data and update the state in the context manager.

[0063] State snapshot step: After the calculation is completed, the context manager's state object is serialized and saved to the checkpoint, waiting for the next trigger.

[0064] In this embodiment, the state serialization can use Pickle, JSON, Protobuf, or Apache ArrowIPC, etc., to improve cross-language compatibility.

[0065] In another embodiment of this example, a schema-free connector for guiding full historical data (DataFrame) may also be included.

[0066] The schema-free connector is configured to: perform automatic inference, automatically inferring the column types of the target system based on the data types of the full historical data, for example, automatically inferring the column types of the target system (such as ClickHouse) based on the data type dtypes of Pandas DataFrame; perform automatic table creation, automatically generating Data Definition Language (DDL) statements and creating the table when the target table is determined not to exist; and provide a unified interface to shield the upper layer from client differences between different underlying systems (such as Kafka, ClickHouse, etc.).

[0067] This invention also provides a streaming calculation method for financial factors based on stateful function abstraction.

[0068] The method includes the following steps: S100, obtain declarative DSL expressions through the expression definition layer; the expression definition layer provides a domain-specific language DSL to represent business logic.

[0069] S200 parses the obtained DSL expression through the DSL parser in the intermediate layer of the Abstract Syntax Tree (AST) and generates a standardized AST node tree based on the parsing results. Each node in the AST node tree is configured with a unique computational signature.

[0070] S300, the execution engine layer performs calculations and performs state management based on the parsed AST.

[0071] In this embodiment, the execution engine layer is the core component of the system, responsible for specific computation execution and state management. In practice, the execution engine layer can be configured with a context manager, a stateful function library, and an execution engine.

[0072] The context manager, as the system's state management center, is configured with a state hosting pool for the unified storage and management of all state objects with state functions.

[0073] The stateful function library contains a set of computation functions configured with standard interfaces. These interfaces physically separate computation logic (Compute) from state data (State). The computation functions obtain state objects through a context manager for incremental computation. A function registry is configured corresponding to the stateful function library for registering functions.

[0074] The execution engine is used to traverse the AST, call the stateful function library according to the driving mode (batch processing or stream processing), and use the context manager to cache and reuse the results of subexpressions.

[0075] Other technical features are described in the preceding embodiments and will not be repeated here.

[0076] The present invention also provides a stream processing system, including a stream computing framework, wherein the stream computing framework is used as the underlying system, and the financial factor stream computing system based on stateful function abstraction is connected on the underlying system as the upper-layer application system.

[0077] The underlying system can be Bytewax, Apache Flink (PyFlink), Kafka Streams, or a self-developed event loop engine system, etc. The upper-layer application system is built on these stream computing frameworks as an application layer. By introducing DSL parsing, unified AST expression, and automatic state management mechanism, the streaming details of the underlying system can be shielded, and users can develop industrial-grade stream and batch processing technologies using declarative mathematical formulas.

[0078] Other technical features are described in the preceding embodiments and will not be repeated here.

[0079] In the above description, the disclosure of this invention is not intended to limit itself to these aspects. Rather, within the scope of the objectives of this disclosure, components can be selectively and operationally combined in any number. Furthermore, terms such as “comprising,” “encompassing,” and “having” should be interpreted by default as inclusive or open-ended, rather than exclusive or closed, unless explicitly defined as such. All technical, scientific, or other terms are to be understood by those skilled in the art, unless defined as such. Public terms found in dictionaries should not be interpreted in the context of the relevant technical documents in an overly idealistic or impractical manner, unless explicitly defined as such in this disclosure. Any modifications or alterations made by those skilled in the art based on the foregoing disclosure are within the scope of the claims.

Claims

1. A streaming financial factor calculation system based on stateful function abstraction, characterized in that... include: The expression definition layer is used to provide a domain-specific language (DSL) for representing business logic, as well as to obtain declarative DSL expressions; The Abstract Syntax Tree (AST) intermediate layer is used to parse the acquired DSL expression into a standardized AST node tree, where each node of the AST node tree is configured with a unique computational signature. The execution engine layer is used to perform calculations and state management based on the parsed AST.

2. The system according to claim 1, characterized in that, The execution engine layer is configured with a context manager, a stateful function library, and an execution engine. The context manager serves as the system's state management center and is configured with a state hosting pool for unified storage and management of all state objects with state functions. The stateful function library contains a set of computation functions configured with a standard interface. The interface physically separates the computation logic from the state data. The computation functions obtain the state object through the context manager and perform incremental computation. The execution engine is used to traverse the AST, call the stateful function library according to the driving mode, and use the context manager to cache and reuse the results of subexpressions.

3. The system according to claim 1 or 2, characterized in that, The DSL is a language specific to the financial field, used to represent business logic in the financial sector; The DSL expression is a financial factor expression, which supports both string expressions and Python operator overloading.

4. The system according to claim 2, characterized in that, The context manager is configured to: manage state, including maintaining state instances for all stateful functions and shielding the creation and updating of state from external access; and, Implement cache management, including caching the computation results of subexpressions based on AST signatures for reuse; as well as, Perform instance management, including managing the lifecycle of function instances.

5. The system according to claim 4, characterized in that, When performing state management, when the execution engine traverses to a stateful function node, it requests the state object corresponding to the stateful function from the context manager. At this point, the context manager is configured to: search the state management pool for the state object of the function; if the state object of the function is not found, automatically create a new state; if the state object of the function already exists, return the existing state.

6. The system according to claim 4, characterized in that, The steps for performing calculations through the execution engine layer are as follows: The execution engine issues a computation request; After the context manager receives the computation request, it checks the subexpression cache; if the subexpression is cached, it returns the cached result; otherwise, it performs the following steps: Get or create a state object; Send the computation logic Compute to the stateful function; After receiving the aforementioned computation logic Compute, the state function solution reads the historical data of the state from the state object and concatenates the historical data of the state with the new data to obtain the concatenated data. The core algorithm is executed on the concatenated data, that is, the calculation is performed; after the calculation is performed, the updated data is obtained, the updated data is written back to the state object, and the calculation result is returned to the context manager; After receiving the calculation result, the context manager stores it in the cache; And, the result is returned to the execution engine.

7. The system according to claim 2, characterized in that, The driving modes include two types: batch processing mode and stream processing mode. When performing calculations, the execution engine layer uses the same AST to automatically adapt to the two driving modes to ensure logical consistency. In the batch processing mode, the context manager temporarily maintains the state in memory, processes the entire historical data DataFrame at once, and destroys the state after the calculation is completed; In the stream processing mode, the AST is encapsulated as an operator of a preset stream computing framework; Additionally, the checkpointing mechanism of the stream computing framework is used to serialize and persistently store all state objects in the context manager.

8. The system according to claim 7, characterized in that, The computation steps in stream processing mode are as follows: Data ingestion steps: After receiving the real-time data stream, the data is assembled into micro-batches; State recovery step: The operator recovers the context manager's state object from the checkpoint; Incremental computation steps: Call the stateful function to process new data and update the state in the context manager; State snapshot step: After the calculation is completed, the context manager's state object is serialized and saved to the checkpoint, waiting for the next trigger.

9. The system according to claim 7, characterized in that, It also includes a schema-free connector, Schema-Free, for guiding full historical data DataFrames, which is configured to: It can automatically infer the column type of the target system based on the data type of the entire historical data set; and, Automatic table creation can automatically generate Data Definition Language (DDL) statements and create the table when the target table does not exist; and, A unified interface is used to shield the upper layer from the differences in clients of different underlying systems.

10. A streaming calculation method for financial factors based on stateful function abstraction, characterized in that... Including the following steps: Declarative DSL expressions are obtained through the expression definition layer; the expression definition layer provides a domain-specific language DSL for representing business logic; The DSL expression is parsed by the DSL parser in the intermediate layer of the Abstract Syntax Tree (AST), and a standardized AST node tree is generated based on the parsing result. Each node in the AST node tree is configured with a unique computation signature. The execution engine layer performs calculations and manages the state based on the parsed AST.

11. A stream processing system, comprising a stream computing framework, characterized in that, The stream computing framework is used as the underlying system, and the system described in any one of claims 1-9 is connected on the underlying system as the upper-layer application system.

Images