Real-time feature engineering processing method and system based on Akka architecture and SQL operator

Abstract

The application discloses a real-time feature engineering processing method and system based on an Akka architecture and SQL operators and belongs to the field of privacy calculation. The application aims to solve the performance bottleneck that it is difficult to complete the calculation of ten thousand-dimensional features in a single request within milliseconds in the scene of financial risk control. The core of the technical scheme is that a real-time feature calculation request is converted into a specific calculation instruction through an asynchronous filtering chain, is submitted to an Akka Actor concurrent task scheduling system, a feature operator is called from a pre-defined SQL operator library, the Actor is scheduled in parallel to execute, and a feature vector is returned by aggregating the calculation result. The system supports request bypass storage and historical accurate replay, and the main use is to provide high-concurrency and low-latency feature calculation for online decision-making, achieve millisecond-level response, improve the flexibility of feature development, and provide the ability of tracing the calculation process.

Description

Technical Field

[0001] This invention relates to the field of privacy computing and real-time decision-making systems, specifically to a real-time feature engineering processing method and system based on the Akka architecture and SQL operators. Background Technology

[0002] In data-driven real-time decision-making, such as financial fraud prevention and transaction monitoring, the system needs to make a judgment within a very short time after receiving a request. This judgment relies on a high-dimensional feature vector calculated in real time from the raw data. This vector may contain thousands or even tens of thousands of feature dimensions, such as the deviation of a user's current transaction amount from the recent average, or the number of geographical locations where the device was active in the past hour. This process of transforming raw data into features usable by the model, known as feature engineering, is the most computationally intensive and performance-demanding part of the entire decision-making chain.

[0003] Currently, real-time feature engineering solutions mainly fall into three technical paths, but none of them can simultaneously meet the requirements of high real-time performance, high-dimensional complex computation, and high development flexibility:

[0004] The first category is solutions based on streaming computing frameworks, such as Apache Flink or Spark Streaming. These solutions write feature computation logic as stream processing jobs. Their disadvantages include inherently high latency; the micro-batch processing architecture results in end-to-end latency typically in the hundreds of milliseconds or even seconds, making it difficult to meet millisecond-level requirements; high resource consumption; the framework itself is heavyweight, with significant startup and scheduling overhead, resulting in low cost-effectiveness in scenarios requiring instantaneous computation and resource release; and complex integration with online systems, leading to high operational costs.

[0005] The second type is a pre-computation-based caching scheme, which pre-computes features and stores them in a high-speed cache such as Redis. This scheme cannot handle real-time variables, cannot compute features that depend on the current request context, and lacks flexibility; it suffers from feature combination explosion, and storing the pre-computation results of all possible features is costly; feature logic is difficult to maintain, scattered across offline scripts and cache keys, making it difficult to track and change.

[0006] The third type involves hard-coding the feature calculation logic into the business backend service. While this method has relatively low latency, it suffers from bottlenecks in concurrency performance. Traditional multi-threaded programming models may experience performance degradation due to lock contention and resource scheduling when dealing with tens of thousands of feature calculations. It also suffers from poor maintainability and scalability, as the feature logic is deeply coupled with the business code, requiring any modifications to be redeveloped, tested, and deployed again, making it unable to respond to rapidly changing business needs. Furthermore, it lacks standardization and reusability, resulting in inconsistent quality of the calculation code.

[0007] Furthermore, existing solutions generally lack support for the observability and traceability of the feature calculation process. When model performance fluctuates or online issues arise, it is difficult to accurately reproduce the feature calculation status of historical requests, leading to difficulties in problem localization and lengthy model iteration and testing cycles.

[0008] Therefore, there is an urgent need in the current technology field for a feature engineering processing solution that can support real-time computation of complex features in milliseconds and tens of thousands of dimensions, while providing agile feature development, deployment and operation and maintenance capabilities, and has the characteristics of fault isolation and traceability of the computation process, so as to support high-requirement real-time intelligent decision-making business. Summary of the Invention

[0009] To address the shortcomings of existing solutions in terms of real-time performance, flexibility, and high-dimensional computing capabilities, a real-time feature engineering processing method and system based on the Akka architecture and SQL operators is provided.

[0010] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A real-time feature engineering processing method based on Akka architecture and SQL operators includes the following steps: The system receives real-time feature calculation requests, converts these requests into calculation instructions using an asynchronous filter chain, and submits the calculation instructions to a concurrent task scheduling system. The concurrent task scheduling system is built based on the Akka Actor model. In the concurrent task scheduling system, based on the calculation instructions, several SQL feature operators to be invoked are determined from a predefined SQL feature operator library; the SQL feature operator library includes standardized calculation units for feature generation and processing. The concurrent execution mechanism of the Akka Actor model allows multiple Actors to be scheduled to execute computational tasks composed of the SQL feature operators in parallel, thereby obtaining computational results. The calculation results of each Actor are aggregated to generate a feature vector; and the feature vector, the real-time feature calculation request and its associated context data are synchronously stored for offline replay.

[0011] As one possible implementation, the SQL feature operators in the SQL feature operator library are managed and dynamically expanded through a service provider interface. The implementation of newly added SQL feature operators is provided in the form of a plug-in and integrated into the SQL feature operator library without restarting the system.

[0012] As one possible implementation, the SQL feature operators in the SQL feature operator library include multiple categories, including at least: data preprocessing operators for data preprocessing, basic aggregation function operators for statistical aggregation, and advanced time series aggregation operators for complex time series analysis.

[0013] As one possible implementation method, it also includes simulation and backtracking of the feature calculation process, specifically: Real-time feature calculation requests processed online and their context data are stored synchronously. In an offline environment, the same concurrent task scheduling system and SQL feature operator library as in the online environment are invoked to recalculate and simulate the real-time feature calculation requests stored synchronously.

[0014] A real-time feature engineering processing system based on Akka architecture and SQL operators, for implementing the method described in any of the following: Request processing module: used to receive real-time feature calculation requests, convert the real-time feature calculation requests into calculation instructions through an asynchronous filter chain, and submit the calculation instructions to the concurrent task scheduling system; the concurrent task scheduling system is built based on the Akka Actor model; Operator library module: used in the concurrent task scheduling system to determine several SQL feature operators to be invoked from a predefined SQL feature operator library according to the calculation instructions; the SQL feature operator library includes standardized calculation units for feature generation and processing; Concurrent scheduling and execution module: used to schedule multiple Actors to execute computational tasks composed of SQL feature operators in parallel through the concurrent execution mechanism of the Akka Actor model, and obtain computational results; Replay simulation module: used to aggregate the calculation results of each Actor, generate feature vectors; and synchronously store the feature vectors, real-time feature calculation requests and their associated context data for offline replay.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the method described in any of the following ways: The system receives real-time feature calculation requests, converts these requests into calculation instructions using an asynchronous filter chain, and submits the calculation instructions to a concurrent task scheduling system. The concurrent task scheduling system is built based on the Akka Actor model. In the concurrent task scheduling system, based on the calculation instructions, several SQL feature operators to be invoked are determined from a predefined SQL feature operator library; the SQL feature operator library includes standardized calculation units for feature generation and processing. The concurrent execution mechanism of the Akka Actor model allows multiple Actors to be scheduled to execute computational tasks composed of the SQL feature operators in parallel, thereby obtaining computational results. The calculation results of each Actor are aggregated to generate a feature vector; and the feature vector, the real-time feature calculation request and its associated context data are synchronously stored for offline replay.

[0016] A computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in any of the following: The system receives real-time feature calculation requests, converts these requests into calculation instructions using an asynchronous filter chain, and submits the calculation instructions to a concurrent task scheduling system. The concurrent task scheduling system is built based on the Akka Actor model. In the concurrent task scheduling system, based on the calculation instructions, several SQL feature operators to be invoked are determined from a predefined SQL feature operator library; the SQL feature operator library includes standardized calculation units for feature generation and processing. The concurrent execution mechanism of the Akka Actor model allows multiple Actors to be scheduled to execute computational tasks composed of the SQL feature operators in parallel, thereby obtaining computational results. The calculation results of each Actor are aggregated to generate a feature vector; and the feature vector, the real-time feature calculation request and its associated context data are synchronously stored for offline replay.

[0017] Compared with the prior art, the present invention has the following beneficial effects: By employing a concurrent scheduling and task decomposition mechanism based on the Akka Actor model, combined with a predefined SQL operator library, this invention can decompose complex feature calculation tasks for a single request into parallel subtasks, completing the real-time generation of tens of thousands of dimensional feature vectors within millisecond latency, supporting real-time decision-making needs in practical scenarios. Through a dynamically extensible SQL operator library, new features can be rapidly developed through configuration and integrated into the system as plugins without requiring restarts or modifications to the core code, significantly shortening the feature iteration cycle and reducing development and maintenance costs. By persisting online request snapshots through a bypass storage mechanism, accurate replay can be performed in an offline environment using the exact same computing engine as online, for problem diagnosis, feature verification, and model training, enhancing the system's observability and maintainability. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the method of the present invention; Figure 2 This is a schematic diagram of the modules of the system of the present invention. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the embodiments. The following embodiments are explanations of the present invention, but the present invention is not limited to the following embodiments.

[0021] Example 1: A real-time feature engineering processing method based on Akka architecture and SQL operators, such as Figure 1 As shown, it includes the following steps: S100: Receive a real-time feature calculation request, convert the real-time feature calculation request into a calculation instruction through an asynchronous filter chain, and submit the calculation instruction to the concurrent task scheduling system; the concurrent task scheduling system is built based on the Akka Actor model; S200. In the concurrent task scheduling system, based on the calculation instructions, several SQL feature operators to be invoked are determined from a predefined SQL feature operator library; the SQL feature operator library includes standardized calculation units for feature generation and processing. S300. Through the concurrent execution mechanism of the Akka Actor model, multiple Actors are scheduled to execute the computational tasks composed of the SQL feature operators in parallel to obtain the computational results. S400: Aggregate the calculation results of each Actor to generate a feature vector; and synchronously store the feature vector, the real-time feature calculation request and its associated context data for offline replay.

[0022] This invention describes a processing engine capable of handling high-concurrency, low-latency feature computation tasks. The overall architecture and data processing flow of the system are as follows: Figure 1 As shown, the system is deployed on a standard server cluster, runs in a Java environment, and relies on the Akka framework for concurrency support.

[0023] S100: Receive a real-time feature calculation request, convert the real-time feature calculation request into a calculation instruction through an asynchronous filter chain, and submit the calculation instruction to the concurrent task scheduling system; the concurrent task scheduling system is built based on the Akka Actor model, specifically as follows: The system receives real-time feature calculation requests from online services via a network interface. In this embodiment, the real-time feature calculation request is a synchronous HTTP request initiated by the client. The real-time feature calculation request specifies the features to be calculated and the original data source to be accessed. This request is typically issued by the online risk control or recommendation engine when a decision is needed. The real-time feature calculation request includes the user identifier, transaction information, and a list of features to be calculated required for this decision. Subsequently, the real-time feature calculation request is received and sent to an asynchronous processing pipeline for standardization processing. This standardization processing includes a flow control module and an instruction generation module. The pipeline first performs flow control, rejecting requests exceeding the limit. The instruction generation module, based on a pre-set feature-operator mapping configuration table, parses and transforms the abstract feature requirements in the business request into a series of explicit steps composed of basic calculation units that can be directly executed by the system—i.e., calculation instructions.

[0024] S200. In the concurrent task scheduling system, based on the calculation instructions, several SQL feature operators to be invoked are determined from a predefined SQL feature operator library; the SQL feature operator library includes standardized calculation units for feature generation and processing, specifically: The computation instructions generated in step S100 are then submitted to the concurrent task scheduling system. In this embodiment, the concurrent task scheduling system is built based on the Akka Actor model. Figure 2 As shown, in a concurrent task scheduling system, complex computational instructions are further parsed, and their inherent dependencies are automatically decomposed and mapped into a task topology graph consisting of multiple interconnected computational units. This task topology graph is represented by a directed acyclic graph data structure, where nodes represent independent feature computation subtasks, and edges represent data dependencies between feature computation subtasks. Based on this task topology graph, several corresponding SQL feature operators are called from a predefined SQL operator library that can be dynamically expanded via plugins. These SQL feature operators encapsulate various standardized computational functions, ranging from basic operations to complex business logic.

[0025] S300. Through the concurrent execution mechanism of the Akka Actor model, multiple Actors are scheduled to execute the computational task composed of the SQL feature operators in parallel to obtain the computational result, specifically: Using a task topology graph, the Akka Actor system acts as the execution unit, creating multiple independent Actor work units for concurrent processing. The SQL feature operators corresponding to various feature calculation subtasks in the task topology graph are assigned to a large number of independent Actors for concurrent execution. Each Actor receives one or more tasks from the task topology graph. These Actors do not interfere with each other, each having its own independent memory space, and they collaborate by sending messages to each other. Within its private memory space, each Actor executing a task calls the specified SQL operator to complete the calculation and exchanges data with other Actors through message passing.

[0026] In the dynamic expansion mechanism of the SQL operator library, each Actor loads the corresponding SQL operator implementation from the SQL operator library according to its assigned feature calculation subtask, passes in the input data required by the feature calculation subtask as parameters, performs calculations in local memory, and encapsulates the result into a message to send to subsequent Actors that depend on this result. This SQL operator library encapsulates various standardized computation units.

[0027] S400: Aggregate the calculation results of each Actor to generate a feature vector; and synchronously store the feature vector, the real-time feature calculation request, and its associated context data for offline replay, specifically: After all the distributed Actors complete their respective calculations, they aggregate the results. The scheduling system, based on the dependencies defined in the task topology graph, aggregates these intermediate results to generate a final, complete feature vector. This feature vector contains thousands or even tens of thousands of dimensional feature values ​​and is returned to the client that initially initiated the request within milliseconds. In addition to real-time processing, the system also has an offline replay function, synchronously saving the actual real-time feature calculation requests and their complete context data online. Based on this, when it is necessary to verify the effect of new features or reproduce historical problems, it is possible to accurately replay the requests at any historical point in time and simulate the feature results in an independent offline environment, using the exact same computing engine and operator library, the exact same code and configuration as online. This can be used for problem diagnosis, feature effect verification, or model training.

[0028] The offline replay function specifically includes the following steps: Step S4001: During online processing, the system asynchronously persists the real-time feature calculation request, the generated feature vector and its associated context data snapshot to the storage system. The associated context data includes all relevant data during the feature calculation process. Step S4002: When starting offline replay, the system retrieves the corresponding snapshot based on the specified request identifier or time range; Step S4003: Load the scheduling system and operator library consistent with the online version in the resource-isolated simulation environment, and re-execute the calculation process; Step S4004: Record the calculation results obtained during the calculation process, and generate a simulation analysis report based on the calculation results for subsequent comparative analysis.

[0029] Example 2: A real-time feature engineering processing system based on Akka architecture and SQL operators, such as Figure 2 As shown, it includes: Request processing module 100: used to receive real-time feature calculation requests, convert the real-time feature calculation requests into calculation instructions through an asynchronous filter chain, and submit the calculation instructions to the concurrent task scheduling system; the concurrent task scheduling system is built based on the Akka Actor model; Operator library module 200: used in the concurrent task scheduling system to determine several SQL feature operators to be invoked from a predefined SQL feature operator library according to the calculation instructions; the SQL feature operator library includes standardized calculation units for feature generation and processing; Concurrent scheduling and execution module 300: Used to schedule multiple Actors to execute computational tasks composed of SQL feature operators in parallel through the concurrent execution mechanism of the Akka Actor model, and obtain computational results; Replay simulation module 400: used to aggregate the calculation results of each Actor, generate feature vectors; and synchronously store the feature vectors, real-time feature calculation requests and their associated context data for offline replay.

[0030] Various changes and modifications made without departing from the spirit and scope of this invention, and all equivalent technical solutions, also fall within the scope of this invention.

[0031] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0032] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0033] This invention is described with reference to flowchart illustrations and / or block diagrams of the method, terminal device (system), and computer program product according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0034] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0035] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0036] It should be noted that: The phrase "an embodiment" or "an embodiment" used in this specification means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the invention. Therefore, the phrase "an embodiment" or "an embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment.

[0037] Furthermore, it should be noted that the shapes and names of the parts and components described in the specific embodiments described in this specification may differ. All equivalent or simple variations made to the structure, features, and principles described in this patent concept are included within the protection scope of this patent. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, as long as they do not depart from the structure of this invention or exceed the scope defined in these claims, they should all fall within the protection scope of this invention.

Claims

1. A real-time feature engineering processing method based on Akka architecture and SQL operators, characterized in that, Includes the following steps: The system receives real-time feature calculation requests, converts these requests into calculation instructions using an asynchronous filter chain, and submits the calculation instructions to a concurrent task scheduling system. The concurrent task scheduling system is built based on the Akka Actor model. In the concurrent task scheduling system, based on the calculation instructions, several SQL feature operators to be invoked are determined from a predefined SQL feature operator library; the SQL feature operator library includes standardized calculation units for feature generation and processing. The concurrent execution mechanism of the Akka Actor model allows multiple Actors to be scheduled to execute computational tasks composed of the SQL feature operators in parallel, thereby obtaining computational results. The calculation results are aggregated to generate a feature vector; the feature vector, the real-time feature calculation request and its associated context data are synchronously stored for offline replay.

2. The real-time feature engineering processing method based on Akka architecture and SQL operators according to claim 1, characterized in that, The SQL feature operators in the SQL feature operator library are managed and dynamically expanded through a service provider interface. The implementation of newly added SQL feature operators is provided in the form of plug-ins and integrated into the SQL feature operator library without restarting the system.

3. The real-time feature engineering processing method based on Akka architecture and SQL operators according to claim 1 or 2, characterized in that, The SQL feature operators in the SQL feature operator library include various categories, including at least: data preprocessing operators for data preprocessing, basic aggregation function operators for statistical aggregation, and advanced time series aggregation operators for complex time series analysis.

4. The real-time feature engineering processing method based on Akka architecture and SQL operators according to claim 1, characterized in that, The method further includes: Real-time feature calculation requests processed online and their context data are stored synchronously. In an offline environment, the same concurrent task scheduling system and SQL feature operator library as in the online environment are invoked to recalculate and simulate the real-time feature calculation requests stored synchronously.

5. A real-time feature engineering processing system based on Akka architecture and SQL operators, used to implement the method according to any one of claims 1 to 4, characterized in that, include: Request processing module: Used to receive real-time feature calculation requests, convert the real-time feature calculation requests into calculation instructions through an asynchronous filter chain, and submit the calculation instructions to the concurrent task scheduling system; The concurrent task scheduling system is built based on the Akka Actor model; Operator library module: used in the concurrent task scheduling system to determine several SQL feature operators to be invoked from a predefined SQL feature operator library according to the calculation instructions; the SQL feature operator library includes standardized calculation units for feature generation and processing; Concurrent scheduling and execution module: used to schedule multiple Actors to execute computational tasks composed of SQL feature operators in parallel through the concurrent execution mechanism of the Akka Actor model, and obtain computational results; Replay simulation module: used to aggregate the calculation results of each Actor, generate feature vectors; and synchronously store the feature vectors, real-time feature calculation requests and their associated context data for offline replay.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method according to any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method according to any one of claims 1 to 4.

Images