Database-embedded feature extraction workflow execution method and system

Abstract

The application discloses a feature extraction workflow execution method and system embedded in a database, and the method comprises the following steps: a feature extraction workflow is defined in an extended SQL syntax; the feature extraction workflow is parsed, structured executable units are extracted as nodes, a topological relationship between the nodes is constructed, and a directed acyclic graph is generated; wherein the executable units comprise data source nodes, calculation / conversion nodes and storage nodes; the nodes in the directed acyclic graph are mapped into feature extraction tasks, an execution plan is generated; according to the execution plan, a task scheduling queue is determined, and corresponding feature extraction tasks are executed according to the task scheduling queue; features extracted by executing the feature extraction tasks are generated into feature vectors, an association between the feature vectors and original data is established, a feature table with vector indexes is formed and stored.

Description

Technical Field

[0001] This invention relates to the field of big data technology, specifically to a method and system for executing a feature extraction workflow embedded in a database. Background Technology

[0002] With the rapid development of artificial intelligence and big data technologies, feature extraction from multimodal data (text, images, audio, video, etc.) has become a crucial step in intelligent applications. Current data feature extraction is generally performed externally to the data layer (database), such as using ETL tools (e.g., Apache Airflow, Apache NiFi) to build feature extraction pipelines at the application layer, or using dedicated AI platforms (e.g., TensorFlow Serving, TorchServe) to provide feature extraction at the service layer. However, existing solutions suffer from the following problems: complex system architecture: feature extraction typically requires deploying a separate ETL pipeline or AI service outside the database, leading to a complex system architecture and high maintenance costs; inefficient data flow: raw data needs to be exported from the database to an external system, and then imported back into the database after feature extraction, generating significant data movement overhead and network transmission latency; difficulty in guaranteeing consistency: the feature extraction process is separated from database transactions, which can easily lead to data inconsistency, especially in concurrent environments, where data inconsistency errors are particularly likely to occur. Summary of the Invention

[0003] To address the shortcomings of existing systems, this invention provides a database-embedded feature extraction workflow execution method and system that simplifies system architecture, improves data flow efficiency and consistency.

[0004] To solve the above problems, the embodiments of the present invention are implemented through the following technical solutions: On the one hand, a database-embedded feature extraction workflow execution method is provided, which runs in the database kernel execution engine, including: Declaratively define the feature extraction workflow using extended SQL syntax; The feature extraction workflow is parsed to extract structured executable units as nodes, and the topological relationships between nodes are constructed to generate a directed acyclic graph; wherein, the executable units include data source nodes, computation / transformation nodes, and storage nodes; The nodes in the directed acyclic graph are mapped to feature extraction tasks, and an execution plan is generated. According to the execution plan, a task scheduling queue is determined, and the corresponding feature extraction tasks are executed according to the task scheduling queue; wherein, the task scheduling queue includes the execution order, the resources required for execution, and the matching feature extraction model for each feature extraction task; The features extracted by each of the aforementioned feature extraction tasks are used to generate feature vectors. The association between the feature vectors and the original data is established to form a feature table with vector indexes and then stored.

[0005] On the other hand, a database-embedded feature extraction workflow execution system is provided, the system being embedded in the database kernel execution engine, comprising: The interface layer is used to declaratively define the feature extraction workflow using extended SQL syntax; The workflow management layer is used to parse the feature extraction workflow, extract structured executable units as nodes, construct the topological relationships between nodes, and generate a directed acyclic graph; wherein, the executable units include data source nodes, computation / transformation nodes, and storage nodes; the nodes in the directed acyclic graph are mapped to feature extraction tasks, and an execution plan is generated; The workflow execution layer is used to determine the task scheduling queue according to the execution plan, and execute the corresponding feature extraction tasks according to the task scheduling queue; wherein, the task scheduling queue includes the execution order, the resources required for execution, and the matching feature extraction model for each feature extraction task; The storage layer is used to generate feature vectors from the features extracted by each feature extraction task, establish the association between the feature vectors and the original data, form a feature table with vector indexes, and store them.

[0006] The database-embedded feature extraction workflow execution method provided in the above embodiments deeply integrates the feature extraction workflow into the database kernel execution engine, eliminating the need to maintain a complex external pipeline system, greatly simplifying the system architecture and reducing operational complexity. The definition, parsing, execution plan generation, and execution of the feature extraction workflow fully utilize the database's transaction, storage, computation, and scheduling capabilities. Thus, the feature extraction workflow shares these capabilities with the database, and each feature extraction task is executed directly on the database's data storage nodes. This avoids the overhead of migrating large amounts of raw data between the database and external systems, reduces external system dependencies, effectively reduces latency, and significantly improves performance. Furthermore, the feature extraction workflow execution is tightly integrated with the database transaction mechanism, ensuring data atomicity, consistency, isolation, and durability during the feature extraction process, thus resolving data inconsistency issues that may arise from external processing.

[0007] In the above embodiments, the database-embedded feature extraction workflow execution system and the corresponding database-embedded feature extraction workflow execution method embodiments belong to the same concept, and thus have the same technical effect as each database-embedded feature extraction workflow execution method embodiment, which will not be repeated here. Attached Figure Description

[0008] Figure 1 This is a schematic diagram of the architecture of a database-embedded feature extraction workflow execution system provided in one embodiment.

[0009] Figure 2 This is a flowchart of a database-embedded feature extraction workflow execution method provided in one embodiment. Detailed Implementation

[0010] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, the phrase "some embodiments" refers to a subset of all possible embodiments. It should be noted that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0013] In the following description, the terms "first, second, and third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0014] See Figure 1 The present invention provides a database-embedded feature extraction workflow execution system. The running program of the database-embedded feature extraction workflow execution system is embedded in the database kernel execution engine and includes: an interface layer 21, a workflow management layer 22, a workflow execution layer 23, a model management layer 24, a storage layer 25, and a monitoring layer 26.

[0015] (1) Interface layer Interface layer 21 serves as the entry point for users to interact with the database-embedded feature extraction workflow execution system. By extending the SQL workflow definition language, it acts as the entry point for receiving declarative definitions of feature extraction workflows, serving as the system's input trigger point. The extended SQL syntax is based on standard SQL syntax, and the interface layer supports SQL workflow definition languages ​​including, but not limited to: Create Feature Workflow, Alterate Feature Workflow, Drop Feature Workflow, Execute Workflow, and Monitor Workflow. By supporting extended standard SQL declarative workflow definitions through the interface layer, the barrier to entry for using database-embedded feature extraction workflow execution methods is lowered. A unified management interface is provided, enabling full lifecycle management of workflow definition, scheduling, execution, monitoring, and debugging. This avoids the problem of workflow definition, scheduling, monitoring, and fault tolerance mechanisms being scattered across different systems, lacking a unified management interface and tools, thus enhancing manageability.

[0016] As a preferred embodiment of the present invention, the extended SQL syntax primarily uses the `CREATE FEATUREWORKFLOW` method to declaratively define the feature extraction workflow to be executed. During definition, at least the data source of the feature extraction workflow, the feature processing logic for the corresponding data source, and the storage location of the feature calculation results must be defined. Here, the data source refers to the storage location in the database of the input data for each feature extraction task in the feature extraction workflow; the feature processing logic refers to the processing of the corresponding input data by each feature extraction task, such as text feature extraction, image feature extraction, normalization, aggregation, filtering, and other feature calculation / transformation operations; and the feature calculation results refer to the path where the results of the feature processing logic are written to storage.

[0017] (2) Workflow Management Layer The workflow management layer 22 is used to parse the feature extraction workflow received from the interface layer, verify and optimize the parsing results, and generate an executable execution plan. Optionally, the workflow management layer includes a parser, a semantic validator, an optimizer, and a plan generator.

[0018] The parser is used to parse the feature extraction workflow. In an optional specific example, when parsing the feature extraction workflow defined by extended SQL syntax, an extended SQL syntax parser needs to be defined first. This extended SQL syntax parser is implemented by extending the standard SQL parser and corresponds to the extended SQL syntax, mainly used to parse the extended SQL syntax. In this embodiment, the extended SQL syntax parser mainly parses the CREATE FEATURE WORKFLOW statement and its options in the interface layer by extending the standard SQL parser.

[0019] A semantic validator is used to perform syntactic and logical validity checks on the parsed results. For example, when the extended SQL syntax parser parses the feature extraction workflow, it uses the semantic validator to perform lexical analysis, syntax analysis, and semantic analysis. Lexical analysis refers to identifying the keywords and identifiers of the feature extraction workflow defined by the extended SQL syntax; syntax analysis refers to constructing an abstract syntax tree of the feature extraction workflow based on the keywords and identifiers identified by lexical analysis; semantic analysis refers to performing feature calculation / transformation operation type detection and dependency analysis on the abstract syntax tree constructed by syntax analysis.

[0020] The optimizer is used to optimize the performance and resources of the validated workflow. For example, it can optimize the dependency analysis of the feature processing logic obtained from semantic analysis, optimize the data flow path according to the data feature processing logic, analyze the data flow direction, and then determine the scheduling strategy, resource strategy, etc.

[0021] The plan generator transforms the optimized workflow into an executable plan. By combining the processing results of the parser, semantic validator, and optimizer, it extracts structured executable units corresponding to the feature extraction workflow as nodes, constructs the topological relationships between nodes, and generates a directed acyclic graph. These structured executable units, also called operators, are independent functional units with clearly defined inputs and outputs, abstracted from each step of data processing. Typical executable units include source operators (e.g., database tables, log files, CSV files, etc.) representing the data source; transformation operators (e.g., performing feature extraction, feature calculation, normalization, aggregation, filtering, etc.); and sink operators (the storage location for the data results from the transformation operators). The topological relationships between nodes are mainly analyzed by examining the data processing flow expressed by the extracted nodes, mapping each transformation node to a feature extraction task, and determining the dependencies, data lineage, and execution parameters between the transformation nodes. Among these, dependencies mainly involve the sequential, parallel, and independent execution order of computation / transformation nodes; data lineage mainly involves the relationships between input data and feature calculation results of different computation / transformation nodes, as well as their mapping relationships with the original data; execution parameters mainly involve the resources required for execution of the corresponding computation / transformation nodes, timeout strategies, and fault tolerance strategies. A directed acyclic graph (DAG) is constructed using executable units as nodes, dependencies as edges, and data lineage and execution parameters as attribute data for nodes and edges.

[0022] In an optional specific example, the parser traverses the feature extraction workflow, extracts each independent computation / transformation action or step, instantiates each computation / transformation action or step into a node, and the storage location of the data and processing results involved in each independent computation / transformation action or step is used as its associated data source node and storage node. After verification and optimization analysis, the data flow direction (dependency) between computation / transformation actions or steps is further analyzed, the required resources (CPU, memory, GPU) for each node are estimated, and the parallelism of the nodes (scheduling strategy) is determined, thereby establishing a directed acyclic graph (DAG). In the DAG, nodes represent data sources, transformations / computations, and storage, and edges represent dependencies. The attribute data of the edges includes recording data lineage (the source of the input data corresponding to the transformation / computation operation and the mapping relationship with the original data), execution parameters (resource configuration, timeout strategy, fault tolerance strategy), etc.

[0023] Once the directed acyclic graph representation of the feature extraction workflow is generated, existing algorithms or plan generators can be used to generate the execution plan for the feature extraction workflow. In this embodiment, the plan generator maps nodes in the directed acyclic graph of the feature extraction workflow to feature extraction tasks. Specifically, it maps the node corresponding to each computation / transformation action or step to a feature extraction task, determines the input and output of the computation / transformation action or step, analyzes the data flow direction and dependencies, thereby determining the execution order of each feature extraction task. Based on the dependencies and parallelism of the nodes, combined with the location information of the data source storage, it plans the data reading path and estimates the required resources for the feature extraction tasks corresponding to each node, thus transforming the feature extraction tasks into an execution plan that the system can understand. The scheduling strategy mainly includes three execution modes: real-time scheduling, batch scheduling, and incremental scheduling. Real-time scheduling refers to event-driven real-time processing with low latency. Batch scheduling refers to time-driven (scheduled) or quantity-driven processing, usually handling the entire dataset, and is more suitable for periods of idle resource usage, but with higher latency. Incremental scheduling refers to processing datasets that have changed, such as newly added or changed data after the previous feature extraction task, based on data changes or timestamp watermarks. It has smooth and low resource consumption and low latency.

[0024] Optionally, the workflow management layer also includes a monitor that, based on the configured fault tolerance management strategy, manages the execution status of each feature extraction task and handles fault recovery according to the working parameters and threshold information of each feature extraction task.

[0025] (3) Workflow Execution Layer The workflow execution layer 23 is used to complete task scheduling and execution within the database process space. Specifically, the workflow execution layer includes a scheduler, an executor, a resource manager, and a state manager.

[0026] The scheduler, based on the scheduling strategy in the execution plan, schedules the corresponding feature extraction tasks to ensure the orderly execution of each feature extraction task in the feature extraction workflow. For the currently executing feature extraction task, data is read from the corresponding input data source, the corresponding feature extraction model in the model repository is called to process (calculate / transform) the data, and the feature calculation results are written to the storage location corresponding to the feature extraction task. It can be understood that, based on the dependencies between feature extraction tasks, the storage location where the feature calculation results of the previous feature extraction task are stored may serve as the storage location (intermediate data source) for the input data of the next feature extraction task. The scheduler, by integrating the information on resource pre-allocation, scheduling strategy, and fault tolerance strategy corresponding to each feature extraction task in the execution plan, optimizes the execution order of each feature extraction task and adds the plan generated by the plan generator to the task scheduling queue, thus achieving orderly task execution.

[0027] Optionally, the scheduler can schedule not only the feature extraction task queue of the feature extraction workflow, but also the feature extraction workflow queue itself. When there is only one feature extraction workflow to be executed in the database, the scheduling strategy only needs to schedule each feature extraction task of that workflow. When there are two or more feature extraction workflows to be executed in the database, the scheduling strategy first needs to schedule each feature extraction workflow queue, and then schedule each feature extraction task of each workflow. The scheduler performs two-level scheduling of the feature extraction task queue and the feature extraction task queue of the feature extraction workflow, which can more flexibly optimize the feature extraction process of the data in the database by comprehensively considering factors such as resources and real-time performance. The scheduler's inputs include the feature extraction workflow and the priority and deadline of the feature extraction tasks, task resource requirements and dependencies, system resource status and load, as well as historical execution statistics and predictions. The scheduler's outputs include the mapping of tasks to resources, execution order and time scheduling, as well as resource reservation and allocation plans. The scheduler's decision algorithms include multi-objective optimization (minimizing completion time, maximizing resource utilization and ensuring fairness), online learning (adjusting scheduling parameters based on historical data), and predictive scheduling (optimizing scheduling based on task features to predict execution time).

[0028] In an optional example, the feature extraction task is scheduled primarily using incremental scheduling, with the following process: First, data changes are detected based on change data capture, trigger mechanisms, and log parsing. Then, incremental tasks are triggered based on the data change detection results. Triggering conditions include data change events, time-based scheduling (periodic), manual triggering, and event accumulation triggering (batch processing). Triggering actions include creating an incremental execution context, extracting incremental change data (incremental data), scheduling the incremental workflow execution, and handling dependencies and conflicts. Next, incremental tasks are executed, including incremental data extraction, incremental feature extraction, feature merging and deduplication, and incremental index updates. Finally, state synchronization and cleanup are performed. Through the database's built-in incremental scheduling algorithm, based on change data capture, near real-time feature extraction of newly added or changed data can be achieved, meeting the low-latency requirements of online inference and real-time recommendation scenarios.

[0029] The executor is responsible for executing feature extraction tasks. Execution follows the scheduling strategy of each feature extraction task in the execution plan, taking into account resource requirements, and may involve serial or parallel execution. The feature extraction tasks executed by the executor can include stages such as data preparation, model loading, inference execution, post-processing, and result output. Data preparation refers to reading input data from the data source associated with the feature extraction task in the database; model loading refers to calling the feature extraction model required by the feature extraction task from the model repository; inference execution refers to using the called feature extraction model to perform feature extraction calculations / transformations on the input data to obtain feature calculation results; post-processing refers to normalizing and reducing the dimensionality of the feature calculation results to generate feature vectors (feature data); result output refers to writing feature vectors back to the database in batches, including storing feature vectors in a feature table, establishing the association between feature vectors and the original data, updating feature indexes (such as vector indexes), and recording feature extraction metadata to maintain the association with the original data.

[0030] The resource manager allocates resources to the current feature extraction task based on the resource policies (performance and resource information) corresponding to each feature extraction task in the execution plan. In this embodiment, the resource manager allocates resources according to the feature extraction task, dynamically selecting whether to execute the feature extraction task in a CPU or GPU environment, and allocating the corresponding system computing resources (CPU, GPU, and / or memory). The resource manager uniformly allocates and utilizes the CPU, GPU, and memory resources of the database cluster, which can improve the utilization rate of computing resources by 30-50% and effectively overcome the problem of data inconsistency caused by the separation of the feature extraction process from database transactions.

[0031] The state manager manages the current feature extraction task's working status and handles fault recovery based on the execution status information (errors and quality information) of each feature extraction task in the execution plan. In this embodiment, the state manager manages the current feature extraction task's working status and handles fault recovery according to the characteristics of the feature extraction task, managing the execution of the feature extraction task and handling fault recovery, including state persistence, monitoring points, recovery mechanisms, and consistency management. The state manager has a complete state management and fault recovery mechanism, which can automatically handle task failures, node anomalies, etc., ensuring the reliable execution of long-cycle workflows and achieving high reliability and fault tolerance.

[0032] (4) Model Management Layer The model management layer 24 can pre-build a model repository for storing feature extraction models. This repository supports version management, hot updates, and deployment management of feature extraction models. Version management is used to control versioning, rollback, compare versions, and manage the lifecycle of feature extraction models. Hot update management is used to manage online updates, traffic switching, zero downtime, and rollback mechanisms for feature extraction models. Deployment management is used to manage loading, resource allocation, performance debugging, and health monitoring of feature extraction models. In this embodiment, the model management layer adopts a plug-in design, which can quickly integrate new feature extraction models and achieve iteration and system expansion of feature extraction models through version management and hot updates. It should be noted that the feature extraction model can refer to a deep learning model trained on a known neural network model, as well as various known traditional algorithm models.

[0033] (5) Storage layer Storage layer 25 refers to the storage location of the raw data and feature vectors for each feature extraction task. Furthermore, the storage layer can also store various intermediate data during the execution of each feature extraction task, such as performance information, resource information, error information, and quality information. Performance information may include throughput, latency, success rate, and SLA (Service Level Agreement); resource information may include CPU utilization, GPU utilization, memory usage, and network I / O; error information may include anomaly detection, error classification, root cause analysis, and automatic repair; and quality information may include feature quality, accuracy metrics, drift detection, and anomaly detection.

[0034] (6) Monitoring layer The monitoring layer 26 is used to monitor the performance, resources, errors, or quality of each feature extraction task.

[0035] In the above description, there is no strict order between the layers involved in the feature extraction workflow execution system embedded in the database. The identifiers “(1)” to “(6)” are only to distinguish similar objects and do not represent a specific order for the objects. It can be understood that “(1)” to “(6)” can be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0036] Please see Figure 2 This invention provides a database-embedded feature extraction workflow execution method, which runs in the database kernel execution engine and includes the following steps: S11 uses extended SQL syntax to declaratively define the feature extraction workflow.

[0037] Extended SQL syntax refers to the declarative definition of feature extraction workflows based on standard SQL syntax. This can include creating, modifying, deleting, executing, and monitoring workflows. In this embodiment, extended SQL syntax primarily uses the `CREATE FEATUREWORKFLOW` function to declaratively define the required feature extraction workflow, which at least defines the data source, feature processing logic, and storage location of the feature processing results.

[0038] S12, the feature extraction workflow is parsed, structured executable units are extracted as nodes, the topological relationships between nodes are constructed, and a directed acyclic graph is generated; wherein, the executable units include data source nodes, computation / transformation nodes and storage nodes.

[0039] An extended SQL syntax parser is defined, which is implemented by extending the standard SQL parser. Corresponding to extended SQL syntax, it is primarily used to parse extended SQL syntax. In this embodiment, the extended SQL syntax parser mainly parses the CREATE FEATUREWORKFLOW statement and its options in the interface layer by extending the standard SQL parser.

[0040] The extended SQL syntax parser needs to perform lexical analysis, syntax analysis, semantic analysis, and optimization analysis when parsing the feature extraction workflow.

[0041] A Directed Acyclic Graph (DAG) consists of nodes, edges, and the attributes corresponding to the nodes and edges. Feature extraction task partitioning maps nodes in the DAG of the feature extraction workflow to feature extraction tasks. For example, it maps the node corresponding to each calculation / transformation action or step to a feature extraction task, and determines the data source and storage location of the feature results corresponding to that feature extraction task based on the data source nodes and storage nodes associated with the nodes corresponding to the calculation / transformation action or step in the DAG.

[0042] The fault-tolerant management strategy, based on the execution data corresponding to the nodes and edges in the directed acyclic graph, obtains the working parameters and threshold information of each feature extraction task to achieve working status management and fault recovery of the feature extraction task.

[0043] S13, map the nodes in the directed acyclic graph to feature extraction tasks and generate an execution plan.

[0044] This execution process mainly includes the scheduling, execution, resource management, and state management stages for each feature extraction task. It is primarily used to call the feature extraction model to execute feature extraction tasks within the process space of the database.

[0045] The scheduling phase mainly includes real-time scheduling, batch scheduling, and incremental scheduling to ensure the orderly execution of each feature extraction task. The scheduling phase can schedule not only the feature extraction task queue of the feature extraction workflow but also the feature extraction workflow queue itself. When there is only one feature extraction workflow to be executed in the database, the scheduling strategy only needs to schedule each feature extraction task within that workflow. When there are two or more feature extraction workflows to be executed, the scheduling strategy can first determine the scheduling of each feature extraction workflow queue, and then determine the scheduling of each feature extraction task within each workflow. In this embodiment, the scheduling phase performs two-level scheduling of both the feature extraction task queue and the feature extraction task queue of the feature extraction task workflow. The inputs to scheduling include the priority and deadline of the feature extraction workflow and tasks, task resource requirements and dependencies, system resource status and load, and historical execution statistics and predictions. The outputs of scheduling include the mapping of tasks to resources, execution order and time schedule, resource reservation and allocation plans, etc. The decision algorithms for scheduling include multi-objective optimization (minimizing completion time, maximizing resource utilization and ensuring fairness), online learning (adjusting scheduling parameters based on historical data), and predictive scheduling (optimizing scheduling based on task features to predict execution time).

[0046] During the execution phase, the system is configured to retrieve the raw data to be extracted from the database storage, call the feature extraction model to perform feature extraction processing, and write the feature vectors back to the database in batches. In this embodiment, the execution of the feature extraction task includes stages such as data preparation, model loading, inference execution, post-processing, and result output. Data preparation refers to reading the raw data from the data source associated with the feature extraction task in the database; model loading refers to calling the feature extraction model required by the feature extraction task from the model repository. Inference execution refers to using the called feature extraction model to perform feature extraction calculations on the raw data to obtain the feature extraction results. Post-processing refers to normalizing and reducing the dimensionality of the feature extraction results to generate feature vectors (feature data). Result output refers to writing the feature result data back to the database in batches, including storing the feature data in a feature table, establishing the association between the feature data and the raw data, updating the feature index (such as a vector index), and recording the feature extraction metadata process to maintain the association with the raw data.

[0047] The resource management phase refers to allocating resources to the current feature extraction task based on the resource management strategy in the execution plan and the work status information (performance and resource information) obtained from monitoring. In this embodiment, the resource management phase allocates resources to the current feature extraction task according to its characteristics, dynamically selects whether to execute the feature extraction task in a CPU or GPU environment, and allocates the corresponding system computing resources (CPU, GPU, and / or memory).

[0048] The state management phase refers to the management of the current feature extraction task's working state and fault recovery based on the fault tolerance management strategy in the execution plan and the working state information (error and quality information) obtained from monitoring. In this embodiment, the state management phase manages the working state and handles fault recovery for the current feature extraction task according to its characteristics, thereby managing the feature extraction workflow state and handling fault recovery. This includes state persistence, monitoring points, recovery mechanisms, and consistency management. The state management phase has a comprehensive state management and fault recovery mechanism, which can automatically handle situations such as feature extraction task failure and node anomalies, ensuring the reliable execution of long-cycle workflows and achieving high reliability and fault tolerance.

[0049] S14. Based on the execution plan, determine the task scheduling queue and execute the corresponding feature extraction tasks according to the task scheduling queue; wherein, the task scheduling queue includes the execution order, required resources, and matched feature extraction models for each feature extraction task.

[0050] The feature extraction model can be a pre-built model repository for storing feature extraction models and supporting version management, hot updates, and deployment management. In this embodiment, the model repository supports the storage, format conversion, metadata, and access control of feature extraction models. Version management is used to manage version control, version rollback, version comparison, and lifecycle of feature extraction models; hot update management is used to manage online updates, traffic switching, zero downtime, and rollback mechanisms for feature extraction models; deployment management is used to manage the loading, resource allocation, performance debugging, and health monitoring of feature extraction models. The model management layer can adopt a plug-in design to quickly integrate new feature extraction models and achieve iterative development and system expansion of feature extraction models through version management and hot updates.

[0051] S15, generate feature vectors from the features extracted by each of the feature extraction tasks, establish the association between the feature vectors and the original data, form a feature table with vector indexes and store it.

[0052] The feature table is stored in a specified storage location within the database. Stored data includes raw data and feature result data, as well as workflow metadata, model, feature, and status information. Workflow metadata includes definition storage, version history, and dependencies; model information includes model files, configuration storage, weight storage, and cache storage; feature information includes vector storage, graph storage, time-series storage, and relational storage; and status information includes execution status, checkpoints, log storage, and audit logs.

[0053] In some embodiments, step S13 includes: The nodes in the directed acyclic graph are mapped to feature extraction tasks. Resources are pre-allocated for the feature extraction tasks, and corresponding scheduling and fault tolerance strategies are determined to generate an execution plan. The pre-allocation of resources includes at least one of the following: CPU, GPU, and memory. The scheduling strategies include at least one of the following: real-time scheduling, batch scheduling, and incremental scheduling. The fault tolerance strategies include at least one of the following: retry, checkpointing, and replication.

[0054] Based on the mapping of nodes in a Directed Acyclic Graph (DAG) to executable feature extraction tasks, a highly efficient and reliable execution plan is generated through systematic resource management, scheduling, and fault tolerance mechanisms. Retry refers to automatically retrying the corresponding operation instead of immediately reporting the error when it fails. Checkpoints refer to periodically saving state snapshots for state rollback after failures. Replication refers to deploying the same data on multiple independent replicas, with replicas maintaining state synchronization through a consensus protocol. The entire framework adopts a modular design, supporting dynamic configuration and adaptive optimization.

[0055] In some embodiments, step S14 includes: Based on the information of resource pre-allocation, scheduling strategy and fault tolerance strategy corresponding to each feature extraction task in the execution plan, the execution order, execution resources required and matching feature extraction model of each feature extraction task are determined and submitted to the execution queue to form a task scheduling queue. According to the task scheduling queue, the feature extraction tasks are executed sequentially. The input data of the current feature extraction task is obtained from the database. Based on the information of the resource pre-allocation, scheduling strategy and fault tolerance strategy of the current feature extraction task, a matching execution environment and a matching feature extraction model are selected for feature extraction, and the feature calculation results are stored in the corresponding storage location.

[0056] In the above embodiments, a highly reliable, high-performance, and scalable feature extraction task execution platform is constructed by designing a complete technical implementation path from task scheduling queue generation to feature extraction task execution, and through refined resource management, intelligent scheduling decisions, multi-level fault tolerance mechanisms, and efficient execution optimization.

[0057] In some embodiments, when the scheduling strategy is incremental scheduling, the execution of the current feature extraction task includes: capturing changed data, obtaining the newly added or changed data after the execution of the previous feature extraction task as input data, performing feature extraction according to the selected matching execution environment and the matching feature extraction model, and storing the feature calculation results in the corresponding storage location.

[0058] The execution of the current feature extraction task also includes: selecting a matching feature extraction model from a pre-built model repository.

[0059] The incremental scheduling strategy for feature extraction tasks can be driven by data changes or timestamp watermarks, and executed based on the new or changed data after the previous feature extraction task. Its resource consumption is smooth and low, which can reduce latency.

[0060] In some embodiments, the feature extraction workflow execution method embedded in the database, during the process of executing the corresponding feature extraction task according to the task scheduling queue, further includes: The execution status of each feature extraction task is monitored; the execution status includes at least one of the following: performance monitoring, resource monitoring, error monitoring, and quality monitoring.

[0061] The execution status of a feature extraction task can include performance, resource, error, and quality information. Performance information further includes throughput, latency, success rate, and SLA (Service Level Agreement); resource information further includes CPU utilization, GPU utilization, memory usage, and network I / O; error information further includes anomaly detection, error classification, root cause analysis, and automatic repair; and quality information further includes feature quality, accuracy metrics, drift detection, and anomaly detection.

[0062] In some embodiments, generating feature vectors from the features extracted by each of the feature extraction tasks, establishing the association between the feature vectors and the original data, forming a feature table with vector indexes, and storing the data includes: The features extracted by performing each of the aforementioned feature extraction tasks are normalized and dimensionality reduced to generate feature vectors; The feature vectors are written back to the database in batches, stored in the feature table, the association between the feature vectors and the original data is established, the vector index is updated, and the feature extraction process information is recorded.

[0063] In the above description of the embodiments, there is no strict order between the steps involved in the method. Where permissible, a specific order or sequence can be interchanged so that the embodiments of this application described herein can be implemented in a sequence other than that illustrated or described herein.

[0064] In summary, this invention declaratively defines the feature extraction workflow by extending SQL syntax; it parses the workflow within the database to generate an execution plan based on a directed acyclic graph; it schedules task execution and manages resources based on the execution plan; it calls the feature extraction model to execute the feature extraction task and stores the results back in the database; and it provides end-to-end monitoring, management, and fault tolerance mechanisms. This invention deeply integrates the feature extraction workflow into the database, simplifying the system architecture, fully utilizing existing database resources, eliminating unnecessary data movement, ensuring data consistency, and significantly improving the efficiency, real-time performance, and manageability of feature extraction.

[0065] It should be noted that, in this document, the term "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0066] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A database-embedded feature extraction workflow execution method, running in the database kernel execution engine, characterized in that, include: Declaratively define the feature extraction workflow using extended SQL syntax; The feature extraction workflow is parsed to extract structured executable units as nodes, and the topological relationships between nodes are constructed to generate a directed acyclic graph; wherein, the executable units include data source nodes, computation / transformation nodes, and storage nodes; The nodes in the directed acyclic graph are mapped to feature extraction tasks, and an execution plan is generated. According to the execution plan, a task scheduling queue is determined, and the corresponding feature extraction tasks are executed according to the task scheduling queue; wherein, the task scheduling queue includes the execution order, the resources required for execution, and the matching feature extraction model for each feature extraction task; The features extracted by each of the aforementioned feature extraction tasks are used to generate feature vectors. The association between the feature vectors and the original data is established to form a feature table with vector indexes, which is then stored.

2. The database-embedded feature extraction workflow execution method according to claim 1, characterized in that, The step of mapping nodes in the directed acyclic graph to feature extraction tasks and generating an execution plan includes: The nodes in the directed acyclic graph are mapped to feature extraction tasks. Resources are pre-allocated for the feature extraction tasks, and corresponding scheduling and fault tolerance strategies are determined to generate an execution plan. The pre-allocation of resources includes at least one of the following: CPU, GPU, and memory. The scheduling strategies include at least one of the following: real-time scheduling, batch scheduling, and incremental scheduling. The fault tolerance strategies include at least one of the following: retry, checkpointing, and replication.

3. The database-embedded feature extraction workflow execution method according to claim 1, characterized in that, The step of determining a task scheduling queue according to the execution plan and executing the corresponding feature extraction task according to the task scheduling queue includes: Based on the information of resource pre-allocation, scheduling strategy and fault tolerance strategy corresponding to each feature extraction task in the execution plan, the execution order, execution resources required and matching feature extraction model of each feature extraction task are determined and submitted to the execution queue to form a task scheduling queue. According to the task scheduling queue, the feature extraction tasks are executed sequentially. The input data of the current feature extraction task is obtained from the database. Based on the information of the resource pre-allocation, scheduling strategy and fault tolerance strategy of the current feature extraction task, a matching execution environment and a matching feature extraction model are selected for feature extraction, and the feature calculation results are stored in the corresponding storage location.

4. The database-embedded feature extraction workflow execution method according to claim 3, characterized in that, When the scheduling strategy is incremental scheduling, the execution of the current feature extraction task includes: capturing changed data, obtaining the newly added or changed data after the execution of the previous feature extraction task as input data, performing feature extraction according to the selected matching execution environment and the matching feature extraction model, and storing the feature calculation results in the corresponding storage location.

5. The database-embedded feature extraction workflow execution method according to claim 3, characterized in that, The execution of the current feature extraction task also includes: Select a matching feature extraction model from a pre-built model repository.

6. The database-embedded feature extraction workflow execution method according to claim 1, characterized in that, The process of executing the corresponding feature extraction task according to the task scheduling queue also includes: The execution status of each feature extraction task is monitored; the execution status includes at least one of the following: performance monitoring, resource monitoring, error monitoring, and quality monitoring.

7. The database-embedded feature extraction workflow execution method according to claim 1, characterized in that, The process of generating feature vectors from the features extracted by each of the feature extraction tasks, establishing the association between the feature vectors and the original data, forming a feature table with vector indexes, and storing the data includes: The features extracted by performing each of the aforementioned feature extraction tasks are normalized and dimensionality reduced to generate feature vectors; The feature vectors are written back to the database in batches, stored in the feature table, the association between the feature vectors and the original data is established, the vector index is updated, and the feature extraction process information is recorded.

8. A feature extraction workflow execution system embedded in a database, characterized in that, include: The interface layer is used to declaratively define the feature extraction workflow using extended SQL syntax; The workflow management layer is used to parse the feature extraction workflow, extract structured executable units as nodes, construct the topological relationships between nodes, and generate a directed acyclic graph; wherein, the executable units include data source nodes, computation / transformation nodes, and storage nodes; the nodes in the directed acyclic graph are mapped to feature extraction tasks, and an execution plan is generated; The workflow execution layer is used to determine the task scheduling queue according to the execution plan, and execute the corresponding feature extraction tasks according to the task scheduling queue; wherein, the task scheduling queue includes the execution order, the resources required for execution, and the matching feature extraction model for each feature extraction task; The storage layer is used to generate feature vectors from the features extracted by each feature extraction task, establish the association between the feature vectors and the original data, form a feature table with vector indexes, and store them.

9. The feature extraction workflow execution system embedded in the database according to claim 8, characterized in that, It also includes a model management layer for storing feature extraction models and for managing the version, hot updates, and deployment of the feature extraction models.

10. The feature extraction workflow execution system embedded in the database according to claim 8, characterized in that, It also includes a monitoring layer for monitoring the execution status of each feature extraction task; wherein the execution status includes at least one of the following: performance monitoring, resource monitoring, error monitoring, and quality monitoring.

Images