Data bloodline tracing and distributed scheduling method based on database hybrid analysis

Abstract

The application discloses a data bloodline tracing and distributed scheduling method based on database hybrid analysis, comprising the following steps: constructing a multi-dimensional data bloodline graph model based on a directed acyclic graph, performing explicit and implicit fusion collection, generating a field-level bloodline link and storing the field-level bloodline link into a graph database, and providing a traceability, influence analysis and compliance audit interface; constructing a multi-level scheduling strategy containing time sequence prediction, and performing adaptive execution and intelligent collaborative scheduling based on real-time and predicted load; calculating a comprehensive load evaluation value of a real-time calculation node, and performing low-loss sharding migration when an expected imbalance occurs; feeding back a high-frequency bloodline link to a scheduling engine, and preferentially scheduling a task to a low-load node in combination with the comprehensive load evaluation value of each node. The application has the initiative defense and self-healing ability by realizing field-level fine tracing, combining the predicted scheduling and precision fusing mechanism, and effectively improving the cluster resource balance degree and core link business continuity.

Description

Technical Field

[0001] This invention relates to the field of big data processing and database management technology, specifically a data lineage tracing and distributed scheduling method based on database hybrid analysis. Background Technology

[0002] As data volumes surge from TB to PB levels, and concurrent requests grow exponentially, enterprise data processing scenarios are gradually shifting towards a HTAP architecture that integrates OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing). However, existing HTAP databases and big data processing systems still face the following technical bottlenecks and limitations: Data lineage tracing suffers from blind spots and coarse granularity: In multi-source heterogeneous data environments, after data undergoes complex processes such as ETL, stream computing, and UDF functions, traditional solutions can only achieve coarse-grained tracing at the table level, failing to effectively resolve implicit lineage relationships (such as field calculation logic within Python / PySpark scripts). This results in extremely high troubleshooting costs when data anomalies or compliance audit requirements arise, and makes it difficult to meet the requirements of regulations such as GDPR for full-chain traceability of personal data.

[0003] Distributed resource scheduling suffers from lag and has weak ability to cope with fluctuations: In HTAP scenarios, OLTP requests and OLAP analysis loads have different competition characteristics for resources such as CPU, memory, and IOPS. Existing static rule scheduling is difficult to adapt to heterogeneous nodes, while dynamic scheduling based on real-time load often has a response lag of 1 to 3 minutes. It cannot effectively prevent the arrival of periodic traffic peaks such as e-commerce promotions and financial settlements, which can easily lead to overload of hot nodes, resulting in business interruption or service quality degradation.

[0004] Data tracing and resource scheduling are disconnected: data lineage tracing is mainly used for offline problem localization, while resource scheduling focuses on real-time load. The two operate independently, lacking an effective information fusion and collaboration mechanism. This prevents the system from using the high-frequency data access links identified by lineage analysis to proactively guide the scheduling engine to pre-allocate resources, thus limiting the overall operational efficiency of the HTAP system. Summary of the Invention

[0005] This invention provides a data lineage tracing and distributed scheduling method based on database hybrid analysis. By achieving fine-grained tracing at the field level and combining predictive scheduling and precision circuit breaking mechanisms, it has proactive defense and self-healing capabilities, effectively improving the balance of cluster resources and the continuity of core link services, thereby solving the problems in the background technology.

[0006] To achieve the above objectives, the technical solution of the present invention is as follows: A data lineage tracing and distributed scheduling method based on database hybrid analysis, which executes the following steps via computer equipment: S1: Construct a multidimensional data lineage graph model based on a directed acyclic graph, where the node types include data entity nodes, processing task nodes, and computation logic nodes, and the edge types include dependency edges representing the direction of data flow, generation edges representing the data generated by the task, and logical edges representing the mapping between fields and computation logic. S2: Based on the bloodline graph model, perform explicit and implicit bloodline data collection. Extract explicit bloodline relationships by parsing the structured query language and task configuration files. At the same time, deduce implicit computational dependencies at the field level by semantic parsing the code script and combining it with the change records in the database transaction log to generate fine-grained bloodline links. S3: Store the fine-grained lineage link in a graph database, and based on the business query request, provide the upper-layer application with interfaces for reverse tracing query, positive impact analysis query and compliance audit query based on the graph database; S4: Construct a multi-level distributed scheduling strategy that includes LSTM time-series prediction and real-time load monitoring. Based on the real-time load index and the future load index predicted by the LSTM model, adaptively select the corresponding strategy from static rule scheduling, real-time dynamic scheduling based on load index, predictive scheduling based on the predicted load, and intelligent collaborative scheduling for task allocation. S5: Based on the scheduling strategy of S4, calculate the comprehensive load assessment value of each computing node in real time. When the comprehensive load assessment value shows an expected imbalance, perform low-loss fragmentation migration and adopt incremental data synchronization and breakpoint resume mechanism during the migration process. S6: The high-frequency data access links and associated task chains identified in the lineage tracing analysis in S3 are used as feedback signals and input to the scheduling engine. The scheduling engine combines the comprehensive load assessment value of each computing node in S5 and prioritizes the allocation of the computing node with the lowest current comprehensive load assessment value to the high-frequency links in the scheduling decision, so as to realize the dynamic matching of data flow and resource allocation.

[0007] Preferably, the specific steps of S1 include: S11, Define data entity nodes The data entity nodes include database tables, field entities, and file system objects; S12, Define the processing task node The processing task nodes include ETL extraction, transformation and loading tasks and Flink stream processing tasks; S13, Define computation logic nodes The computational logic node includes user-defined functions, join operations, and aggregation operations; S14 defines the edge set E, and constructs data dependency edges, task generation edges, and field logic edges based on the node definitions in S11 to S13.

[0008] Preferably, the specific steps of S2 include: S21, for relational databases and big data computing components, parses structured query language statements and task configuration files, extracts the mapping relationship between input data sources and output data targets, and forms explicit lineage; S22 uses a pre-built code semantic parser to perform static analysis on the code script and combines it with change records in the database write-ahead log to identify field-level computational dependencies. S23 involves cross-validating and merging the kinship data obtained in S21 and S22 to generate a fine-grained kinship link covering the entire link.

[0009] Preferably, the specific steps of S3 include: S31 uses Neo4j graph database as the underlying storage engine to convert the fine-grained lineage links generated by S2 into graph structure nodes and edges for persistent storage. S32 provides a source tracing query interface, which responds to query requests for target data by traversing the graph database in reverse to locate all upstream source data; S33 provides an impact analysis interface that, in response to a change request for source data, forward infers the scope of the impact of the change on downstream applications; S34 provides a compliance audit interface and displays the entire flow path of personal data based on graph database path retrieval.

[0010] Preferably, the specific steps of S4 include: S41, When the cluster load is in a small-scale stable state, execute static rule scheduling based on static mapping of sharding keys and preset resource quotas; S42, when the cluster load experiences regular dynamic fluctuations, performs real-time dynamic scheduling based on multi-dimensional real-time load indicators and migration throttling mechanisms; S43, when the cluster faces periodic peak traffic, executes predictive scheduling based on the LSTM time series prediction model and performs warm-up migration; S44, when the cluster contains heterogeneous computing nodes and faces complex mixed loads, performs intelligent cooperative scheduling based on reinforcement learning decision-making and multi-agent collaboration.

[0011] Preferably, the specific constraint method for predictive scheduling based on the LSTM time series prediction model in S43 is as follows: A multidimensional training set is constructed based on historical load data, and a multidimensional input sequence with a historical time window length of n is defined. ; in Let k be the load vectors for each dimension at time k; Train an LSTM network to output a multidimensional prediction vector for future time t. Its prediction function is expressed as:

[0012] in This represents the mapping function of the LSTM network model. The set of parameters learned by the network; During the training of the LSTM network model, the loss function Loss is based on the historical comprehensive load evaluation value. The weighted construction is performed, and the specific calculation formula is as follows:

[0013] in:

[0014] in As the baseline weight of the sample, For adjustment coefficients, The comprehensive load assessment value calculated for historical time k makes the LSTM model tend to fit the load change patterns under high-frequency fluctuations and high-load historical windows during training; the trained prediction vectors Compared with the actual load vector Compare and calculate the mean absolute percentage error:

[0015] When the prediction accuracy When the predicted vector is greater than or equal to the preset accuracy threshold, the predicted vector will be... The updated input is incorporated into the evaluation model to participate in joint scheduling calculations. When the prediction accuracy ACC is less than the preset accuracy threshold, a prediction accuracy circuit breaker strategy is executed, forcing the dynamic fusion weight factor to... When the value is equal to 1, the predictive scheduling is disabled and degraded to the real-time dynamic scheduling. Joint scheduling is re-enabled after the accuracy of the prediction model recovers to a level greater than or equal to the preset accuracy threshold.

[0016] Preferably, the specific steps of S5 include: S51 collects real-time data on CPU utilization, memory utilization, network bandwidth utilization, and IOPS utilization of computing nodes. S52, the comprehensive load assessment value of each node calculated according to the preset weighted load assessment model. Calculate the load value of this node and compare it with the system's preset load balancing target threshold. Load deviation between :

[0017] When the load deviation When the load exceeds the preset tolerance range, it is determined that the node has a current load imbalance or a future expected load imbalance; S53, when a load imbalance is detected and the overall system load does not exceed the preset circuit breaker threshold, the scheduling engine triggers a shard migration action; S54 uses incremental data synchronization during fragment migration and records the breakpoint resume position. After the migration is completed, a status check is performed to ensure data consistency.

[0018] Preferably, the calculation method for the weighted load assessment model in S52 is as follows: Construct a joint evaluation metric that includes the current real-time load and the predicted future load, and define the final comprehensive scheduling load evaluation value of a node at the current time t. The calculation formula is as follows:

[0019] in, The real-time load assessment value at the current time t is calculated using the following formula:

[0020] in, These represent CPU utilization, memory utilization, network bandwidth utilization, and IOPS utilization, respectively. For future time-based The future load assessment value is calculated using the following formula:

[0021] in The multidimensional prediction vectors predicted by the LSTM model The component in the i-th dimension; the weight coefficient Adaptive normalization calculation based on predicted load is employed:

[0022] in The smoothing coefficient; the dynamic fusion weighting factor The calculation formula is:

[0023] Where ACC represents the prediction accuracy. The preset accuracy threshold, To control the dynamic fusion weight factor The slope parameter that smoothly transitions to 1 when the prediction accuracy exceeds the effective lower limit.

[0024] Preferably, the specific steps of S6 include: S61, based on the lineage tracing results in S3, identifies high-frequency data access links across tables and tasks, as well as associated high-frequency computing task chains; S62, map the high-frequency data access path and computing task chain to scheduling priority labels; S63, the scheduling priority label is input into the scheduling strategy of S4. When performing task allocation, the comprehensive load assessment value of each computing node mentioned in S5 is combined to prioritize scheduling tasks in the high-frequency computing task chain to the computing node with the lowest current comprehensive load assessment value, so as to realize dynamic matching of data flow and resource allocation.

[0025] As can be seen from the above technical solution compared with the prior art, the present invention has the following beneficial effects: 1. This invention constructs a three-dimensional directed acyclic graph lineage model containing data entities, processing tasks, and computational logic nodes. It employs a data acquisition mechanism that integrates explicit SQL parsing and implicit code semantic parsing, and stores the data in a graph database. This enables the system to achieve precise end-to-end tracing at the field level. The high-frequency data access links and associated task chains identified by lineage analysis are used as feedback signals and input back to the scheduling engine. The scheduling engine then proactively allocates high-performance computing nodes to the high-frequency business links, achieving dynamic matching of data flow and resource allocation. This significantly improves the system response speed and anomaly detection efficiency in high-frequency mixed load scenarios.

[0026] 2. This invention integrates the LSTM time-series prediction model with multi-dimensional real-time load monitoring to construct a multi-level scheduling strategy from static rules to intelligent collaboration. It also designs a joint evaluation index based on the current real-time load and the future predicted load, enabling the scheduling engine to detect periodic traffic peaks in advance and perform pre-warming migration. At the same time, once the LSTM prediction accuracy falls below a preset threshold, the system will automatically force the evaluation weight to focus on the real-time load, smoothly degrading to stable and reliable real-time dynamic scheduling. This mechanism ensures the predictability of scheduling during daily peak periods and has a high availability backup capability when the prediction model fails or encounters sudden abnormal traffic, greatly improving the business continuity and resource balance of the distributed cluster. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the method steps in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0029] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention, but should not be used to limit the scope of the present invention.

[0030] This invention provides a data lineage tracing and distributed scheduling method based on database hybrid analysis, such as... Figure 1 As shown, perform the following steps using a computer device: S1: Construct a multidimensional data lineage graph model based on a directed acyclic graph, where the node types include data entity nodes, processing task nodes, and computation logic nodes, and the edge types include dependency edges representing the direction of data flow, generation edges representing the data generated by the task, and logical edges representing the mapping between fields and computation logic. S2: Based on the bloodline graph model, perform explicit and implicit bloodline data collection. Extract explicit bloodline relationships by parsing the structured query language and task configuration files. At the same time, deduce implicit computational dependencies at the field level by semantic parsing the code script and combining it with the change records in the database transaction log to generate fine-grained bloodline links. S3: Store the fine-grained lineage link in a graph database, and based on the business query request, provide the upper-layer application with interfaces for reverse tracing query, positive impact analysis query and compliance audit query based on the graph database; S4: Construct a multi-level distributed scheduling strategy that includes LSTM time-series prediction and real-time load monitoring. Based on the real-time load index and the future load index predicted by the LSTM model, adaptively select the corresponding strategy from static rule scheduling, real-time dynamic scheduling based on load index, predictive scheduling based on the predicted load, and intelligent collaborative scheduling for task allocation. S5: Based on the scheduling strategy of S4, calculate the comprehensive load assessment value of each computing node in real time. When the comprehensive load assessment value shows an expected imbalance, perform low-loss fragmentation migration and adopt incremental data synchronization and breakpoint resume mechanism during the migration process. S6: The high-frequency data access links and associated task chains identified in the lineage tracing analysis in S3 are used as feedback signals and input to the scheduling engine. The scheduling engine combines the comprehensive load assessment value of each computing node in S5 and prioritizes the allocation of the computing node with the lowest current comprehensive load assessment value to the high-frequency links in the scheduling decision, so as to realize the dynamic matching of data flow and resource allocation.

[0031] Example: In this embodiment, a big data hybrid analysis system is used as an example. This system deploys a data lake, a real-time computing engine, and a distributed relational database, and needs to handle a mixed load of user transactions and complex operational analysis reports simultaneously. Therefore, a data lineage tracing and distributed scheduling method based on database hybrid analysis is introduced from this invention, and the specific implementation is as follows: S11, Define a data entity node VD, which includes database tables, field entities, and file system objects; for example, the "order table" in the transaction database and the "order amount" and "user ID" field entities in the table, as well as the "user click stream log file" on the HDFS file system are uniformly defined as data entity nodes VD.

[0032] S12, define a processing task node VT, which includes an ETL extraction, transformation and loading task and a Flink stream processing task; for example, define the "Daily Order Summary ETL Offline Task" executed at midnight every day and the Flink stream computing task responsible for real-time calculation of "Real-time User Profile" as the processing task node VT.

[0033] S13, define a computational logic node VL, which includes user-defined functions, join operations, and aggregation operations; for example, the custom UDF function that performs currency conversion on "order amount" during data cleaning, the JOIN operation that associates "user ID" and "product ID", and the SUM aggregation operation on transaction data are defined as computational logic nodes VL.

[0034] S14 defines edge set E. Based on the node definitions in S11 to S13, data dependency edges, task generation edges, and field logic edges are constructed. For example, a dependency edge is constructed from the "order table" to the ETL offline task, a generation edge is constructed from the ETL task to the "daily summary wide table", and a logic edge is constructed from the "order amount" field to generate the "dollar amount" field after the UDF calculation.

[0035] By constructing a multidimensional DAG model containing three-dimensional nodes (VD, VT, and VL) and three types of edges, this invention overcomes the technical limitation of traditional lineage tracing, which can only track coarse-grained relationships at the table level. It achieves a three-dimensional, three-dimensional association between data entities, computational logic, and processing tasks. The field logical edges connected by S14 significantly enhance the richness of metadata, laying a solid data topology foundation for subsequent fine-grained (field-level) lineage tracing and filling the gap in logical metadata management in traditional methods.

[0036] S21, for relational databases and big data computing components, parses structured query language statements and task configuration files, extracts the mapping relationship between input data sources and output data targets, and forms an explicit lineage; for example, the system parses Spark task configuration files and SQL statements, extracts "ods_trade_log" as the input source and "dw_trade_analysis" as the output target, establishes basic table-level mappings, and forms an explicit lineage chain.

[0037] S22 uses a pre-built code semantic parser to perform static analysis on the code script and combines it with change records in the database write-ahead log to identify field-level computational dependencies. For example, the code semantic parser deeply analyzes the PySpark script developed for business applications and accurately identifies the underlying field computational dependency logic of `real_pay = original_price * 0.8 - coupon`. At the same time, it combines the change records in the underlying database WAL log and cross-compares the flow of field data within the database to deduce the implicit dependency path.

[0038] S23 cross-validates and merges the lineage data obtained in S21 and S22 to generate a fine-grained lineage chain covering the entire chain. For example, the "table-to-table" mapping relationship explicitly extracted in S21 is cross-validated and automatically merged with the "field-to-field" calculation logic implicitly derived in S22 to finally generate a complete fine-grained lineage chain of "original amount field of order table -> 0.8 deduction UDF -> coupon deduction UDF -> target actual payment amount field".

[0039] By employing a dual-mode (explicit and implicit) data acquisition mechanism, this invention effectively addresses the technical pain point of existing technologies that rely solely on SQL parsing to capture complex internal logic and implicit field dependencies within Python / PySpark scripts. Through S23 cross-validation and merging, the derivation rate of implicit lineage is increased to over 95%, enabling data operations personnel to quickly and accurately pinpoint the source of data anomalies at the field level. This completely eliminates the audit blind spots caused by the opaque data flow in traditional technologies, significantly reducing the operational costs of manual investigation.

[0040] S31 uses Neo4j graph database as the underlying storage engine, transforming the fine-grained lineage links generated by S2 into graph structure nodes and edges for persistent storage; for example, it transforms field-level dependencies into nodes and relation edges in the graph database, supporting efficient storage of ultra-large-scale lineage networks.

[0041] S32 provides a source tracing query interface, responding to query requests for target data by traversing the graph database in reverse to locate all upstream source data. For example, when the data in the "Double 11 Promotion Activity Effect Report" is abnormal, the operation and maintenance personnel initiate a query through the front-end source tracing interface. The graph database completes a deep traversal in milliseconds and reverses the location to find that the root cause of the abnormality is an accidental deletion operation of an upstream cleaning script.

[0042] S33 provides an impact analysis interface to respond to change requests for source data and to deduce the scope of the impact of the change in downstream applications. For example, when the field type of the underlying "product base table" changes, the system actively triggers the impact analysis interface to deduce that the change will affect three downstream data API interfaces and two AI prediction models, and the system will then automatically issue a maintenance warning.

[0043] S34 provides a compliance audit interface, which uses graph database path retrieval to display the entire flow path of personal data. For example, in the face of data privacy compliance audit requirements, the system can quickly retrieve the entire lifecycle flow path of a specific user behavior data in the database from "collection-cleaning-aggregation-application" through the compliance audit interface and generate a compliance report.

[0044] By employing the Neo4j graph database for persistent storage of lineage links, this invention overcomes the technical shortcomings of traditional relational databases in terms of low performance when traversing multi-hop lineage links. By providing three core query interfaces, the anomaly detection time is reduced from "hours" in traditional solutions to "seconds / milliseconds," while efficiently supporting compliance audit requirements of laws and regulations such as GDPR, thus achieving complete observability, controllability, and traceability of data flow.

[0045] S41, when the cluster load is in a small-scale stable state, static rule scheduling based on shard key static mapping and preset resource quotas is executed; for example, during the business off-peak period from 3 am to 5 am, the system directly performs interference-free static scheduling based on the fixed shard key mapping table and preset CPU / memory quotas.

[0046] S42, when the cluster load experiences regular dynamic fluctuations, executes real-time dynamic scheduling based on multi-dimensional real-time load indicators and migration throttling mechanisms; for example, during the daily daytime business fluctuation period, the system collects multi-dimensional load indicators in real time and smoothly fine-tunes according to the set throttling mechanism to avoid jitter caused by frequent migrations.

[0047] S43, when the cluster faces periodic peak traffic, executes predictive scheduling based on the LSTM time series prediction model and performs warm-up migration; The specific method of predictive scheduling based on the LSTM time series prediction model in S43 is as follows: A multidimensional training set is constructed based on historical load data, and a multidimensional input sequence with a historical time window length of n is defined. ; in Let k be the load vectors for each dimension at time k; Train an LSTM network to output a multidimensional prediction vector for future time t. Its prediction function is expressed as:

[0048] in This represents the mapping function of the LSTM network model. The set of parameters learned by the network; During the training of the LSTM network model, the loss function Loss is based on the historical comprehensive load evaluation value. The weighted construction is performed, and the specific calculation formula is as follows:

[0049] in:

[0050] in As the baseline weight of the sample, For adjustment coefficients, The comprehensive load assessment value calculated for historical time k makes the LSTM model tend to fit the load change patterns under high-frequency fluctuations and high-load historical windows during training; the trained prediction vectors Compared with the actual load vector Compare and calculate the mean absolute percentage error:

[0051] When the prediction accuracy When the predicted vector is greater than or equal to the preset accuracy threshold, the predicted vector will be... The updated input is incorporated into the evaluation model to participate in joint scheduling calculations. When the prediction accuracy ACC is less than the preset accuracy threshold, a prediction accuracy circuit breaker strategy is executed, forcing the dynamic fusion weight factor to... When the value is equal to 1, the predictive scheduling is disabled and degraded to the real-time dynamic scheduling. Joint scheduling is re-enabled after the accuracy of the prediction model recovers to a level greater than or equal to the preset accuracy threshold.

[0052] For example, in response to predictable periodic traffic peaks, the system triggers the LSTM prediction model in advance, combines the load data from the past three months to generate an estimated load vector for the next half hour, and performs a pre-warming data sharding migration 30 seconds before the peak to ensure that nodes are ready with computing power to meet the peak.

[0053] S44 When the cluster contains heterogeneous computing nodes and faces complex mixed loads, it performs intelligent collaborative scheduling based on reinforcement learning decision-making and multi-agent collaboration. For example, for complex mixed scenarios such as real-time writing with high IOPS requirements and analysis queries with high computing power requirements, the system adopts a multi-agent collaboration mechanism and automatically balances the resource allocation ratio of various requests through a reinforcement learning decision-making model.

[0054] By establishing a four-level hierarchical scheduling architecture, this invention overcomes the shortcomings of traditional single static or delayed dynamic scheduling strategies that cannot adapt to varying load scenarios. In particular, after introducing the S43 LSTM time-series prediction model, the system's resource scheduling capability has been upgraded from a passive, delayed response to a proactive, forward-looking defense. This allows the machine to proactively pre-allocate resources before traffic peaks arrive, greatly improving resource balance among nodes and effectively avoiding the risk of overloaded hotspot nodes caused by sudden traffic surges.

[0055] S51 collects real-time data on CPU utilization, memory utilization, network bandwidth utilization, and IOPS utilization of computing nodes. For example, during peak online transaction periods, the distributed scheduling component acquires various hardware load metrics of each computing node at extremely high frequency to perceive system pressure in real time.

[0056] S52, calculate the load deviation between the node's load value and the system's preset balance target threshold based on the comprehensive load assessment value of each node calculated by the preset weighted load assessment model; when the load deviation exceeds the preset tolerance range, determine that the node has a current load imbalance or a future expected load imbalance.

[0057] The calculation method for the weighted load assessment model in S52 is as follows: Construct a joint evaluation metric that includes the current real-time load and the predicted future load, and define the final comprehensive scheduling load evaluation value of a node at the current time t. The calculation formula is as follows:

[0058] in, The real-time load assessment value at the current time t is calculated using the following formula:

[0059] in, These represent CPU utilization, memory utilization, network bandwidth utilization, and IOPS utilization, respectively. For future time-based The future load assessment value is calculated using the following formula:

[0060] in The multidimensional prediction vectors predicted by the LSTM model The component in the i-th dimension; the weight coefficient Adaptive normalization calculation based on predicted load is employed:

[0061] in The smoothing coefficient; the dynamic fusion weighting factor The calculation formula is:

[0062] Where ACC represents the prediction accuracy. The preset accuracy threshold, To control the dynamic fusion weight factor The slope parameter that smoothly transitions to 1 when the prediction accuracy exceeds the effective lower limit.

[0063] For example, the system inputs the real-time collected load value and the load value predicted by LSTM to calculate the expected comprehensive load assessment value of the node in 5 minutes. If the expected load value of the node in the future significantly exceeds the balancing threshold, the system will identify it as a node expected to be unbalanced in the future, even if the current node is not under great pressure, and take timely preventive actions.

[0064] S53, when a load imbalance is detected and the overall system load does not exceed the preset circuit breaker threshold, the scheduling engine triggers a shard migration action; for example, if the system determines that a shard node will reach a bottleneck in 3 minutes and the overall cluster resource utilization rate is still at a safe level, the scheduling engine immediately issues an instruction to migrate the hot data shard to an idle node.

[0065] S54 employs incremental data synchronization during shard migration and records breakpoint resume positions. After migration, a status check is performed to ensure data consistency. For example, during migration, the system only synchronizes incremental data where differences arise between two nodes and uses a breakpoint recording mechanism to ensure that if network fluctuations cause migration interruptions, the next migration can resume directly from the point of interruption. After migration, a consistency status check is performed based on the Raft protocol to ensure seamless service operation.

[0066] By combining the S52 weighted load assessment model with LSTM prediction vectors, this invention overcomes the shortcomings of existing technologies that rely solely on real-time load, resulting in a decision-making lag of 1-3 minutes. In particular, the introduction of a dynamic circuit breaker mechanism based on prediction accuracy allows the system to automatically revert to pure real-time load scheduling when sudden abnormal traffic causes LSTM model predictions to become inaccurate (below the accuracy threshold), effectively avoiding the cascading problems that may be caused by AI model "misjudgments." Combined with low-loss fragmentation migration and breakpoint resumption, this truly achieves high availability and strong consistency in the HTAP system during resource reallocation, with extremely fast fault self-healing speed.

[0067] S61, based on the lineage tracing results in S3, identifies high-frequency data access links across tables and tasks, as well as associated high-frequency computing task chains; for example, through cross-analysis of the lineage graph and online call chain, the system identifies that the "Double 11 Flash Sale Business" accesses the "User Account Table" and "Inventory Snapshot Table" at high frequency within milliseconds, and associates the links of multiple high-concurrency UDF computing tasks, marking them as high-frequency data access links.

[0068] S62, the high-frequency data access path and computing task chain are mapped to scheduling priority labels; for example, the system maps the high-frequency link to the "highest priority" label and ordinary offline report analysis query to the "low priority" label.

[0069] In step S63, the scheduling priority tag is input into the scheduling strategy in step S4. When allocating tasks, based on the comprehensive load assessment value of each computing node described in step S5, tasks in the high-frequency computing task chain are prioritized and scheduled to the computing node with the lowest current comprehensive load assessment value, thereby achieving dynamic matching between data flow and resource allocation. For example, after receiving this tag, during the most intense period of the Double 11 promotion, the scheduling engine breaks the conventional queuing mechanism and forcibly schedules the highest priority task chain, "flash sale business," to the newly deployed, most powerful, and currently most idle node, ensuring that the most critical business data flow can obtain the strongest computing power and achieving precise resource allocation.

[0070] This invention transforms S3 lineage analysis results into high-frequency link tags and inputs them back into the scheduling engine. This allows scheduling decisions to move beyond blind queuing and become more targeted, proactively ensuring maximum computing power for the most critical business links. Through this cross-layer collaborative optimization, the throughput and response speed of the entire HTAP database under concurrency surges are significantly improved, ensuring the stability of core business links.

[0071] It is understood that the systems, devices, and storage media provided in the embodiments of the present invention correspond to the methods provided in the embodiments of the present invention, and the explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding parts of the above methods.

[0072] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0073] For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media.

[0074] The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0075] It should be noted that in this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0076] Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0077] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0078] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A data lineage tracing and distributed scheduling method based on database hybrid analysis, characterized in that, Perform the following steps using a computer device: S1: Construct a multidimensional data lineage graph model based on a directed acyclic graph, where the node types include data entity nodes, processing task nodes, and computation logic nodes, and the edge types include dependency edges representing the direction of data flow, generation edges representing the data generated by the task, and logical edges representing the mapping between fields and computation logic. S2: Based on the bloodline graph model, perform explicit and implicit bloodline data collection. Extract explicit bloodline relationships by parsing the structured query language and task configuration files. At the same time, deduce implicit computational dependencies at the field level by semantic parsing the code script and combining it with the change records in the database transaction log to generate fine-grained bloodline links. S3: Store the fine-grained lineage link in a graph database, and based on the business query request, provide the upper-layer application with interfaces for reverse tracing query, positive impact analysis query and compliance audit query based on the graph database; S4: Construct a multi-level distributed scheduling strategy that includes LSTM time-series prediction and real-time load monitoring. Based on the real-time load index and the future load index predicted by the LSTM model, adaptively select the corresponding strategy from static rule scheduling, real-time dynamic scheduling based on load index, predictive scheduling based on the predicted load, and intelligent collaborative scheduling for task allocation. S5: Based on the scheduling strategy of S4, calculate the comprehensive load assessment value of each computing node in real time. When the comprehensive load assessment value shows an expected imbalance, perform low-loss fragmentation migration and adopt incremental data synchronization and breakpoint resume mechanism during the migration process. S6: The high-frequency data access links and associated task chains identified in the lineage tracing analysis in S3 are used as feedback signals and input to the scheduling engine. The scheduling engine combines the comprehensive load assessment value of each computing node in S5 and prioritizes the allocation of the computing node with the lowest current comprehensive load assessment value to the high-frequency links in the scheduling decision, so as to realize the dynamic matching of data flow and resource allocation.

2. The data lineage tracing and distributed scheduling method based on database hybrid analysis as described in claim 1, characterized in that: The specific steps of S1 include: S11, Define data entity nodes The data entity nodes include database tables, field entities, and file system objects; S12, Define the processing task node The processing task nodes include ETL extraction, transformation and loading tasks and Flink stream processing tasks; S13, Define computational logic nodes The computational logic node includes user-defined functions, join operations, and aggregation operations; S14 defines the edge set E, and constructs data dependency edges, task generation edges, and field logic edges based on the node definitions in S11 to S13.

3. The data lineage tracing and distributed scheduling method based on database hybrid analysis as described in claim 1, characterized in that: The specific steps of S2 include: S21, for relational databases and big data computing components, parses structured query language statements and task configuration files, extracts the mapping relationship between input data sources and output data targets, and forms explicit lineage; S22 uses a pre-built code semantic parser to perform static analysis on the code script and combines it with change records in the database write-ahead log to identify field-level computational dependencies. S23 involves cross-validating and merging the kinship data obtained in S21 and S22 to generate a fine-grained kinship link covering the entire link.

4. The data lineage tracing and distributed scheduling method based on database hybrid analysis as described in claim 1, characterized in that: The specific steps of S3 include: S31 uses Neo4j graph database as the underlying storage engine to convert the fine-grained lineage links generated by S2 into graph structure nodes and edges for persistent storage. S32 provides a source tracing query interface, which responds to query requests for target data by traversing the graph database in reverse to locate all upstream source data; S33 provides an impact analysis interface that, in response to a change request for source data, forward infers the scope of the impact of the change on downstream applications; S34 provides a compliance audit interface, which uses graph database path retrieval to display the entire flow path of data.

5. The data lineage tracing and distributed scheduling method based on database hybrid analysis as described in claim 1, characterized in that: The specific steps of S4 include: S41, When the cluster load is in a small-scale stable state, execute static rule scheduling based on static mapping of sharding keys and preset resource quotas; S42, when the cluster load experiences regular dynamic fluctuations, performs real-time dynamic scheduling based on multi-dimensional real-time load indicators and migration throttling mechanisms; S43, when the cluster faces periodic peak traffic, executes predictive scheduling based on the LSTM time series prediction model and performs warm-up migration; S44, when the cluster contains heterogeneous computing nodes and faces complex mixed loads, performs intelligent cooperative scheduling based on reinforcement learning decision-making and multi-agent collaboration.

6. The data lineage tracing and distributed scheduling method based on database hybrid analysis as described in claim 5, characterized in that: The specific method of predictive scheduling based on the LSTM time-series prediction model in S43 is as follows: A multidimensional training set is constructed based on historical load data, and a multidimensional input sequence with a historical time window length of n is defined. ; in Let k be the load vectors for each dimension at time k; Train an LSTM network to output a multidimensional prediction vector for future time t. Its prediction function is expressed as: in This represents the mapping function of the LSTM network model. The set of parameters learned by the network; During the training of the LSTM network model, the loss function Loss is based on the historical comprehensive load evaluation value. The weighted construction is performed, and the specific calculation formula is as follows: in: in As the baseline weight of the sample, For adjustment coefficients, The comprehensive load assessment value calculated for historical time k makes the LSTM model tend to fit the load change patterns under high-frequency fluctuations and high-load historical windows during training; the trained prediction vectors Compared with the actual load vector Compare and calculate the mean absolute percentage error: When the prediction accuracy When the predicted vector is greater than or equal to the preset accuracy threshold, the predicted vector will be... The updated input is incorporated into the evaluation model to participate in joint scheduling calculations. When the prediction accuracy ACC is less than the preset accuracy threshold, a prediction accuracy circuit breaker strategy is executed, forcing the dynamic fusion weight factor to... When the value is equal to 1, the predictive scheduling is disabled and degraded to the real-time dynamic scheduling. Joint scheduling is re-enabled after the accuracy of the prediction model recovers to a level greater than or equal to the preset accuracy threshold.

7. The data lineage tracing and distributed scheduling method based on database hybrid analysis as described in claim 6, characterized in that: The specific steps of S5 include: S51 collects real-time data on CPU utilization, memory utilization, network bandwidth utilization, and IOPS utilization of computing nodes. S52, the comprehensive load assessment value of each node calculated according to the preset weighted load assessment model. Calculate the load value of this node and compare it with the system's preset load balancing target threshold. Load deviation between : When the load deviation When the load exceeds the preset tolerance range, it is determined that the node has a current load imbalance or a future expected load imbalance; S53, when a load imbalance is detected and the overall system load does not exceed the preset circuit breaker threshold, the scheduling engine triggers a shard migration action; S54 uses incremental data synchronization during fragment migration and records the breakpoint resume position. After the migration is completed, a status check is performed to ensure data consistency.

8. The data lineage tracing and distributed scheduling method based on database hybrid analysis as described in claim 7, characterized in that: The calculation method for the weighted load assessment model in S52 is as follows: Construct a joint evaluation metric that includes the current real-time load and the predicted future load, and define the final comprehensive scheduling load evaluation value of a node at the current time t. The calculation formula is as follows: in, The real-time load assessment value at the current time t is calculated using the following formula: in, These represent CPU utilization, memory utilization, network bandwidth utilization, and IOPS utilization, respectively. For future time-based The future load assessment value is calculated using the following formula: in The multidimensional prediction vectors predicted by the LSTM model The component in the i-th dimension; the weight coefficient Adaptive normalization calculation based on predicted load is employed: in The smoothing coefficient; the dynamic fusion weighting factor The calculation formula is: Where ACC represents the prediction accuracy. The preset accuracy threshold, To control the dynamic fusion weight factor The slope parameter that smoothly transitions to 1 when the prediction accuracy exceeds the effective lower limit.

9. The data lineage tracing and distributed scheduling method based on database hybrid analysis as described in claim 1, characterized in that: The specific steps of S6 include: S61, based on the lineage tracing results in S3, identifies high-frequency data access links across tables and tasks, as well as associated high-frequency computing task chains; S62, map the high-frequency data access path and computing task chain to scheduling priority labels; S63, the scheduling priority label is input into the scheduling strategy of S4. When performing task allocation, the comprehensive load assessment value of each computing node mentioned in S5 is combined to prioritize scheduling tasks in the high-frequency computing task chain to the computing node with the lowest current comprehensive load assessment value, so as to realize dynamic matching of data flow and resource allocation.

Images