Data weaving system and method based on dynamic metadata identification and DAG optimization

Abstract

The invention provides a data knitting system and method based on dynamic metadata identification and DAG optimization, and relates to the technical field of data knitting, the data knitting system comprises a metadata management layer, a data processing layer and an execution layer, the metadata management layer comprises a dynamic metadata identification module and a labeling storage engine module, the data processing layer comprises an adaptive DAG optimization engine module and a dynamic task distributor module, and the execution layer comprises a data computing node cluster module and a multi-source data connector module; the method comprises the steps that a DAG task scheduling strategy is established, and dynamic optimization is achieved; according to the method, the service labels and DAG task scheduling are deeply bound to form a'data-calculation-service 'closed loop, the execution sequence and resource allocation of the tasks can be adjusted according to the service labels changing in real time, so that the task scheduling not only considers the use of calculation resources, but also can optimize the tasks in real time according to service requirements, and the task scheduling efficiency is improved. Therefore, the overall treatment efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of data weaving technology, and in particular to a data weaving system and method based on dynamic metadata identification and DAG optimization. Background Art

[0002] In a data weaving architecture, data source integration, task scheduling, and resource optimization are key to ensuring efficient data processing and system performance. However, existing technologies still have significant pain points in these areas, particularly in metadata management, DAG task scheduling, and cross-data source collaboration.

[0003] Traditional metadata management methods typically rely on static data storage structures, such as obtaining metadata from a database through a JDBC interface. While this approach can provide basic metadata access, it lacks the ability to dynamically expand and flexibly identify metadata.

[0004] The DAG (Directed Acyclic Graph) operation graph is a technology that implements task scheduling by decomposing tasks into multiple independent steps. Each step represents an operator, and these operators are connected by the edges of the DAG graph to form a dependency relationship for task execution. Although DAG can provide efficient task decoupling and maintainability, traditional DAG task scheduling technology relies on static task allocation rules and lacks sufficient flexibility when facing dynamically changing workloads. For example, the system cannot adjust task allocation in real time based on metadata status (such as changes in data volume, task priority, etc.). This static rule leads to resource waste and response delays, especially in scenarios that require fast response.

[0005] Multi-source data federated query technology allows the system to access multiple heterogeneous data sources (such as relational databases, NoSQL databases, file storage, distributed storage, etc.) at the same time, and can perform joint queries across data sources. When operating on multi-source data, traditional technologies usually fail to fully consider the collaborative relationship between metadata and DAG task scheduling. Metadata identification and DAG task scheduling fail to effectively link, making it impossible to accurately match computing resource allocation with actual data requirements and task scheduling requirements. This leads to inefficient utilization of computing resources and bottlenecks in query performance. Therefore, the present invention proposes a data weaving system and method based on dynamic metadata identification and DAG optimization to solve the problems existing in the prior art. Summary of the Invention

[0006] In response to the above problems, the purpose of the present invention is to propose a data weaving system and method based on dynamic metadata identification and DAG optimization. This data weaving system and method based on dynamic metadata identification and DAG optimization can reduce data processing delays, improve multi-source query performance, and solve the problems existing in the prior art.

[0007] To achieve the purpose of the present invention, the present invention is implemented through the following technical solution: a data weaving method based on dynamic metadata identification and DAG optimization, characterized in that it includes the following steps:

[0008] Step 1: Establish a DAG task scheduling strategy to achieve dynamic optimization;

[0009] Step 2: Identify the metadata status and trigger the DAG task scheduling policy update when a change occurs;

[0010] Step 3: Link the dynamic optimization update results of step 2 with the external data source mapping.

[0011] A further improvement is that in step 1, the specific method of establishing the DAG task scheduling strategy is as follows:

[0012] A multidimensional scheduling model based on metadata state vector is established, and dynamic optimization is achieved through priority weight coefficient α, resource elasticity factor β and data lineage correlation γ.

[0013] A further improvement is that in step 2, an improved genetic algorithm is used to optimize the scheduling strategy. The specific steps are as follows:

[0014] S1: Gene encoding, which encodes the DAG node execution order and resource allocation scheme into chromosomes;

[0015] S2: Fitness function, used to evaluate the quality of the current chromosome. Its formula is:

[0016] F = α*(1-delay rate) + β*resource utilization + γ*task completion;

[0017] S3: Iterative optimization, generating a new generation of scheduling solutions based on the latest metadata status every N minutes, where N>1.

[0018] A further improvement is that the dynamic optimization includes the following optimization operations:

[0019] A1. Task sharding and reorganization: Perform DAG graph structure analysis on operators and merge related tasks with high γ values into atomic execution units.

[0020] A2, resource preemptive allocation: When α>0.8, the GPU resource reservation mechanism is triggered and thread allocation of low-priority tasks is suspended;

[0021] A3. Incremental checkpoint mechanism: Periodically saves intermediate state data during execution. When a task fails, only the most recent checkpoint needs to be rolled back.

[0022] Further improvements include: the priority weight coefficient automatically calculates the task priority based on the urgency marked in the metadata tag; the data lineage correlation calculates the data dependency strength between tasks based on the metadata lineage map, optimizes the task sharding and merging strategy; and the resource elasticity factor dynamically adjusts the parallelism based on the real-time monitored CPU / memory usage. The formula is as follows:

[0023] β = 1 / (1 + e^(-k*(1-resource availability))).

[0024] The data weaving system based on dynamic metadata identification and DAG optimization includes a metadata management layer, a data processing layer, and an execution layer. The metadata management layer synchronizes metadata status to the data processing layer, and the data processing layer sends resource scheduling instructions to the execution layer. The execution layer executes the instructions, wherein the metadata management layer includes:

[0025] Dynamic metadata identification module, used to expand the metadata storage structure through the JDBC interface, set up business tags, priority tags, data lineage relationship fields, and identify metadata identification status;

[0026] The data processing layer includes:

[0027] Adaptive DAG optimization engine module, used to establish DAG task scheduling strategies and achieve dynamic optimization;

[0028] The dynamic task allocator module is used to trigger the update of DAG task scheduling strategy, adjust task scheduling and reallocate resources.

[0029] The execution layer includes:

[0030] The multi-source data connector module is used to link the dynamic optimization update results with the external data source mapping.

[0031] A further improvement is that a tag storage engine module is also provided in the metadata management layer for fast reading and writing of metadata tags.

[0032] A further improvement is that a data computing node cluster module is also provided in the execution layer for actual computing and data processing of tasks.

[0033] The beneficial effects of the present invention are: the present invention deeply binds business tags with DAG task scheduling to form a "data-computing-business" closed loop, and can adjust the execution order and resource allocation of tasks according to real-time changing business tags. Therefore, task scheduling not only takes into account the use of computing resources, but also can optimize tasks in real time according to business needs, thereby improving overall processing efficiency. Based on the metadata status, the system can adjust the parallelism of tasks in real time to achieve lightweight elastic resource allocation, thereby improving resource utilization. Therefore, the present invention has achieved remarkable results in resource optimization, task scheduling, query performance and other aspects. The present invention not only improves resource utilization and reduces data processing delays, but also significantly improves multi-source data query performance, and ultimately improves the overall efficiency and business responsiveness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the system structure of the present invention.

[0035] Figure 2 It is a schematic diagram of the analysis and adjustment process of the present invention. DETAILED DESCRIPTION

[0036] In order to deepen the understanding of the present invention, the present invention will be further described in detail below with reference to the examples. The examples are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0037] Example 1: According to Figure 1 As shown, this embodiment proposes a data weaving system based on dynamic metadata identification and DAG optimization, including a metadata management layer, a data processing layer and an execution layer.

[0038] The metadata management layer synchronizes metadata status with the data processing layer. It includes a dynamic metadata identification module and a tagged storage engine module. The metadata management layer is primarily responsible for metadata storage, classification, identification, and management. It interacts with external databases to ensure that all metadata in the system is effectively managed and updated in real time. The dynamic metadata identification module extends the metadata storage structure through a JDBC interface, establishing business tags, priority tags, and data lineage fields. This allows users to annotate and categorize metadata in real time (e.g., "high priority" or "sensitive data") via the API or UI. By dynamically updating metadata tags, the system can adapt to diverse business needs and supports real-time metadata tagging and classification via the API or UI, enhancing system flexibility and maintainability. The tagged storage engine module uses a lightweight key-value store (such as RocksDB) to quickly read and write metadata tags. Because the tagged storage engine focuses on metadata management, it ensures rapid access and update of metadata tags when processing large amounts of data, improving data processing efficiency and responsiveness.

[0039] The data processing layer sends resource scheduling instructions to the execution layer. The data processing layer includes an adaptive DAG optimization engine module and a dynamic task allocator module. The data processing layer is responsible for scheduling and optimizing task execution, ensuring efficient allocation and utilization of computing resources. The adaptive DAG optimization engine dynamically adjusts task scheduling strategies based on the metadata status and resource availability of the current task. This engine uses the DAG (Directed Acyclic Graph) model to decompose complex computing tasks into multiple steps and optimizes tasks based on multi-dimensional information such as task priority, resource utilization, and data lineage. The adaptive optimization engine adjusts the DAG execution strategy based on real-time metadata changes (such as task priority and resource usage) to ensure optimal task execution. The dynamic task allocator module allocates specific computing tasks to compute nodes in the execution layer based on the scheduling instructions from the adaptive DAG optimization engine. It not only considers the resource requirements of tasks, but also dynamically allocates tasks based on task priority and the current state of available resources. This module ensures efficient utilization of system resources (such as CPU, GPU, and memory) and that tasks are properly scheduled according to their priority.

[0040] The execution layer executes instructions and is primarily responsible for actual computation and task execution, ensuring that task scheduling instructions received from the data processing layer are effectively executed. The execution layer includes a data computing node cluster module and a multi-source data connector module. The data computing node cluster module contains multiple compute node clusters, which are responsible for executing assigned computational tasks. Each node can process tasks in parallel. The execution layer's compute node clusters dynamically adjust their computing resources (e.g., CPU, memory, GPU, etc.) based on the task scheduling instructions from the data processing layer to maximize task processing efficiency—that is, the actual computation and data processing for the tasks. In high-concurrency scenarios, the execution layer automatically adjusts the degree of parallelism to ensure efficient system operation even with limited resources. The multi-source data connector module connects and manages data from multiple data sources. It supports cross-database and cross-file system queries and data operations, ensuring access to data from different sources during task execution. The execution layer uses multi-source data connectors to ensure data flow between different compute nodes, supporting complex cross-source queries and computation tasks, thereby ensuring successful task completion.

[0041] Through the effective collaboration of the metadata management layer, the data processing layer, and the execution layer, the data weaving system based on dynamic metadata identification and DAG optimization proposed in this embodiment can achieve efficient scheduling and execution of large-scale data tasks.

[0042] Example 2: According to Figure 1-Figure 2 As shown, this embodiment proposes a data weaving method based on dynamic metadata identification and DAG optimization, including the following steps:

[0043] Step 1: Establish a DAG task scheduling strategy to achieve dynamic optimization. The specific method is as follows:

[0044] A multidimensional scheduling model based on metadata state vectors is established. By building a multidimensional scheduling model, the order and priority of task execution can be dynamically adjusted to achieve efficient resource allocation and task scheduling. Dynamic optimization is then achieved through the priority weight coefficient α, resource elasticity factor β, and data lineage correlation γ.

[0045] The priority weight coefficient automatically calculates the task priority based on the urgency marked in the metadata tag. The larger the value of α, the higher the priority of the task, and the system will give priority to the task. Tags include "high priority", "normal priority", "low priority", etc. Through these tags, the scheduling system can dynamically calculate the priority of each task to ensure that urgent tasks are processed in a timely manner.

[0046] The data lineage correlation degree calculates the data dependency strength between tasks based on the metadata lineage graph. The data lineage graph represents the dependency relationship between tasks and data, calculates the data transmission order between tasks, and then optimizes the task sharding and merging strategies. Based on the data lineage correlation degree γ between tasks, highly dependent tasks can be merged into atomic execution units to avoid the waste of computing resources caused by excessive segmentation. At the same time, it optimizes the execution order of tasks and improves computing efficiency.

[0047] The resource elasticity factor dynamically adjusts the degree of parallelism based on real-time monitoring of CPU / memory usage. The value of β is automatically adjusted by monitoring resource availability. When resource utilization is low, the degree of parallelism is increased to improve computing efficiency. When resource utilization is high, the degree of parallelism is reduced to avoid system overload. The formula is as follows:

[0048] β = 1 / (1 + e^(-k*(1-resource availability))).

[0049] In this formula, resource availability is a value between 0 and 1, representing the proportion of currently available resources (for example, remaining CPU or memory capacity). The constant k controls the sensitivity of the resource elasticity factor and is the base of the exponential function e, describing the relationship between resource availability and the resource elasticity factor. A larger value of k indicates a more sensitive system to resource changes. This formula allows the system to adjust the degree of parallelism of tasks based on the current availability of resources.

[0050] Step 2: The dynamic metadata identification module identifies metadata identifier changes and triggers DAG task scheduling policy updates to achieve dynamic optimization. Specifically, this is usually triggered by user operations (for example, task priority, resource usage, data lineage, etc.).

[0051] An improved genetic algorithm is used to optimize the scheduling strategy. The genetic algorithm is an optimization algorithm that simulates natural selection and genetic mechanisms. It seeks the global optimal solution through continuous iterative evolution. The specific steps are as follows:

[0052] S1: Gene encoding: The execution order of DAG nodes and resource allocation schemes are encoded into chromosomes. In genetic algorithms, task scheduling schemes are converted into chromosomes (i.e., genes). Genes represent the order of task execution and resource allocation strategies. The execution order of DAG nodes and resource allocation schemes (such as CPU, memory, and GPU resource allocation) are encoded into chromosomes, allowing genetic algorithms to process and optimize these schemes. Each chromosome contains different elements of task scheduling, such as the order of task execution, resource allocation, and other scheduling-related parameters.

[0053] S2: Fitness function, used to evaluate the quality of the current chromosome, so as to adjust the task scheduling plan to minimize delays, maximize resource utilization and task completion, and thus obtain the optimal scheduling strategy. Its formula is:

[0054] F = α*(1-delay rate) + β*resource utilization + γ*task completion;

[0055] S3: Iterative optimization. A new generation of scheduling solutions is generated every N minutes based on the latest metadata status, where N>1. The genetic algorithm is an iterative optimization process. In each generation, chromosomes with higher fitness are selected according to the fitness function, and operations such as crossover and mutation are performed on them to generate a new generation of scheduling solutions.

[0056] Dynamic optimization is the process of flexibly adjusting tasks based on the current system status each time a task is scheduled. This can improve the efficiency of scheduling strategies, reduce resource waste, and increase the reliability of task execution. The optimization includes the following:

[0057] A1. Task Slicing and Reorganization: The system analyzes the DAG structure of operators and merges related tasks with high γ values into atomic execution units. Specifically, based on the task structure of the DAG, the system analyzes the operators and merges highly related tasks (i.e., those with high data lineage relationship values γ) into atomic execution units. This reduces data transmission and scheduling overhead between tasks, while improving the system's parallel execution efficiency.

[0058] A2. Preemptive resource allocation: When α > 0.8, the GPU resource reservation mechanism is triggered, pausing thread allocation for low-priority tasks. This ensures that high-priority tasks receive sufficient computing resources under limited resources, thereby improving the execution efficiency of critical tasks.

[0059] A3. Incremental Checkpoint Mechanism: This mechanism periodically saves intermediate state data during execution. When a task fails, it simply rolls back to the most recent checkpoint, eliminating the need to restart the entire task from the beginning. This mechanism effectively reduces restart time in the event of a task failure, improving system reliability and recovery speed.

[0060] Step 3: Synchronize updates to multiple data source connectors and link them with external data source mappings. Specifically, after completing the dynamic optimization of the DAG task scheduling strategy, synchronize updates to multiple data source connectors and ensure that these updates can be linked and mapped with external data sources.

[0061] As shown in the following table:

[0062]

[0063]

[0064] As shown in the above table, the present invention realizes dynamic adjustment of task scheduling strategy in seconds (strategy takes effect in <500ms) through the α / β / γ three-dimensional control model.

[0065] Resource utilization optimization: The elasticity factor β reduces the CPU idle rate to below 8% (benchmark dataset: TPC-DS10TB).

[0066] System self-healing capability: The incremental checkpoint mechanism reduces data recalculation during fault recovery by 92%.

[0067] Implementation example:

[0068]

[0069] Specifically, during implementation, the system first checks whether the priority in the metadata is "high." If the task is high-priority, the system calls the allocate_gpu_resources() method to allocate GPU resources. If the task's priority is not "high," the system checks the task's data size. If the data size is greater than 1TB, the system calls the split_task_into_subtasks() method to split the task into multiple subtasks for parallel processing. This breaks down large data processing tasks into smaller parts, improving processing efficiency and accelerating task execution.

[0070] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the above-described embodiments. The above-described embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and improvements may be made to the present invention without departing from the framework and scope of application of the present invention. Such changes and improvements are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A data weaving method based on dynamic metadata identification and DAG optimization, characterized by: The following steps are involved: Step 1: Establish a DAG task scheduling strategy to achieve dynamic optimization; Step 2: Identify the metadata status and trigger the DAG task scheduling policy update when a change occurs; Step 3: Link the dynamic optimization update results of step 2 with the external data source mapping.

2. The data weaving method based on dynamic metadata identification and DAG optimization according to claim 1 is characterized in that: In step 1, the specific method of establishing the DAG task scheduling strategy is as follows: A multidimensional scheduling model based on metadata state vector is established, and dynamic optimization is achieved through priority weight coefficient α, resource elasticity factor β and data lineage correlation γ.

3. The data weaving method based on dynamic metadata identification and DAG optimization according to claim 1 is characterized in that: In step 2, an improved genetic algorithm is used to optimize the scheduling strategy. The specific steps are as follows: S1: Gene encoding, which encodes the DAG node execution order and resource allocation scheme into chromosomes; S2: Fitness function, used to evaluate the quality of the current chromosome. Its formula is: F = α*(1-delay rate) + β*resource utilization + γ*task completion; S3: Iterative optimization, generating a new generation of scheduling solutions based on the latest metadata status every N minutes, where N>1.

4. The data weaving method based on dynamic metadata identification and DAG optimization according to claim 1 is characterized in that: The dynamic optimization includes the following optimization operations: A1. Task sharding and reorganization: Perform DAG graph structure analysis on operators and merge related tasks with high γ values into atomic execution units. A2, resource preemptive allocation: When α>0.8, the GPU resource reservation mechanism is triggered and thread allocation of low-priority tasks is suspended; A3. Incremental checkpoint mechanism: Periodically saves intermediate state data during execution. When a task fails, only the most recent checkpoint needs to be rolled back.

5. The data weaving method based on dynamic metadata identification and DAG optimization according to claim 1 is characterized in that: The priority weight coefficient automatically calculates the task priority based on the urgency marked in the metadata tag. The data lineage correlation calculates the data dependency strength between tasks based on the metadata lineage map and optimizes the task sharding and merging strategy. The resource elasticity factor dynamically adjusts the parallelism based on the real-time monitored CPU / memory usage. The formula is as follows: β = 1 / (1 + e^(-k*(1-resource availability))).

6. The data weaving system based on dynamic metadata identification and DAG optimization includes a metadata management layer, a data processing layer, and an execution layer, and is characterized by: The metadata management layer synchronizes the metadata state to the data processing layer, and the data processing layer sends the resource scheduling instructions to the execution layer, and the execution layer executes the instructions. The metadata management layer includes: Dynamic metadata identification module, used to expand the metadata storage structure through the JDBC interface, set up business tags, priority tags, data lineage relationship fields, and identify metadata identification status; The data processing layer includes: Adaptive DAG optimization engine module, used to establish DAG task scheduling strategies and achieve dynamic optimization; The dynamic task allocator module is used to trigger the update of DAG task scheduling strategy, adjust task scheduling and reallocate resources. The execution layer includes: The multi-source data connector module is used to link the dynamic optimization update results with the external data source mapping.

7. The data weaving system based on dynamic metadata identification and DAG optimization according to claim 6 is characterized in that: The metadata management layer also includes a tag storage engine module for fast reading and writing of metadata tags.

8. The data weaving system based on dynamic metadata identification and DAG optimization according to claim 7 is characterized in that: The execution layer also includes a data computing node cluster module for actual task calculation and data processing.

Images