Data resource scheduling method and related product

Abstract

The invention discloses a data resource scheduling method and related products, which can be applied to the technical field of big data storage, and comprises the following steps: acquiring a forward feedback signal; in combination with a preset data consanguinity heterogeneous topological graph, adopting a reverse conduction algorithm to reversely propagate the forward feedback signal to a data storage node on which the terminal service node depends, and determining a global utility weight corresponding to the data storage node based on a multi-source contribution attribution model; comparing the global utility weight with a preset resource scheduling threshold value, and correspondingly generating a resource scheduling instruction which comprises a physical storage medium migration instruction and / or a calculation task priority adjustment instruction; and dynamically scheduling the data object corresponding to the data storage node based on the resource scheduling instruction. Therefore, the data value corresponding to each data storage node is fully analyzed by introducing a reverse conduction algorithm and a multi-source contribution attribution model, the importance of bottom data is reversely deduced from a terminal service operation result, and the scheduling rationality of data resources is improved.

Description

A method for scheduling data resources and related products Technical Field

[0001] This application relates to the field of big data storage technology, and in particular to a data resource scheduling method and related products. Background Technology

[0002] In existing big data platforms, data lifecycle management (DLM) and hot / cold data tiering typically rely on the physical attributes of the data or access statistics, such as file size, creation time, last access time, and simple access frequency. The system uses these metrics to decide whether to retain data in high-performance memory or solid-state drives, or migrate it to lower-cost, lower-performance hard disk drives or cold storage.

[0003] However, this data resource scheduling method based on physical attributes or access statistics has obvious limitations. The system often has difficulty perceiving the logical importance of data. Some data that is small in size or has a low number of accesses but is actually very important is easily misjudged as "cold data" and evicted or compressed from the high-speed storage layer, which leads to the problem of low rationality in data resource scheduling.

[0004] Therefore, how to improve the rationality of data resource scheduling is an urgent problem that needs to be solved by those skilled in the art. Summary of the Invention

[0005] To address the aforementioned issues, this application provides a data resource scheduling method and related products. By introducing a reverse propagation algorithm and a multi-source contribution attribution model, the data value corresponding to each data storage node is fully analyzed, and the importance of the underlying data is inferred from the terminal service operation results, thereby improving the rationality of data resource scheduling.

[0006] In a first aspect, embodiments of this application provide a data resource scheduling method, comprising: acquiring a positive feedback signal from a terminal service node; combining a preset data lineage heterogeneous topology map, using a reverse propagation algorithm to propagate the positive feedback signal in reverse to the data storage node on which the terminal service node depends, and determining the global utility weight corresponding to the data storage node based on a multi-source contribution attribution model; the preset data lineage heterogeneous topology map includes data dependencies between data storage nodes, processing logic nodes, and terminal service nodes; comparing the global utility weight with a preset resource scheduling threshold, and generating a resource scheduling instruction based on the comparison result; the resource scheduling instruction includes a physical storage medium migration instruction and / or a computation task priority adjustment instruction; and dynamically scheduling the data objects corresponding to the data storage node based on the resource scheduling instruction.

[0007] Optionally, the scheduling method further includes: obtaining task scheduling logs and data processing scripts in a distributed data warehouse; performing abstract syntax tree parsing on the data processing scripts and identifying data dependencies between data storage nodes, processing logic nodes, and terminal service nodes; and constructing a preset data lineage heterogeneous topology graph containing the data storage nodes, the processing logic nodes, and the terminal service nodes based on the data dependencies.

[0008] Optionally, obtaining the positive feedback signal of the terminal service node includes: obtaining the request latency and request frequency of the terminal service node; and determining the positive feedback signal corresponding to the terminal service node based on the request latency and the request frequency using a preset feedback calculation model.

[0009] Optionally, the expression corresponding to the preset feedback calculation model is: In the formula As a positive feedback signal, For the requested delay, For request frequency, As the first weighting coefficient, This is the second weighting coefficient.

[0010] Optionally, determining the global utility weight corresponding to the data storage node based on the multi-source contribution attribution model includes: determining a target terminal service node; the target terminal service node is any one of the terminal service nodes; constructing a target set corresponding to the target terminal service node; the target set includes target data storage nodes that have a dependency relationship with the target terminal service node; determining the target Shapley value contribution of the target data storage node to the target terminal service node based on the complete permutation order of the target set; and determining the global utility weight corresponding to the data storage node based on the target Shapley value contribution.

[0011] Optionally, the scheduling method further includes: when the size of the target set is greater than a preset size threshold, determining the sampling order of the target set based on the Monte Carlo sampling method; determining the estimated Shapley value contribution of the target data storage node to the target terminal service node based on the sampling order of the target set; and determining the global utility weight corresponding to the data storage node based on the estimated Shapley value contribution.

[0012] Optionally, comparing the global utility weight with a preset resource scheduling threshold and generating a resource scheduling instruction based on the comparison result includes: if the global utility weight is greater than a first resource scheduling threshold, generating a first physical storage medium migration instruction; the first physical storage medium migration instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be migrated to the first storage medium; if the global utility weight is less than a second resource scheduling threshold, generating a second physical storage medium migration instruction; the second physical storage medium migration instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be migrated to the second storage medium; the first resource scheduling threshold is greater than the second resource scheduling threshold, and the read / write rate of the first storage medium is greater than the read / write rate of the second storage medium.

[0013] The step of comparing the global utility weight with a preset resource scheduling threshold and generating a resource scheduling instruction based on the comparison result includes: if the global utility weight is greater than a third resource scheduling threshold, generating a first computing task priority adjustment instruction; the first computing task priority adjustment instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be processed using a first priority queue and / or a first resource quota; if the global utility weight is less than a fourth resource scheduling threshold, generating a second computing task priority adjustment instruction; the second computing task priority adjustment instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be processed using a second priority queue and / or a second resource quota; the third resource scheduling threshold is greater than the fourth resource scheduling threshold, the priority of the first priority queue is higher than the priority of the second priority queue, and the first resource quota is greater than the second resource quota.

[0014] The scheduling method further includes: monitoring metadata change events in the distributed data warehouse; and incrementally updating the preset data lineage heterogeneous topology graph based on the change events.

[0015] Secondly, embodiments of this application provide a data resource scheduling device, comprising: a signal acquisition module for acquiring positive feedback signals from terminal service nodes; a weight calculation module for combining a preset data lineage heterogeneous topology map, using a reverse propagation algorithm to propagate the positive feedback signals in reverse to the data storage nodes on which the terminal service nodes depend, and determining the global utility weight corresponding to the data storage nodes based on a multi-source contribution attribution model; the preset data lineage heterogeneous topology map includes data dependency relationships between data storage nodes, processing logic nodes, and terminal service nodes; a resource control module for comparing the global utility weight with a preset resource scheduling threshold, and generating resource scheduling instructions based on the comparison results; the resource scheduling instructions include physical storage medium migration instructions and / or computation task priority adjustment instructions; and a scheduling execution module for dynamically scheduling the data objects corresponding to the data storage nodes based on the resource scheduling instructions.

[0016] As can be seen from the above technical solutions, compared with the prior art, this application has the following advantages: The data resource scheduling method provided in this application first obtains the positive feedback signal from the terminal service node. Then, combined with a preset data lineage heterogeneous topology map, a reverse propagation algorithm is used to propagate the positive feedback signal back to the data storage nodes on which the terminal service node depends, and a multi-source contribution attribution model is used to determine the global utility weight corresponding to the data storage node. The preset data lineage heterogeneous topology map includes the data dependencies between data storage nodes, processing logic nodes, and terminal service nodes. Finally, the global utility weight is compared with a preset resource scheduling threshold, and a resource scheduling instruction is generated based on the comparison result. The data objects corresponding to the data storage nodes are then dynamically scheduled based on the resource scheduling instruction. The resource scheduling instruction includes physical storage media migration instructions and / or computation task priority adjustment instructions. Thus, by introducing a reverse propagation algorithm and a multi-source contribution attribution model to fully analyze the data value corresponding to each data storage node, the importance of the underlying data is inferred from the terminal service operation results, improving the rationality of data resource scheduling. Attached Figure Description

[0017] Figure 1 is a flowchart of a data resource scheduling method provided in an embodiment of this application; Figure 2 is a schematic diagram of a preset data lineage heterogeneous topology and its reverse transmission provided in an embodiment of this application; Figure 3 is a schematic diagram of the allocation logic of Shapley value contribution provided in an embodiment of this application; Figure 4 is a structural schematic diagram of a data resource scheduling device provided in an embodiment of this application. Detailed Implementation

[0018] As mentioned earlier, existing data resource scheduling methods suffer from low scheduling rationality. Specifically, these methods typically rely on the physical attributes or access statistics of the data, such as file size, creation time, last access time, and simple access frequency. However, these methods based on physical attributes or access statistics often fail to perceive the "logical importance" of the data. For example, a small configuration table or a dictionary table with a low update frequency may directly affect the availability of a core transaction system or real-time recommendation service. In existing scheduling methods, such data is often misjudged as "cold data" due to its small size and low access frequency, and thus evicted from high-speed storage or compressed and archived. When core business processes suddenly access this data during peak periods, it needs to be reloaded from low-speed media, resulting in significant access latency and even service jitter. Therefore, existing data resource scheduling methods suffer from low scheduling rationality.

[0019] To address the aforementioned issues, this application provides a data resource scheduling method. First, positive feedback signals from terminal service nodes are obtained. Then, based on a preset heterogeneous data lineage topology, a reverse propagation algorithm is used to propagate the positive feedback signals back to the data storage nodes on which the terminal service nodes depend. A multi-source contribution attribution model is used to determine the global utility weight corresponding to each data storage node. The preset heterogeneous data lineage topology includes data dependencies between data storage nodes, processing logic nodes, and terminal service nodes. Finally, the global utility weights are compared with a preset resource scheduling threshold. Based on the comparison result, resource scheduling instructions are generated, and the data objects corresponding to the data storage nodes are dynamically scheduled based on these instructions. The resource scheduling instructions include physical storage media migration instructions and / or computation task priority adjustment instructions.

[0020] In this way, by introducing the reverse propagation algorithm and the multi-source contribution attribution model, the data value corresponding to each data storage node is fully analyzed, and the importance of the underlying data is inferred from the terminal service operation results, thereby improving the rationality of data resource scheduling.

[0021] It should be noted that the data resource scheduling method and related products provided in this application can be applied to the field of big data storage technology. The above are merely examples and do not limit the application areas of the data resource scheduling method and related products provided in this application.

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0023] Figure 1 is a flowchart of a data resource scheduling method provided in an embodiment of this application. Referring to Figure 1, the data resource scheduling method provided in this embodiment may include: S101: Obtaining a positive feedback signal from a terminal service node.

[0024] In practical applications, the first step is to obtain the positive feedback signals from the terminal service nodes. There are typically multiple terminal service nodes (at least two), and the positive feedback signals from each node need to be acquired. These positive feedback signals simultaneously reflect the service's performance requirements (low latency) and business pressure (high queries per second), and the importance of the business is reflected through service level weights.

[0025] Furthermore, since the methods for obtaining positive feedback signals are not entirely the same, this application embodiment can describe one possible method of obtaining them.

[0026] In one scenario, S101: Obtaining a positive feedback signal from a terminal service node may specifically include: obtaining the request latency and request frequency of the terminal service node; and determining the positive feedback signal corresponding to the terminal service node based on the request latency and request frequency using a preset feedback calculation model.

[0027] Furthermore, the expression corresponding to the preset feedback calculation model is: In the formula As a positive feedback signal, For the requested delay, For request frequency, As the first weighting coefficient, This is the second weighting coefficient.

[0028] In practical applications, the positive feedback signals of terminal service nodes are related to their key service quality indicators, namely request latency, the number of requests per unit time (request frequency), and service level information (e.g., core online transactions, important online services, offline reports, etc.). The system continuously monitors the service quality indicators of each terminal service node during runtime. When acquiring positive feedback signals from terminal service nodes, a preset feedback calculation model is introduced, and the positive feedback signal for each terminal service node is calculated based on the expression corresponding to the preset feedback calculation model. Specifically, the expression corresponding to this preset feedback calculation model is as follows: In the formula As a positive feedback signal, The request latency (average response latency within a preset monitoring time window). This refers to the request frequency (the total number of requests within the same time window). As the first weighting coefficient, This is the second weighting coefficient. and The coefficient value is non-negative, and the coefficient value corresponding to core transaction services is usually higher than that of offline reporting services, in order to ensure that core transaction services generate a larger positive feedback signal F under the same L and Q conditions.

[0029] Furthermore, since there are different ways to construct a preset data lineage heterogeneous topology map, this application embodiment can describe one possible construction method.

[0030] In one instance, the scheduling method further includes: acquiring task scheduling logs and data processing scripts from a distributed data warehouse; performing abstract syntax tree parsing on the data processing scripts and identifying data dependencies between data storage nodes, processing logic nodes, and terminal service nodes; and constructing a preset data lineage heterogeneous topology graph containing the data storage nodes, the processing logic nodes, and the terminal service nodes based on the data dependencies.

[0031] In practical applications, the system first retrieves task scheduling logs from the task scheduling system and corresponding data processing scripts from the code repository or script management system. Then, it performs abstract syntax tree parsing on each data processing script to identify the input data source and output data target (i.e., identifying the data dependencies between data storage nodes, processing logic nodes, and terminal service nodes). For example, if the system identifies data tables A and B as input data storage nodes and data table C as output data storage node, then directed edges from nodes A and B to node C are established in the topology graph. Furthermore, it identifies which terminal service nodes directly read which data storage nodes. For instance, if it identifies that "real-time recommendation engine service S" reads table C through processing logic node X, then directed edges are established in the topology graph from data storage node C to processing logic node X, and then from processing logic node X to terminal service node S. In this way, the nodes are connected by directed edges representing data flow, forming a directed graph structure from upstream data storage nodes, through processing logic nodes, to downstream terminal service nodes, thus creating a pre-defined heterogeneous topology graph with a data lineage. Furthermore, by eliminating loops or breaking loops according to time dependencies, the graph structure can satisfy the constraints of a Directed Acyclic Graph (DAG), thereby facilitating subsequent reverse propagation computation.

[0032] S102: Combining the preset data lineage heterogeneous topology graph, the positive feedback signal is propagated in reverse to the data storage node on which the terminal service node depends using the reverse propagation algorithm, and the global utility weight corresponding to the data storage node is determined based on the multi-source contribution attribution model; the preset data lineage heterogeneous topology graph contains the data dependency relationships between data storage nodes, processing logic nodes and terminal service nodes.

[0033] In practical applications, a pre-defined heterogeneous data lineage topology graph contains data dependencies between data storage nodes, processing logic nodes, and terminal service nodes. Based on this graph, a reverse propagation algorithm can be executed. For each terminal service node and its corresponding positive feedback signal, the signal is first allocated to the data storage node or intermediate processing logic node that directly provides the data, and then propagates upstream layer by layer along the directed edges of the topology. Figure 2 is a schematic diagram of a pre-defined heterogeneous data lineage topology graph and its reverse propagation provided in an embodiment of this application. Referring to Figure 2, taking the directed edges on the pre-defined heterogeneous data lineage topology graph, including data storage nodes (tables A and B) pointing to processing logic nodes (merged calculation), and merged calculation pointing to terminal service nodes (recommendation service S), as an example, after the reverse propagation algorithm, for recommendation service S, its corresponding positive feedback signal is first allocated to the processing logic node that provides the data, and then propagates along the directed edges of the topology to tables A and B. Tables A and B are the data storage nodes that recommendation service S depends on. Furthermore, during the reverse propagation process, the positive feedback signal can be initially weighted and attenuated based on path length or dependency strength before being input into the multi-source contribution attribution model. Further, the contributions of Table A and Table B in the recommendation service S need to be divided. Specifically, this application's embodiment introduces a multi-source contribution attribution model, which initially distributes the positive feedback signal corresponding to the terminal service node based on path weight or edge weight, for example, by weighting according to path length, access frequency of each node in the processing link, etc., and then further refines the contribution calculation by combining the Shapley value, ultimately obtaining the global utility weight corresponding to each data storage node.

[0034] Furthermore, since there are different ways to determine the global utility weight, this application embodiment can describe one possible determination method.

[0035] In one scenario, determining the global utility weight corresponding to the data storage node based on a multi-source contribution attribution model includes: determining a target terminal service node; the target terminal service node being any one of the terminal service nodes; constructing a target set corresponding to the target terminal service node; the target set containing target data storage nodes that have a dependency relationship with the target terminal service node; determining the target Shapley value contribution of the target data storage node to the target terminal service node based on the complete permutation order of the target set; and determining the global utility weight corresponding to the data storage node based on the target Shapley value contribution.

[0036] In practical applications, to address the contribution sharing problem when multiple upstream data storage nodes jointly support terminal services, this application introduces the Shapley value algorithm model from the multi-source contribution attribution model. Based on the marginal contribution of each participating node under different arrangements, the contribution value relative to each terminal service node is calculated. The contribution values ​​from multiple terminal service nodes are then normalized and summed to obtain the global utility weight of the data storage node. Taking any one of the terminal service nodes (the target terminal service node) as an example, suppose the target terminal service node S depends on the target data storage nodes (table A and table B). The participant set (target set) is denoted as N = {A, B}. Then, a revenue function v(S) is defined to describe the revenue of the target terminal service node S when only a subset of participants are available. This revenue can be selected as an indicator such as service availability, throughput, or latency compliance rate. Where v({A}) represents the revenue of the target terminal service node S when only table A is available; v({B}) represents the revenue when only table B is available; v({A, B}) represents the revenue when both A and B are available simultaneously; v( The value represents the benefit when neither of the two is available, and is typically considered to be 0 or a very low value. In the standard Shapley value definition, all permutations of the participant set N are iterated through, calculating the incremental benefit each participant brings when "joining the alliance" in different permutations, and averaging all permutation results as the participant's (target) Shapley value contribution. For example, if calculations or estimations show that the service performance of S decreases by approximately 80% when only table A is missing, and by approximately 20% when only table B is missing, then it can be approximated that A contributes approximately 80% to S and B contributes approximately 20%. If an intermediate node C exists, the feedback value passed from the target terminal service node S to the intermediate node C is then distributed to A and B proportionally after passing back to C. Finally, after calculating the total (target) Shapley value contribution for each terminal service node corresponding to the target data storage node, these contributions are normalized and summed to obtain the global utility weight of the target data storage node. Based on this operation, all data storage nodes are traversed, and the global utility weight corresponding to each data storage node is obtained.

[0037] Furthermore, since the methods for determining the global utility weights differ depending on the size of the target set, this application embodiment can describe one possible determination method.

[0038] In one case, the scheduling method further includes: when the size of the target set is greater than a preset size threshold, determining the sampling order of the target set based on the Monte Carlo sampling method; determining the estimated Shapley value contribution of the target data storage node to the target terminal service node based on the sampling order of the target set; and determining the global utility weight corresponding to the data storage node based on the estimated Shapley value contribution.

[0039] In practical applications, for complex scenarios involving multiple upstream nodes, the number of permutations of the participant set increases exponentially with the number of participants (target data storage nodes). To reduce computational complexity, this application introduces a preset size threshold and Monte Carlo sampling. When the size of the target set (number of participants) exceeds the preset size threshold, Monte Carlo sampling is used for approximate calculation. Specifically, firstly, several permutation order samples are randomly generated for the target set N, i.e., the sampled permutation order corresponding to the target set is determined, rather than all permutation orders. Then, the marginal contribution of each participant is calculated for each sampled permutation order, and the average of the marginal contributions in all samples is taken as the estimated Shapley value contribution of each participant. Finally, after calculating all (estimated) Shapley value contributions corresponding to the target data storage node for each terminal service node, these contribution values ​​are normalized and accumulated to obtain the global utility weight of the target data storage node. Based on this operation, all data storage nodes are traversed to obtain the global utility weight corresponding to each data storage node.

[0040] S103: Compare the global utility weight with the preset resource scheduling threshold, and generate a resource scheduling instruction based on the comparison result; the resource scheduling instruction includes a physical storage medium migration instruction and / or a computing task priority adjustment instruction.

[0041] In practical applications, a preset resource scheduling threshold can be introduced. The global utility weight of each data storage node, calculated based on the threshold, is compared to determine the importance of the data object and generate corresponding resource scheduling instructions. Resource scheduling comprises two dimensions: storage media migration and computational task priority adjustment. For high-value data, not only must storage be fast, but the extraction, transformation, and loading (ETL) tasks that generate it must also run quickly. Therefore, this application introduces two scheduling instructions: a physical storage media migration instruction and a computational task priority adjustment instruction. These two instructions work together to achieve data resource scheduling.

[0042] Furthermore, the methods for generating resource scheduling instructions are not entirely the same, so this application embodiment can describe one possible generation method.

[0043] In one scenario, S103: The global utility weight is compared with a preset resource scheduling threshold, and a resource scheduling instruction is generated based on the comparison result. Specifically, this may include: if the global utility weight is greater than a first resource scheduling threshold, a first physical storage medium migration instruction is generated; the first physical storage medium migration instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be migrated to the first storage medium; if the global utility weight is less than a second resource scheduling threshold, a second physical storage medium migration instruction is generated; the second physical storage medium migration instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be migrated to the second storage medium; the first resource scheduling threshold is greater than the second resource scheduling threshold, and the read / write rate of the first storage medium is greater than the read / write rate of the second storage medium.

[0044] In practical applications, regarding storage media migration, two different preset resource scheduling thresholds can be set: a first resource scheduling threshold (high threshold) and a second resource scheduling threshold (low threshold). When the global utility weight of a data storage node is greater than the first resource scheduling threshold, the data object corresponding to that data storage node is considered to have high business value for multiple service scenarios. Therefore, a first physical storage media migration instruction can be generated for that data storage node to instruct the data object of that data storage node to be migrated to the first storage medium. The first storage medium is a solid-state drive (SSD) or a dedicated cache layer with high read / write speeds. Conversely, when the global utility weight of a data storage node is less than the second resource scheduling threshold, the data object corresponding to that data storage node is considered to have low value in the current stage when used by multiple services. Therefore, a second physical storage media migration instruction can be generated for that data storage node to instruct the data object of that data storage node to be migrated to the second storage medium. Furthermore, for this type of data object, a high compression ratio data format can be selected for storage to save storage space. The second storage medium is a mechanical hard drive (HDD) or a cold storage system with lower read / write speeds. Understandably, this application introduces a hysteresis strategy, which forms a buffer zone between the first and second resource scheduling thresholds by setting a first resource scheduling threshold greater than a second resource scheduling threshold. When the global utility weight is between the first and second resource scheduling thresholds, the storage status of the data resources is maintained without migration, that is, the current storage medium of the data object remains unchanged, thereby preventing frequent "ping-pong" migration of data between hot and cold media.

[0045] Figure 3 is a schematic diagram of the allocation logic of Shapley value contribution provided in an embodiment of this application. Referring to Figure 3, if the first resource scheduling threshold is set to 0.7 and the second resource scheduling threshold is set to 0.3, when the marginal contributions of data storage node A (core configuration table) and data storage node B (auxiliary log table) are high and low respectively, and their global utility weights are 0.8 and 0.2 respectively, then based on the data resource scheduling method provided in this embodiment, the core configuration table will be stored on a solid-state drive or a dedicated cache layer, while the auxiliary log table will be stored on a mechanical hard drive or a cold storage system.

[0046] Furthermore, since the methods for generating resource scheduling instructions are not entirely the same, this application embodiment can describe one possible generation method.

[0047] In one scenario, S103: The global utility weight is compared with a preset resource scheduling threshold, and a resource scheduling instruction is generated based on the comparison result. Specifically, this may further include: if the global utility weight is greater than a third resource scheduling threshold, a first computational task priority adjustment instruction is generated; the first computational task priority adjustment instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be processed using a first priority queue and / or a first resource quota; if the global utility weight is less than a fourth resource scheduling threshold, a second computational task priority adjustment instruction is generated; the second computational task priority adjustment instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be processed using a second priority queue and / or a second resource quota; the third resource scheduling threshold is greater than the fourth resource scheduling threshold, the priority of the first priority queue is higher than the priority of the second priority queue, and the first resource quota is greater than the second resource quota.

[0048] In practical applications, if we explain this from the perspective of adjusting the priority of computing tasks, the system can also adjust the scheduling priority and resource allocation of data processing tasks used to generate high-weight data storage nodes at the computing cluster level. Specifically, this application embodiment can also introduce computing priority thresholds, namely, a third resource scheduling threshold and a fourth resource scheduling threshold. When the global utility weight of a data storage node is greater than the third resource scheduling threshold, it is considered that the data object corresponding to the data storage node has high business value for multiple current service scenarios. Therefore, a first computing task priority adjustment instruction can be generated for the data storage node, which instructs the system to identify the data processing task that generates the data object corresponding to the data storage node and, through interaction with the task scheduling system, allocate the task to the first priority (high priority) queue and / or allocate a first resource quota (more CPU, memory, and other computing resource quotas) to it. Conversely, for data storage nodes whose global utility weight is significantly lower than the threshold and are mainly used for low-priority analysis or reporting, their corresponding data processing tasks can be adjusted to a low-priority queue or have their computing resource quotas limited, so as to free up more cluster resources to ensure the production of critical data and the stability of core services. Specifically, a fourth resource scheduling threshold can be introduced. When the global utility weight is lower than the fourth resource scheduling threshold, the data object corresponding to the data storage node is considered to have low value when used by multiple services in the current stage. Therefore, a second computing task priority adjustment instruction can be generated for this data storage node. This instruction instructs the system to identify the data processing task that generated the data object corresponding to the data storage node and, through interaction with the task scheduling system, allocate the task to the second priority (low priority) queue and / or allocate a second resource quota (less CPU, memory, and other computing resource quotas) to it. The third resource scheduling threshold is greater than the fourth resource scheduling threshold, the processing priority of the first priority queue is higher than the processing priority of the second priority queue, and the first resource quota is greater than the second resource quota. It is understood that this application also introduces a hysteresis strategy in the dimension of computing task priority adjustment, by setting the third resource scheduling threshold to be greater than the fourth resource scheduling threshold to form a buffer between the third and fourth resource scheduling thresholds. When the global utility weight is between the third and fourth resource scheduling thresholds, the current storage status of data resources is maintained without queueing and / or resource quota adjustments, i.e., the current computing task priority is maintained, thereby preventing data from frequently migrating between hot and cold media.

[0049] Furthermore, since the incremental update methods of the topology graph are not entirely the same, the embodiments of this application can describe one possible update method.

[0050] In one instance, the method further includes: monitoring metadata change events in the distributed data warehouse; and incrementally updating the preset data lineage heterogeneous topology based on the change events.

[0051] In practical applications, the system can continuously monitor changes to metadata and task scheduling in the distributed data warehouse. When a new data processing task is released or the logic of an existing task is changed, the system incrementally updates the preset heterogeneous topology graph of data lineage and recalculates the global utility weights of the affected data storage nodes based on the updated topology, thereby updating the resource scheduling strategy online. Specifically, when a new task is added, its script is first parsed and the abstract syntax tree is updated to identify new data storage nodes, processing logic nodes, and potentially new terminal service nodes. Then, relevant nodes and directed edges are incrementally added or modified in the preset heterogeneous topology graph of data lineage. Finally, the affected data storage nodes and terminal service nodes are marked, and local reverse propagation and Shapley value calculation are performed on the marked nodes to update the global utility weights of the corresponding data storage nodes.

[0052] S104: Dynamically schedule the data objects corresponding to the data storage node based on the resource scheduling instruction.

[0053] In practical applications, after determining the resource scheduling instructions corresponding to each data storage node, the data resources of each data storage node can be dynamically scheduled based on these instructions.

[0054] In summary, this application first obtains the positive feedback signal from the terminal service node. Then, combining a pre-defined heterogeneous data lineage topology, a reverse propagation algorithm is used to propagate the positive feedback signal back to the data storage nodes on which the terminal service node depends, and a multi-source contribution attribution model is used to determine the global utility weight corresponding to the data storage node. The pre-defined heterogeneous data lineage topology includes the data dependencies between data storage nodes, processing logic nodes, and terminal service nodes. Finally, the global utility weight is compared with a pre-defined resource scheduling threshold, and resource scheduling instructions are generated based on the comparison results. Dynamic scheduling of data objects corresponding to the data storage nodes is then performed based on these resource scheduling instructions. These resource scheduling instructions include physical storage media migration instructions and / or computation task priority adjustment instructions. Thus, by introducing a reverse propagation algorithm and a multi-source contribution attribution model to fully analyze the data value corresponding to each data storage node, the importance of underlying data is inferred from the terminal service operation results, improving the rationality of data resource scheduling.

[0055] Figure 4 is a schematic diagram of a data resource scheduling device provided in an embodiment of this application. Referring to Figure 4, the data resource scheduling device 400 includes: a signal acquisition module 401, used to acquire positive feedback signals from terminal service nodes; a weight calculation module 402, used to combine a preset data lineage heterogeneous topology map, use a reverse propagation algorithm to propagate the positive feedback signals back to the data storage nodes on which the terminal service nodes depend, and determine the global utility weight corresponding to the data storage nodes based on a multi-source contribution attribution model; the preset data lineage heterogeneous topology map includes data dependencies between data storage nodes, processing logic nodes, and terminal service nodes; a resource control module 403, used to compare the global utility weight with a preset resource scheduling threshold, and generate resource scheduling instructions based on the comparison results; the resource scheduling instructions include physical storage medium migration instructions and / or computation task priority adjustment instructions; and a scheduling execution module 404, used to dynamically schedule the data objects corresponding to the data storage nodes based on the resource scheduling instructions.

[0056] As one implementation method, the data resource scheduling device 400 further includes: a graph construction module; the graph construction module is used to obtain task scheduling logs and data processing scripts in the distributed data warehouse; perform abstract syntax tree parsing on the data processing scripts and identify the data dependency relationships between data storage nodes, processing logic nodes and terminal service nodes; and construct a preset data lineage heterogeneous topology graph containing the data storage nodes, the processing logic nodes and the terminal service nodes based on the data dependency relationships.

[0057] As one implementation method, regarding how to obtain the positive feedback signal of the terminal service node, the signal acquisition module 401 is specifically used to: acquire the request latency and request frequency of the terminal service node; and determine the positive feedback signal corresponding to the terminal service node based on the request latency and the request frequency using a preset feedback calculation model.

[0058] Furthermore, the expression corresponding to the preset feedback calculation model is: In the formula As a positive feedback signal, For the requested delay, For request frequency, As the first weighting coefficient, This is the second weighting coefficient.

[0059] As one implementation method, regarding how to determine the global utility weight, the aforementioned weight calculation module 402 is specifically used for: determining a target terminal service node; the target terminal service node being any one of the terminal service nodes; constructing a target set corresponding to the target terminal service node; the target set containing target data storage nodes that have a dependency relationship with the target terminal service node; determining the target Shapley value contribution of the target data storage node to the target terminal service node based on the complete permutation order of the target set; and determining the global utility weight corresponding to the data storage node based on the target Shapley value contribution.

[0060] As one implementation method, regarding how to determine the global utility weight, the aforementioned weight calculation module 402 is further configured to: when the size of the target set is greater than a preset size threshold, determine the sampling order of the target set based on the Monte Carlo sampling method; determine the estimated Shapley value contribution of the target data storage node to the target terminal service node based on the sampling order of the target set; and determine the global utility weight corresponding to the data storage node based on the estimated Shapley value contribution.

[0061] As one implementation method, regarding how to generate resource scheduling instructions, the resource control module 403 is specifically used to: if the global utility weight is greater than a first resource scheduling threshold, generate a first physical storage medium migration instruction; the first physical storage medium migration instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be migrated to the first storage medium; if the global utility weight is less than a second resource scheduling threshold, generate a second physical storage medium migration instruction; the second physical storage medium migration instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be migrated to the second storage medium; the first resource scheduling threshold is greater than the second resource scheduling threshold, and the read / write rate of the first storage medium is greater than the read / write rate of the second storage medium.

[0062] As one implementation method, regarding how to generate resource scheduling instructions, the resource control module 403 is further configured to: if the global utility weight is greater than a third resource scheduling threshold, generate a first computing task priority adjustment instruction; the first computing task priority adjustment instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be processed using a first priority queue and / or a first resource quota; if the global utility weight is less than a fourth resource scheduling threshold, generate a second computing task priority adjustment instruction; the second computing task priority adjustment instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be processed using a second priority queue and / or a second resource quota; the third resource scheduling threshold is greater than the fourth resource scheduling threshold, the priority of the first priority queue is higher than the priority of the second priority queue, and the first resource quota is greater than the second resource quota.

[0063] As one implementation method, the data resource scheduling device 400 further includes: a topology incremental update module; the topology incremental update module is used to monitor metadata change events in the distributed data warehouse; and to incrementally update the preset data lineage heterogeneous topology map based on the change events.

[0064] In summary, this application first obtains the positive feedback signal from the terminal service node. Then, combining a pre-defined heterogeneous data lineage topology, a reverse propagation algorithm is used to propagate the positive feedback signal back to the data storage nodes on which the terminal service node depends, and a multi-source contribution attribution model is used to determine the global utility weight corresponding to the data storage node. The pre-defined heterogeneous data lineage topology includes the data dependencies between data storage nodes, processing logic nodes, and terminal service nodes. Finally, the global utility weight is compared with a pre-defined resource scheduling threshold, and resource scheduling instructions are generated based on the comparison results. Dynamic scheduling of data objects corresponding to the data storage nodes is then performed based on these resource scheduling instructions. These resource scheduling instructions include physical storage media migration instructions and / or computation task priority adjustment instructions. Thus, by introducing a reverse propagation algorithm and a multi-source contribution attribution model to fully analyze the data value corresponding to each data storage node, the importance of underlying data is inferred from the terminal service operation results, improving the rationality of data resource scheduling.

[0065] In addition, this application also provides a data resource scheduling device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the data resource scheduling method described above.

[0066] In addition, this application also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data resource scheduling method described above.

[0067] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for scheduling data resources, characterized in that, The scheduling method includes: acquiring positive feedback signals from terminal service nodes; using a reverse propagation algorithm to propagate the positive feedback signals back to the data storage nodes on which the terminal service nodes depend, based on a preset data lineage heterogeneous topology graph, and determining the global utility weight corresponding to the data storage nodes based on a multi-source contribution attribution model; the preset data lineage heterogeneous topology graph includes data dependencies between data storage nodes, processing logic nodes, and terminal service nodes; comparing the global utility weight with a preset resource scheduling threshold, and generating resource scheduling instructions based on the comparison results; the resource scheduling instructions include physical storage media migration instructions and / or computation task priority adjustment instructions; and dynamically scheduling the data objects corresponding to the data storage nodes based on the resource scheduling instructions.

2. The scheduling method according to claim 1, characterized in that, The scheduling method further includes: acquiring task scheduling logs and data processing scripts in a distributed data warehouse; performing abstract syntax tree parsing on the data processing scripts and identifying data dependencies between data storage nodes, processing logic nodes, and terminal service nodes; and constructing a preset data lineage heterogeneous topology graph containing the data storage nodes, processing logic nodes, and terminal service nodes based on the data dependencies.

3. The scheduling method according to claim 1, characterized in that, The step of obtaining the positive feedback signal of the terminal service node includes: obtaining the request latency and request frequency of the terminal service node; and determining the positive feedback signal corresponding to the terminal service node based on the request latency and the request frequency using a preset feedback calculation model.

4. The scheduling method according to claim 3, characterized in that, The expression corresponding to the preset feedback calculation model is: In the formula As a positive feedback signal, For the requested delay, For request frequency, As the first weighting coefficient, This is the second weighting coefficient.

5. The scheduling method according to claim 1, characterized in that, The step of determining the global utility weight corresponding to the data storage node based on the multi-source contribution attribution model includes: determining the target terminal service node; the target terminal service node is any one of the terminal service nodes; constructing a target set corresponding to the target terminal service node; the target set includes target data storage nodes that have a dependency relationship with the target terminal service node; determining the target Shapley value contribution of the target data storage node to the target terminal service node based on the complete permutation order of the target set; and determining the global utility weight corresponding to the data storage node based on the target Shapley value contribution.

6. The scheduling method according to claim 5, characterized in that, The scheduling method further includes: when the size of the target set is greater than a preset size threshold, determining the sampling order of the target set based on the Monte Carlo sampling method; determining the estimated Shapley value contribution of the target data storage node to the target terminal service node based on the sampling order of the target set; and determining the global utility weight corresponding to the data storage node based on the estimated Shapley value contribution.

7. The scheduling method according to claim 1, characterized in that, The step of comparing the global utility weight with a preset resource scheduling threshold and generating a resource scheduling instruction based on the comparison result includes: if the global utility weight is greater than a first resource scheduling threshold, generating a first physical storage medium migration instruction; the first physical storage medium migration instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be migrated to the first storage medium; if the global utility weight is less than a second resource scheduling threshold, generating a second physical storage medium migration instruction; the second physical storage medium migration instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be migrated to the second storage medium; the first resource scheduling threshold is greater than the second resource scheduling threshold, and the read / write rate of the first storage medium is greater than the read / write rate of the second storage medium.

8. The scheduling method according to claim 1, characterized in that, The step of comparing the global utility weight with a preset resource scheduling threshold and generating a resource scheduling instruction based on the comparison result includes: if the global utility weight is greater than a third resource scheduling threshold, generating a first computing task priority adjustment instruction; the first computing task priority adjustment instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be processed using a first priority queue and / or a first resource quota; if the global utility weight is less than a fourth resource scheduling threshold, generating a second computing task priority adjustment instruction; the second computing task priority adjustment instruction is used to instruct the data objects of the data storage node corresponding to the global utility weight to be processed using a second priority queue and / or a second resource quota; the third resource scheduling threshold is greater than the fourth resource scheduling threshold, the priority of the first priority queue is higher than the priority of the second priority queue, and the first resource quota is greater than the second resource quota.

9. The scheduling method according to claim 2, characterized in that, The scheduling method further includes: monitoring metadata change events in the distributed data warehouse; and incrementally updating the preset data lineage heterogeneous topology graph based on the change events.

10. A data resource scheduling device, characterized in that, include: The signal acquisition module is used to acquire the positive feedback signal from the terminal service node; The weight calculation module is used to combine a preset data lineage heterogeneous topology map, use a reverse propagation algorithm to propagate the positive feedback signal back to the data storage node on which the terminal service node depends, and determine the global utility weight corresponding to the data storage node based on a multi-source contribution attribution model; the preset data lineage heterogeneous topology map contains the data dependency relationships between data storage nodes, processing logic nodes and terminal service nodes. The resource control module is used to compare the global utility weight with a preset resource scheduling threshold and generate a resource scheduling instruction based on the comparison result; the resource scheduling instruction includes a physical storage medium migration instruction and / or a computing task priority adjustment instruction; the scheduling execution module is used to dynamically schedule the data objects corresponding to the data storage node based on the resource scheduling instruction.

Images