A hierarchical storage method and system based on task execution results
By quantitatively evaluating the contribution and confidence metrics of task nodes and dynamically adjusting data storage strategies, the problems of resource waste and insufficient reliability in traditional storage solutions are solved, achieving more efficient data management.
Patent Information
- Application Number
- CN202611388474.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-08
- Publication Date
- 2026-11-17
AI Technical Summary
Traditional tiered storage solutions fail to fully consider the actual semantic role of data during query task execution, and storage strategies lack dynamic correlation and fine-grained matching of data replica redundancy strategies, resulting in resource waste and insufficient reliability.
The hierarchical storage method based on task execution results dynamically constructs data storage mapping relationships by quantitatively evaluating the contribution and confidence indicators of task nodes, adjusting fault domain hierarchy and replica redundancy strategies, and optimizing storage resource utilization.
It optimizes storage resource utilization, reduces operation and maintenance costs, and improves the efficiency and security of accessing high-value data.
Smart Images

Figure CN123115144A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of distributed storage and data processing technology, specifically to a hierarchical storage method and system based on task execution results. Background Technology
[0002] With the advent of the big data era, distributed systems face the challenge of storing and managing massive amounts of data. Traditional tiered storage solutions typically use static or semi-static metrics such as data access frequency and data age to stratify data storage. For example, they periodically monitor the capacity utilization of the storage system, trigger data migration when the capacity of high-priority storage devices reaches a threshold, and assess the value of data objects in the storage devices and migrate them accordingly.
[0003] However, such solutions have the following drawbacks:
[0004] (1) The data value assessment has a single dimension and fails to fully consider the actual semantic role that the data plays in the process of querying tasks;
[0005] (2) The formulation of storage strategy lacks dynamic correlation with task execution quality, and cannot adaptively adjust storage strategy according to the quality differences of task execution results;
[0006] (3) The determination of the data copy redundancy strategy is relatively rough and fails to match the reliability and consistency requirements of the data itself. Summary of the Invention
[0007] To overcome the above-mentioned technical defects, the present invention provides a hierarchical storage method and system based on task execution results.
[0008] To solve the above problems, the present invention is implemented according to the following technical solution:
[0009] In a first aspect, the present invention provides a hierarchical storage method based on task execution results, comprising: acquiring a query task input by a user; performing intent recognition and decomposition processing on the query task to generate a task flow consisting of multiple task nodes, wherein the task nodes have data dependencies; traversing and executing each task node in the task flow to generate a task execution result corresponding to each task node; quantitatively evaluating the task execution quality index of each task node based on the task execution result of each task node, wherein the task execution quality index includes at least a contribution index value for characterizing the semantic contribution of the task node to the final query result, and a confidence index value for characterizing the logical consistency of the output of the task node; in response to the evaluation result of the task execution quality index, constructing a data storage mapping relationship: determining the target fault domain level according to the contribution index value based on a preset contribution-fault domain hierarchy mapping rule; and determining a replica redundancy strategy according to the confidence index value; and writing the task execution result into a logical data block of the corresponding logical storage partition of the distributed system based on the data storage mapping relationship to achieve hierarchical storage based on data.
[0010] In conjunction with the first aspect, the present invention provides a first specific implementation of the first aspect. Specifically, the step of constructing a data storage mapping relationship in response to the evaluation result of the task execution quality index includes: obtaining the node position of any task node in the task flow in the logical topology graph, and calculating the topological distance between the node position and each logical storage partition; dividing the multiple logical storage partitions into multiple fault domain levels according to the mapping relationship between the topological distance and the fault domain level, wherein the logical storage partitions within the same fault domain level have the same logical coupling degree with any task node, and the logical coupling degree increases as the topological distance decreases; determining the target storage partition corresponding to each fault domain level based on the data volume information of the task execution result of any task node, the task execution quality index, and the current load status of each logical storage partition in each fault domain level; and generating a replica redundancy strategy for the output data of any task node according to the target storage partition corresponding to each fault domain level, so as to construct the data storage mapping relationship.
[0011] In conjunction with the first aspect, the present invention provides a second specific implementation of the first aspect. Specifically, obtaining the node position of any task node in the task flow in the logical topology graph and calculating the topology distance between the node position and each logical storage partition includes: parsing a pre-constructed logical topology graph, which consists of task nodes, logical storage partitions, and communication links connecting the two; locating the current node coordinates of any task node in the logical topology graph and extracting the partition identifier of the target logical storage partition; traversing the transmission paths from the current node coordinates to the target node coordinates of each logical storage partition; counting the number of intermediate switching nodes or routing nodes passed through the transmission paths and determining the counted number of nodes as the topology distance; and generating a set of topology distances with the any task node as the starting point and each logical storage partition as the ending point, based on the topology distances corresponding to each logical storage partition.
[0012] In conjunction with the first aspect, the present invention provides a third specific implementation of the first aspect. Specifically, determining the target storage partition corresponding to each fault domain level based on the data volume information of the task execution result of any task node, the task execution quality index, and the current load status of each logical storage partition in each fault domain level includes: constructing or updating a logical storage resource tree, wherein the leaf nodes in the logical storage resource tree are mapped one-to-one with the logical storage partitions in the distributed system, and the non-leaf nodes are used to characterize the fault domain range corresponding to their subtrees; wherein each logical storage partition maintains first metadata representing the available storage space, and each non-leaf node maintains second metadata representing the cumulative available storage space in its subtree; based on the setting in the task execution quality index... The reliability index value is used to determine the replica redundancy coefficient for the target fault domain level according to a preset confidence-replica coefficient mapping rule; the total required storage space for the task execution result is calculated, and the product of the data volume information and the replica redundancy coefficient is used as the total required storage space; a depth-first search is performed on the logical storage resource tree based on the total required storage space; during the search process, if the second metadata of the current non-leaf node is less than the total required storage space, the subtree of the node is pruned; if the current node is a leaf node and its first metadata is greater than or equal to the total required storage space, the leaf node is marked as a candidate node; the logical storage partition mapped by the marked candidate node is determined as the target storage partition for the target fault domain level.
[0013] In conjunction with the first aspect, this invention provides a fourth specific implementation of the first aspect. Specifically, the step of quantitatively evaluating the task execution quality indicators of each task node based on the task execution results of each task node includes: aggregating the output data of all task nodes to generate a first overall execution result, and attaching a unique logical reference identifier to the first overall execution result, the logical reference identifier being used to associate the full task node index of the first overall execution result; simulating the removal of the output data of the target task node, and aggregating the output data of the remaining task nodes to generate a second overall execution result; inputting the first overall execution result and the second overall execution result into a preset semantic similarity calculation model, calculating the semantic difference between the two, and quantifying the semantic difference into the contribution index value of the target task node; retrieving the runtime log data of the target task node, and extracting the task execution error rate and average response latency within a preset time window; normalizing the error rate and the average response latency respectively to obtain a dimensionlessly uniform error rate score and latency score; calculating a reliability score based on a preset weighted scoring function; and mapping the reliability score to the interval [0,1]. The confidence index value is calculated; and the contribution index value is combined with the confidence index value to generate a task execution quality index that includes contribution and confidence.
[0014] In conjunction with the first aspect, the present invention provides a fifth specific implementation of the first aspect. Specifically, the step of inputting the first overall execution result and the second overall execution result into a preset semantic similarity calculation model to calculate the semantic difference between the two includes: extracting the logical reference identifier of the first overall execution result; constructing an attention weight matrix composed of the mapping relationship between the result fragment and the output data of each task node; wherein the values in the attention weight matrix are contribution weight values, used to characterize the probability of semantic influence of the output data of each task node on a specific result fragment; based on the attention weight matrix, locating the weight vector corresponding to the target fragment, the weight vector containing the contribution weight values of each task node to the target fragment; filtering task nodes whose contribution weight values are greater than a preset threshold and identifying them as key nodes affecting the target fragment; based on the index information in the weight vector, extracting feature vectors semantically aligned with the target fragment from the output data of the key nodes; calculating the feature vector missing rate in the second overall execution result, and calculating the semantic difference based on the feature vector missing rate.
[0015] In conjunction with the first aspect, the present invention provides a sixth specific implementation of the first aspect. Specifically, the quantitative evaluation of the task execution quality indicators of each task node further includes determining the confidence index value, including: for a target task node, invoking a preset introspection logic, and generating a reflection text characterizing the node's self-verification of its output based on the task execution result and execution trajectory of the target task node, the reflection text containing logical consistency verification information and anomaly attribution information; performing semantic parsing on the reflection text to extract positive feature words characterizing execution certainty and negative feature words characterizing execution uncertainty; and calculating the confidence index value of the target task node based on a preset confidence mapping rule and the weighted proportion of the positive and negative feature words; wherein the confidence index value is positively correlated with the weight of the positive feature words, and is used to quantitatively characterize the reliability of the task execution result of the target task node.
[0016] In conjunction with the first aspect, the present invention provides a seventh specific implementation of the first aspect, specifically further including: monitoring the current load status of each logical partition in the distributed system and the task execution quality indicators; triggering a dynamic scaling mechanism in response to the detection of data write congestion on high-contribution task nodes or loss of storage replicas on low-confidence task nodes; obtaining pre-configured global resource pool information, wherein the global resource pool information defines the maximum total number of physical nodes that can be used to execute the task flow; calculating the resource consumption weight of each task node based on the historical resource consumption baseline and the current task execution quality indicators of each task node in the task flow; logically slicing the maximum total number of physical nodes proportionally based on the resource consumption weight to generate a scheduling quota exclusive to each task node; constructing a hash ring space based on virtual nodes, mapping the scheduling quota to continuous hash slot intervals, and allocating the hash slot intervals to each task node; broadcasting the allocation results of the hash slot intervals and the corresponding storage tiering strategy to each execution node; so that each execution node can create or destroy task instances within its respective hash slot interval according to its local computing power, and obtain the unique access address of the instance by preempting the hash slot.
[0017] In conjunction with the first aspect, the present invention provides an eighth specific implementation of the first aspect. Specifically, the step of constructing a hash ring space based on virtual nodes, mapping the scheduling quota to a continuous hash slot interval, and allocating the hash slot interval to each task node includes: constructing a ring hash space of a preset length using a consistent hashing algorithm, and generating multiple virtual nodes within the ring hash space; responding to the registration request of each task node, obtaining the physical resource identifier and its instance quota corresponding to each task node; determining the number of virtual nodes corresponding to each task node based on the size of the instance quota, and establishing a mapping relationship between the virtual nodes and the physical resource identifier; uniformly dividing the ring hash space into a fixed number of hash slots, and mapping the virtual nodes to the corresponding hash slot intervals, such that each hash slot interval is continuously distributed on the ring hash space; and based on the mapping relationship, routing data read / write requests carried within the hash slot intervals to the corresponding task nodes to complete the allocation of the hash slot intervals to each task node.
[0018] Secondly, the present invention also provides a hierarchical storage system based on task execution results, comprising: a task flow construction module configured to acquire a query task input by a user, perform intent recognition and decomposition processing on the query task, and generate a task flow consisting of multiple task nodes, wherein the task nodes have data dependencies; a task flow execution module configured to traverse and execute each task node in the task flow, and generate a task execution result corresponding to each task node; and a quality evaluation module configured to quantitatively evaluate the task execution quality indicators of each task node based on the task execution results of each task node, wherein the task execution quality indicators include at least the following: The system includes a contribution index value representing the semantic contribution of the task node to the final query result, and a confidence index value representing the logical consistency of the task node's output; a mapping relationship construction module configured to construct a data storage mapping relationship in response to the evaluation results of the task execution quality index: determining the target fault domain level based on the contribution index value according to a preset contribution-fault domain hierarchy mapping rule; and determining a replica redundancy strategy based on the confidence index value; and a hierarchical storage execution module configured to write the task execution result into the corresponding logical partition of the distributed system based on the data storage mapping relationship to achieve hierarchical data storage.
[0019] Compared with the prior art, the beneficial effects of the present invention are:
[0020] After traversing each task node in the task execution flow, the contribution and confidence index values of each task node are quantitatively evaluated to construct a data storage mapping relationship based on data value density. The target fault domain level is dynamically adapted according to the contribution, and the replica redundancy strategy is adaptively adjusted according to the confidence. In response to the mapping relationship, the task execution results are written to the corresponding logical partition, enabling the storage system to perform differentiated management based on the actual semantic value and logical reliability of the data. This solves the problems of rigid storage strategies and inability to perceive data value in traditional technologies, which lead to storage resource waste and insufficient data reliability. It optimizes storage resource utilization, reduces system operation and maintenance costs, and improves the access efficiency and security of high-value data. Attached Figure Description
[0021] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:
[0022] Figure 1 This is a flowchart illustrating a hierarchical storage method based on task execution results according to the present invention.
[0023] Figure 2 This is a block diagram of an electronic device used to implement embodiments of the present invention;
[0024] In the picture:
[0025] 10 - Electronic device; 101 - Computing unit; 102 - ROM; 103 - RAM; 104 - Bus; 105 - I / O interface; 106 - Input unit; 107 - Output unit; 108 - Storage unit; 109 - Communication unit. Detailed Implementation
[0026] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0027] like Figures 1-2 As shown, this invention provides a hierarchical storage method and system based on task execution results.
[0028] Example 1
[0029] This application provides a hierarchical storage method based on task execution results, applicable to intelligent query systems with multi-task node collaborative processing. In this system, a complex user query is broken down into multiple task nodes with data dependencies, each producing intermediate or final results after execution. This application aims to dynamically construct data storage mapping relationships by quantitatively evaluating the output quality of each task node, thereby writing different data into the most suitable logical partition in the distributed system and achieving optimal allocation of storage resources.
[0030] (1) Query task: refers to the data query or analysis request input by the user that requires the distributed system to perform calculations and return results, such as structured query language query, natural language query or machine learning inference request, etc.
[0031] (2) Task node: refers to the smallest executable unit obtained after the intent recognition and decomposition of the query task. Each task node corresponds to a subtask in the query task. Task nodes can have data dependencies, that is, the output of one task node can be used as the input of another task node.
[0032] (3) Task flow: A directed acyclic graph structure composed of multiple task nodes organized according to data dependencies, which represents the complete execution logic of the query task.
[0033] (4) Contribution index value: A quantitative value used to characterize the degree of semantic contribution of a task node to the final query result. The higher the contribution, the more important the output data of the task node is to the final query result, and the higher its storage value density.
[0034] (5) Confidence index value: A quantitative value used to characterize the consistency and reliability of the output logic of a task node. The higher the confidence level, the more reliable the execution result of the task node and the lower the risk of error.
[0035] (6) Fault Domain Level: Refers to the different logical isolation levels divided according to the topological distance between the logical storage partition and the task node, with the task node as the reference point. The smaller the topological distance, the lower the fault domain level, the closer the storage location is to the task node, and the lower the access latency; the larger the topological distance, the higher the fault domain level, and the stronger the fault isolation capability.
[0036] (7) Logical topology diagram: A logical structure diagram consisting of task nodes, logical storage partitions and communication links connecting the two, used to describe the logical connection relationship between components in a distributed system.
[0037] (8) Topological distance: refers to the number of intermediate switching nodes or routing nodes passed between the starting node and the target node in the logical topology diagram, and is used to quantitatively represent the topological distance between two nodes.
[0038] (9) Logical storage resource tree: A tree-shaped data structure for managing storage space, containing non-leaf nodes and leaf nodes. Each leaf node uniquely maps to a logical data block, which is used to accelerate the retrieval and allocation of available storage space.
[0039] (10) Replica redundancy strategy: refers to the strategy of determining the number of data replicas in a distributed system based on the importance and reliability requirements of the data.
[0040] (11) Ring Hash Space: A ring-shaped space of hash values constructed based on the consistent hash algorithm, used to map the instance quota of task nodes to a continuous range of hash slots, so as to realize the distributed scheduling and scaling of task instances.
[0041] Please see Figure 1 , Figure 1 This is a flowchart illustrating a hierarchical storage method based on task execution results provided in an embodiment of this application. This hierarchical storage method can be executed by the aforementioned hierarchical storage system and includes:
[0042] S10: Obtain the query task input by the user, perform intent recognition and decomposition processing on the query task, and generate a task flow consisting of multiple task nodes, wherein the task nodes have data dependencies.
[0043] Specifically, the query task input by the user can be a query statement in natural language form. After obtaining the user's input query task, the system calls the intent recognition module to perform semantic parsing of the query task and identify the user's query intent. The intent recognition is implemented based on a natural language understanding method using a deep learning model, including: first, performing intent recognition, and then performing slot filling to form a structured semantic framework. For example, when a user inputs "Please calculate the sales revenue of each product category in the past month and find the top three categories with the highest sales revenue", the intent recognition module identifies that the query task contains two sub-intents: calculating sales revenue and sorting / filtering. After completing intent recognition, the system decomposes the query task into multiple sub-tasks with data dependencies, each sub-task corresponding to a task node. The data dependencies include: sequential dependency, parallel dependency, and conditional dependency. Among them, sequential dependency means that the later node depends on the output result of the earlier node; parallel dependency means that multiple nodes can be executed simultaneously, but the final result needs to be summarized; conditional dependency means that the execution of the later node depends on the execution result of the earlier node.
[0044] For example, the above query task can be broken down into the following task nodes: Task Node A, used to retrieve sales details for the past month; Task Node B, used to summarize sales revenue by product category; Task Node C, used to sort the summarized results in descending order; and Task Node D, used to extract the top three results. Task Node B depends on the output of Task Node A, Task Node C depends on the output of Task Node B, and Task Node D depends on the output of Task Node C, thus forming a task flow with sequential dependencies.
[0045] S20: Traverse and execute each task node in the task flow, and generate the task execution result corresponding to each task node.
[0046] Specifically, the system determines the execution order of each task node based on the data dependencies between them in the task flow. For task nodes without dependencies, parallel execution can be used to improve processing efficiency; for task nodes with dependencies, they are executed sequentially according to the order of their dependencies. The system schedules distributed computing resources to execute the subtasks corresponding to each task node. After each task node completes its execution, it generates a corresponding task execution result. The task execution result includes, but is not limited to, the data set output by the node, execution logs, execution time, resource consumption, and the size of intermediate results. The system persistently stores the execution results of each task node and associates each result with a corresponding node identifier for subsequent quality evaluation.
[0047] S30: Based on the task execution results of each task node, quantitatively evaluate the task execution quality indicators of each task node. The task execution quality indicators include at least a contribution index value used to characterize the semantic contribution of the task node to the final query result, and a confidence index value used to characterize the logical consistency of the output of the task node.
[0048] Specifically, after obtaining the task execution results of each task node, the system calls the quality assessment module to quantitatively evaluate the execution quality of each task node. The quality assessment module evaluates the execution quality of task nodes from at least two dimensions: contribution index and confidence index.
[0049] The contribution metric is used to characterize the degree of semantic contribution of a task node to the final query result. In other words, the contribution metric reflects the magnitude of the semantic role played by the execution result of the task node in the final output of the entire query task. The higher the contribution metric, the greater the impact of the execution result on the final query answer, and correspondingly, the higher the storage value of the result data.
[0050] The confidence score is used to characterize the logical consistency of the task node's output. In other words, the confidence score reflects whether the execution result of the task node is logically consistent, reliable, and without contradictions. The higher the confidence score of a task node, the more reliable its output result is, and correspondingly, the result data requires higher reliability assurance.
[0051] In a preferred embodiment, the step of quantitatively evaluating the task execution quality indicators of each task node based on the task execution results of each task node includes:
[0052] S301: Generate a first overall execution result based on the output data of all task nodes, and attach a unique logical reference identifier to the first overall execution result. The logical reference identifier is used to associate the index of all task nodes that generated the first overall execution result.
[0053] Specifically, after all task nodes in the task flow have been executed, the system retrieves the output data of each task node. Following the data dependencies and final result assembly logic defined in the task flow, the system combines, summarizes, or concatenates the output data of all task nodes to generate a complete final query result, which serves as the first overall execution result. This first overall execution result represents the most complete query answer the system can provide without removing any task node outputs. The system assigns a globally unique logical reference identifier to this first overall execution result and establishes a mapping relationship between this logical reference identifier and the full list of task node indexes that participated in generating the result.
[0054] S302: Simulate the removal of the output data of the target task node, and generate a second overall execution result based on the output data of the remaining task nodes.
[0055] Specifically, the system selects a target task node from the task flow and removes or masks its output data from the overall execution result, simulating the scenario where the task node is not executed or fails to execute. Then, based on the output data of all other task nodes in the task flow except the target task node, the system regenerates the query result according to the same final result assembly logic, serving as a second overall execution result. This second overall execution result reflects the query result lacking the contribution of the target task node.
[0056] It should be noted that if data dependencies in the task flow prevent downstream nodes from executing properly due to a lack of input data after removing a target task node, the system will replace the removed output data with a preset default value or null value to fill the input gap in the downstream nodes, ensuring that the second overall execution result can still be generated. The selection rules for this default value or null value must ensure that it does not introduce additional semantic information, thereby avoiding deviations in subsequent semantic difference calculations. For example, for numerical aggregation results, 0 or NULL can be used; for text results, empty strings or specific placeholders can be used.
[0057] S303: Input the first overall execution result and the second overall execution result into a preset semantic similarity calculation model, calculate the semantic difference between the two, and quantify the semantic difference into the contribution index value of the target task node.
[0058] Specifically, the system invokes a pre-trained or configured semantic similarity calculation model to compare the first and second overall execution results at the semantic level. This semantic similarity calculation model measures the closeness of the two results in terms of semantic content. The semantic similarity calculation model adopts a pre-trained language model based on the Transformer architecture. First, the first and second overall execution results are processed into textual representations, converting them into an input format acceptable to the model. For structured query results, the system serializes them into standardized text descriptions, such as "Product category A has sales of X yuan, ranking 1st; Product category B has sales of Y yuan, ranking 2nd...". Then, the textual representations of the two results are input into the pre-trained language model to extract their corresponding semantic vector representations, i.e., sentence vectors or document vectors. Subsequently, the cosine similarity or Euclidean distance between the two semantic vectors is calculated to obtain the semantic similarity value S_sim, where the semantic similarity value ranges from [0,1], with 1 indicating complete semantic similarity. The system defines the semantic difference as Δ=1-S_sim. The closer Δ is to 1, the greater the semantic change in the final result after removing the target task node, indicating a higher contribution of the task node; the closer Δ is to 0, the smaller the semantic change after removal, indicating a lower contribution of the task node.
[0059] The system directly uses the calculated semantic difference Δ as the contribution index value C_i of the target task node, or multiplies Δ by a preset scaling factor and maps it to a preset contribution value range [0,1]. The higher the contribution index value, the greater the semantic contribution of the task node to the final query result. Using the above method, the system executes operations S302 to S303 for each task node in the task flow to obtain the set of contribution index values {C_1, C_2, ..., C_n} for all task nodes, where n is the total number of task nodes.
[0060] S304: Retrieve the runtime log data of the target task node and extract the task execution error rate and average response latency within a preset time window.
[0061] Specifically, the system retrieves runtime log data of the target task node during execution from the log storage module of the distributed computing framework or the monitoring system. This log data includes at least the timestamp of task execution, execution status (success / failure / abnormality), exception type, and response latency of each operation or subtask.
[0062] The system extracts log records within a preset time window based on the current time point and statistically analyzes two key indicators: (1) task execution error rate E_rate, which is the ratio of the number of operations that fail or encounter anomalies within the time window to the total number of operations; and (2) average response latency L_avg, which is the average time taken for all successful operations within the time window. The error rate reflects the stability of task execution, and the average response latency reflects the efficiency of task execution. Together, they form the basis for evaluating the confidence of the data.
[0063] S305: Normalize the error rate and the average response delay to obtain a dimensionlessly consistent error rate score and delay score; calculate the reliability score based on a preset weighted scoring function.
[0064] Specifically, the system calculates the reliability score Rel_i of the target task node based on the error rate E_rate and the average response latency L_avg, combined with a preset tolerance threshold. The reliability score is used to quantify the credibility of the task node's output—the lower the error rate and the shorter the response latency, the higher the reliability.
[0065] In one specific implementation, the weighted scoring function adopts the following formula: Where E_threshold is the preset upper limit of error rate tolerance. When the error rate exceeds this threshold, the min function limits the error rate penalty term to 1, indicating that the penalty of the error rate dimension in the reliability score reaches its maximum value. L_min and L_max are the preset lower and upper limits of the normal range of response latency, respectively. When the average response latency is lower than L_min, it is considered an ideal state, and its normalized value is 0. When the average response latency exceeds L_max, it is considered a severe delay, and its normalized value is limited to 1 by the min function. and The preset weighting coefficients satisfy... ,and , ∈(0,1), which can be flexibly adjusted according to the system's sensitivity to error rate and response latency. The value of Rel_i ranges from [0,1], with the closer to 1 indicating higher reliability and the closer to 0 indicating lower reliability.
[0066] S306: Map the reliability score to a confidence index value within the interval [0,1]; and combine the contribution index value with the confidence index value to generate a task execution quality index that includes contribution and confidence.
[0067] Specifically, after obtaining the contribution index value C_i and the reliability score Rel_i of the target task node, the system directly maps the reliability score Rel_i to the confidence index value Conf_i, i.e., Conf_i = Rel_i. Since Rel_i itself is already within the interval [0,1], this mapping is an identity mapping and no additional scaling or transformation operations are required.
[0068] In a preferred embodiment, the task execution quality index is represented as a vector: Quality_i = (C_i, Conf_i), where C_i is the contribution index value and Conf_i is the confidence index value. This vector carries two independent dimensions of quality information: the contribution C_i measures the semantic contribution of the task node to the final query result, and the confidence Conf_i measures the reliability of the task node's output result.
[0069] In a preferred embodiment, the step of inputting the first overall execution result and the second overall execution result into a preset semantic similarity calculation model to calculate the semantic difference between the two includes:
[0070] S3031: Extract the logical reference identifier of the first overall execution result and construct an attention weight matrix consisting of the mapping relationship between the result fragment and the output data of each task node; wherein, the values in the attention weight matrix are contribution weight values, which are used to characterize the semantic influence probability of the output data of each task node on a specific result fragment.
[0071] Specifically, the system performs structured parsing of the initial overall execution result. During the actual query execution process, when generating the final result, the system embeds a logical reference identifier for each result fragment. This logical reference identifier records which task nodes' output data were used to generate the result fragment and what operations were performed. For example, the sales summary value in a row of the final result may reference the "Sales Amount" field in the detailed data provided by task node A, and the "Product Category" grouping information provided by task node B.
[0072] The system iterates through all result segments in the first overall execution result and extracts the corresponding set of logical reference identifiers for each result segment. Each identifier in this set points to a specific task node. Based on the extracted reference relationships, the system constructs an attention weight matrix M, where the rows of the matrix correspond to each result segment (denoted as F_1, F_2, ..., F_m), and the columns correspond to each task node in the task flow (denoted as T_1, T_2, ..., T_n). Each element m_{j,k} in the matrix represents the contribution weight value of task node T_k to result segment F_j.
[0073] The contribution weight value is determined as follows: if the result fragment F_j directly references the output data of task node T_k, then m_{j,k} is set to 1, or assigned a real value greater than 0 based on the number of references, the proportion of data volume, or the weight proportion of the node's output data in the fragment generation operation; if it is not referenced, the value is 0. If a result fragment is generated by weighted aggregation of the outputs of multiple task nodes, the contribution weight value reflects the weight proportion of each node's output in the aggregation.
[0074] S3032: Based on the attention weight matrix, locate the weight vector corresponding to the target segment, wherein the weight vector contains the contribution weight values of each task node to the target segment.
[0075] Specifically, the system selects a result segment to be analyzed from the first overall execution result as the target segment. The target segment can be any result segment, but is preferably the core segment that has the greatest semantic impact on the final answer. Based on the row corresponding to the target segment in the contribution weight matrix, the system extracts all column values of that row to form an n-dimensional weight vector W = [m_{j,1}, m_{j,2}, ..., m_{j,n}], where n is the total number of task nodes. Each element in this weight vector explicitly indicates the degree of contribution of the corresponding task node to the target segment.
[0076] S3033: Filter out task nodes whose contribution weight value is greater than a preset threshold and identify them as key nodes that affect the target segment.
[0077] Specifically, the system sets a preset threshold τ based on the weight vector W. The value of the threshold τ depends on the rules for assigning contribution weight values: when the contribution weight values are assigned using binary representation, τ can be set to 0; when the contribution weight values are assigned using continuous real values, τ can be set to 0.1 or dynamically determined based on the quantiles of the weight distribution. The system iterates through all elements in the weight vector, filters out task nodes whose contribution weight values are greater than τ, and marks these nodes as the set K of key nodes that affect the target segment. Key nodes are task nodes whose output data has a substantial effect on the generation of the target segment, while nodes with very low or zero contribution weight values are considered semantically irrelevant to the target segment.
[0078] S3034: Based on the index information in the weight vector, extract the feature vector that is semantically aligned with the target fragment from the output data of the key node.
[0079] Specifically, based on a determined set of key nodes K, the system extracts semantically aligned content from the output data of each key node. The alignment operation is based on the index information in the weight vector and the data lineage recorded during execution, meaning the system can clearly trace which field or record of which task node each part of the target fragment originates from.
[0080] The extraction method depends on the data type: if the target fragment is structured data, the system extracts the corresponding field values or record sets from the output data of the key node as feature vectors; if the target fragment is unstructured text, the system uses a semantic encoder to extract the phrases or sentences most semantically relevant to the target fragment from the output text of the key node and encodes them as vector representations. The system aggregates the feature vectors of all key nodes to form a support feature vector set V_support = {v_1, v_2, ..., v_|K|}, where |K| is the number of key nodes.
[0081] S3035: Calculate the feature vector missing rate in the second overall execution result, and calculate the semantic difference based on the feature vector missing rate.
[0082] Specifically, the system compares the supporting feature vector set with the second overall execution result. Since the second overall execution result lacks the output data of the target task node, if the target task node belongs to the key node set K, the feature vectors from the target task node in the supporting feature vector set will not find corresponding matching content in the second overall execution result; if the target task node does not belong to the key node set K, its feature vector missing rate is zero.
[0083] The system attempts to match each item in the supporting feature vector set within the second overall execution result. Matching methods can be: for structured data, exact matching of field values or record identifiers is used; for unstructured text, semantic vector similarity matching is used. The number of successfully matched feature vectors is counted. Let the total number of feature vectors be N_support = |K|, and the number of successful matches be N_match. The feature missing rate is defined as: Missing Rate = (N_support − N_match) / N_support. This missing rate directly reflects the proportion of feature content used to support the semantics of the target fragment lost after removing the target task node, which is the semantic difference degree corresponding to that target fragment. When calculating only for a single target fragment, this missing rate is the semantic difference degree Δ of the target task node, and thus serves as the contribution index value C_i of that task node.
[0084] In a preferred embodiment, the quantitative evaluation of the task execution quality indicators of each task node further includes determining the confidence index value. Specifically, this includes:
[0085] S3A-1: For the target task node, a preset self-reflection logic is invoked. Based on the task execution result and execution trajectory of the target task node, a reflection text is generated that represents the node's verification of its own output. The reflection text includes logical consistency verification information and anomaly attribution information.
[0086] Specifically, the system pre-configures an introspection logic module for each task node. This introspection logic is a set of verification rules and inference engines embedded in the task execution engine, used to perform self-examination and consistency checks on the execution process and output results of the task node after its execution is completed.
[0087] The execution of the introspection logic is based on two input data sources: (1) the task execution result of the target task node, that is, the specific data content output by the node; (2) the execution trajectory of the node, including the sequence of each operation recorded during the execution process, the source and characteristics of the input data, changes in intermediate variables, exception capture records, and system resource usage. The execution trajectory is recorded and persistently stored by the task node in real time during execution.
[0088] In a preferred embodiment, the introspection logic is constructed based on the reasoning capabilities of a large language model. Specifically, the system structures and organizes the task execution results and execution trajectory of the target task node, constructs a prompt text containing contextual information, and inputs this prompt text into a pre-deployed large language model, which can be an LLaMA2 large language model. The prompt text explicitly requires the large language model to check the logical consistency between the execution results and the execution trajectory from the perspective of a verifier, and to perform attribution analysis on any anomalies that occur during execution, ultimately outputting the reflection conclusion in natural language text form. Leveraging its semantic understanding and logical reasoning capabilities, the large language model can identify implicit contradictions, omissions, or abnormal patterns in the execution results and express them as parsable reflection text.
[0089] After the system invokes the introspection logic, it will automatically perform the following verification operations and generate reflection text:
[0090] (a) Logical consistency check: The introspective logic checks whether the output data meets the preset constraints. For example, for task nodes that perform data filtering operations, it checks whether the filtered data still meets the filtering conditions in the original query; for nodes that perform aggregation operations, it checks whether the aggregation results are within a reasonable numerical range; for task nodes that perform join operations, it checks whether the join key matches in both the left and right tables. If there are a large number of unmatched records, the "low join key matching rate" verification information is recorded in the reflection text.
[0091] (b) Anomaly Attribution Information: The introspection logic categorizes and summarizes all anomalies captured during execution and attempts to attribute them to specific upstream data issues or system environment problems. For example, if a "field length exceeds limit" error occurs multiple times during execution, the attribution information is "the length of a field in the input data exceeds the target table definition"; if a "connection timeout" error occurs, the attribution information is "downstream storage node response delay".
[0092] The introspective logic outputs the above verification results and attribution information in the form of natural language text or structured text, forming a reflective text. This reflective text can be a standardized log description or a narrative text that is closer to natural language expression.
[0093] S3A-2: Perform semantic analysis on the reflected text to extract positive feature words representing execution determinism and negative feature words representing execution uncertainty.
[0094] Specifically, the system invokes the natural language processing semantic parsing module to perform deep semantic parsing on the reflective text generated by S3A-1. This semantic parsing module includes a pre-built feature word dictionary and a deep learning-based text classifier.
[0095] The initial construction of the feature word dictionary comes from two approaches: first, manually summarizing and labeling positive and negative feature words based on frequently occurring verification conclusion terms in historical task execution logs; and second, automatically selecting candidate words highly correlated with execution determinism / uncertainty from massive historical reflection texts using a statistical keyword extraction algorithm. Furthermore, the feature word dictionary is not static but dynamically updated through the following mechanism: after each task node is completed, the system uses the generated reflection text and the actual execution result quality of the corresponding task node as feedback signals. These feedback signals are used to fine-tune the weights of existing words in the feature word dictionary, while simultaneously identifying new frequently occurring words significantly correlated with execution quality and adding them to the feature word dictionary.
[0096] The feature word dictionary pre-defines two types of words:
[0097] Positive feature words: used to express the certainty, reliability and consistency of the execution result, such as: "verification passed", "match successful", "completely consistent", "meets expectations", "within normal range", "complete", "accurate", "no anomalies", "stable", "reliable", etc.
[0098] Negative feature words: used to express uncertainty, anomalies and potential problems in the execution results, such as: "not matched", "deviation", "out of limit", "abnormal", "ignored", "missing", "warning", "error", "inconsistency", "suspected", "attributed to", "timeout", "retry", etc.
[0099] The semantic parsing module performs word segmentation, part-of-speech tagging, and dependency parsing on the reflective text. Then, it traverses each word and phrase in the text, matches them using a feature word dictionary, and identifies and extracts all positive and negative feature words.
[0100] S3A-3: Based on the preset confidence mapping rules, calculate the confidence index value of the target task node according to the weighted proportion of the positive and negative feature words.
[0101] Specifically, after extracting positive and negative feature words, the system calculates the confidence index value Conf_i based on a preset confidence mapping rule. This confidence mapping rule uses the weighted proportion of positive and negative feature words as the core calculation factor. The formula for calculating the confidence index value Conf_i is: Conf_i = (W_positive × N_positive) / (W_positive × N_positive + W_negative × N_negative + ε), where N_positive is the total frequency of positive feature words extracted from the reflective text, and N_negative is the total frequency of negative feature words extracted; W_positive and W_negative are the preset weight coefficients for positive and negative feature words, respectively, and W_positive > W_negative > 0, to reflect that the positive incentive effect of positive evidence on confidence is greater than the inhibitory effect of negative evidence; ε is a preset small positive number used to prevent division by zero errors when the denominator is zero. When the weighted frequency of positive feature words in the reflective text is much higher than that of negative feature words, Conf_i approaches 1; when the weighted frequency of negative feature words is dominant, Conf_i approaches 0; when both are zero, Conf_i = 0.
[0102] S3A-4: The confidence index value is positively correlated with the weight of the positive feature words, and is used to quantify the reliability of the task execution result of the target task node.
[0103] Specifically, the confidence index Conf_i has the following characteristics: the more positive feature words there are and the higher their weights, the closer the value of Conf_i is to 1, indicating that the execution result of the task node is logically highly self-consistent, has few anomalies, and is highly reliable; conversely, the more negative feature words there are and the higher their weights, the closer the value of Conf_i is to 0, indicating that the execution result of the task node has many logical inconsistencies or anomalies and is unreliable.
[0104] S40: In response to the evaluation results of the task execution quality indicators, construct a data storage mapping relationship: determine the target fault domain level according to the contribution index value, and determine the replica redundancy strategy according to the confidence index value; wherein, the target fault domain level is negatively correlated with the topological distance of the task node, and the replica redundancy strategy is negatively correlated with the confidence index value.
[0105] Specifically, after obtaining the contribution and confidence scores of each task node, the system constructs a corresponding data storage mapping relationship for the execution results of each task node based on these two scores. The data storage mapping relationship includes at least two dimensions: target fault domain hierarchy and replica redundancy strategy.
[0106] In a preferred embodiment, the step of constructing a data storage mapping relationship in response to the evaluation result of the task execution quality index includes:
[0107] S401: Obtain the node position of any task node in the task flow in the logical topology graph, and calculate the topological distance between the node position and multiple logical storage partitions.
[0108] Specifically, the system pre-constructs a logical topology graph for the distributed system. This logical topology graph can be a directed or undirected graph structure. The nodes in the graph include two types: one type is task nodes, which are computing nodes that execute query subtasks; the other type is logical storage partition nodes, which are abstract representations of each logical partition in the distributed system. The edges in the graph represent data flows or network connections, and the weights of the edges can reflect network latency, bandwidth, or connection type, etc.
[0109] Regarding the pre-construction process of the logical topology graph: During cluster deployment or storage partition adjustments, the system automatically generates a logical topology graph by reading the physical connection relationships of switches, routers, and storage nodes, combined with the network identifiers and port mapping information of each node. After construction, the logical topology graph is persistently stored in the metadata management module as a graph data structure. When a node joins or leaves the cluster, the system incrementally updates the topology graph, rather than rebuilding it entirely.
[0110] For any task node in the task flow, the system first locates it in the logical topology graph based on its node identifier, obtaining its coordinates, i.e., its index position in the graph data structure. Then, using these coordinates as the starting point and each logical storage partition node as the ending point, the system searches for the shortest transmission path from the starting point to each ending point using a graph traversal algorithm. The system counts the number of intermediate switching or routing nodes along each transmission path and determines this number as the topological distance from the task node to the corresponding logical storage partition.
[0111] If a task node is directly connected to a logical storage partition, meaning there are no intermediate switching or routing nodes in the transmission path, the topology distance is 0. If there is a switch in the path, the topology distance is 1; if there are two routers connected in series, the topology distance is 2, and so on. The smaller the topology distance, the closer the logical distance between the task node and the logical storage partition, the fewer network hops are required for data writing and reading, the lower the transmission latency, and the higher the logical coupling.
[0112] S402: According to the mapping relationship between topological distance and fault domain level, the multiple logical storage partitions are divided into multiple fault domain levels, wherein the logical storage partitions within the same fault domain level have the same logical coupling degree with any of the task nodes.
[0113] Specifically, the system pre-defines a mapping table between topology distance ranges and fault domain levels. Fault domain levels, from lowest to highest, include, but are not limited to: disk-level fault domains, node-level fault domains, rack-level fault domains, cluster-level fault domains, and data center-level fault domains. A smaller topology distance indicates a closer logical distance between the task node and the storage partition, resulting in higher data read / write efficiency, suitable for storing high-contribution data, and therefore a lower corresponding fault domain level. Conversely, a larger topology distance indicates a greater logical distance, resulting in relatively lower read / write efficiency, suitable for storing low-contribution data, and therefore a higher corresponding fault domain level.
[0114] Based on the specific numerical values of the topological distance from any task node to each logical storage partition, and in conjunction with the aforementioned mapping table, the system divides multiple logical storage partitions into different fault domain levels. After division, all logical storage partitions within the same fault domain level have a logical coupling degree with any task node that falls within the same preset range. In other words, storage partitions belonging to the same fault domain level are equivalent in terms of their distance from any task node in the logical topology, and they have the same access efficiency and coupling characteristics.
[0115] For example, the mapping table can be set as follows:
[0116] Mapping table
[0117] Topological distance Fault Domain Hierarchy illustrate 1~2 L1 (Low Failure Domain, spanning disks / nodes) Highest coupling, lowest disaster recovery granularity 3~5 L2 (Medium Fault Domain, Across Racks) Medium coupling ≥6 L3 (High Fault Domain, Cross-Cluster / Data Center Level) Lowest coupling, highest disaster recovery granularity
[0118] S403: Based on the data volume information of the task execution result of any task node, the task execution quality index, and the current load status of each logical storage partition in each fault domain level, determine the target storage partition corresponding to each fault domain level.
[0119] After completing the fault domain hierarchy division, each fault domain hierarchy may contain multiple candidate logical storage partitions. The system needs to select one or more optimal target storage partitions from each fault domain hierarchy for actual data storage. The selection process comprehensively considers the following three dimensions:
[0120] (1) Data volume information: The task execution result of any task node has a specific data volume, such as MB, GB or TB. Different logical storage partitions have different capacity limits and current remaining capacity. The system prioritizes logical storage partitions with sufficient remaining capacity to accommodate the data volume and will not trigger the capacity threshold, so as to avoid write failures or frequent data migration due to insufficient capacity.
[0121] (2) The quality indicators for task execution are contribution index value and confidence index value. The contribution index value determines the tendency of the fault domain level where the data should be stored. However, within the same fault domain level, the selection can be further refined based on the confidence index value. For example, for data with low confidence, logical partitions with more reliable storage media are preferred within the same fault domain level.
[0122] (3) Current load status: Each logical storage partition reports its current load status in real time, including but not limited to: IOPS utilization, bandwidth utilization, read / write latency, CPU utilization, etc. The system prioritizes logical storage partitions with a current load status in the light load or normal range as target storage partitions to avoid writing data to hot partitions.
[0123] The system can use a weighted scoring method for comprehensive decision-making: for each candidate logical storage partition, a comprehensive score is calculated based on the above three dimensions of indicators, and the logical storage partition with the highest comprehensive score is determined as the target storage partition corresponding to the fault domain level. Specifically, the comprehensive score Score = a·Cap_Score + b·Quality_Score + c·Load_Score, where Cap_Score is determined by the ratio of remaining capacity to data volume, Quality_Score is determined by the weighted average of contribution and confidence, Load_Score is determined by the reverse normalized value of the load state, and a, b, and c are preset weight coefficients.
[0124] Through the above comprehensive decision-making, the system determines a corresponding target storage partition for each fault domain level. For example, for fault domain level L1, the target storage partition is determined as P1; for fault domain level L2, the target storage partition is determined as P3; and for fault domain level L3, the target storage partition is determined as P5.
[0125] S404: Generate a replica redundancy strategy for the output data of any task node according to the target storage partition corresponding to each fault domain level, so as to construct the data storage mapping relationship.
[0126] Specifically, the system selects a corresponding target storage partition for each fault domain level. Subsequently, based on the preset desired replica hub and the target storage partitions for each fault domain level, the system generates a replica redundancy policy. The replica redundancy policy includes at least the following information: the total number of replicas, the target storage partition identifier for each replica, the write order and consistency level of each replica, and fault domain constraints. After the replica redundancy policy is generated, the system persistently stores this policy as an important component of the data storage mapping relationship. This data storage mapping relationship not only records which specific logical storage partitions the output data of any task node should be written to, but also records information such as the master-slave relationship between replicas, synchronization strategies, and failover rules.
[0127] In a preferred embodiment, obtaining the node position of any task node in the task flow in the logical topology graph and calculating the topological distance between the node position and multiple logical storage partitions includes:
[0128] S4011: Parse the pre-built logical topology graph, which consists of task nodes, logical storage partitions, and communication links connecting the two.
[0129] Specifically, during the initialization phase or before each query task execution, the system loads and parses a pre-built logical topology graph from the distributed system's metadata management module. This logical topology graph is a logical abstraction of the distributed system's physical network architecture. Its node set contains two types of nodes: one type is task nodes, which are the computation nodes that execute the various subtasks in the query task, such as SparkExecutor, FlinkTaskManager, or custom worker nodes; the other type is logical storage partition nodes, which are abstract representations of various logical partitions in the distributed system, with each logical partition corresponding to a set of backend physical storage media. The edges in the graph represent communication links between task nodes and logical storage partitions, and the edge attributes can include metadata such as the network type, average transmission latency, and bandwidth limit of the link.
[0130] In a specific instance, the logical topology graph is stored using a directed acyclic graph or a general undirected graph data structure. Each node has a globally unique node identifier, and each edge records the source node identifier and the target node identifier. The system loads the entire topology graph into memory by reading the topology graph configuration file or calling the metadata interface, forming a graph data structure that can be operated on by graph traversal algorithms.
[0131] S4012: Locate the current node coordinates of any task node in the logical topology diagram, and extract the partition identifier of the target logical storage partition.
[0132] Specifically, the system performs node matching and location in the parsed logical topology graph based on the node identifier of any task node, and obtains the current node coordinates of that node in the topology graph. Here, node coordinates refer to the index position of the node in the graph data structure or a node object reference.
[0133] Simultaneously, the system obtains a list of partition identifiers for all available logical storage partitions in the distributed system. Each logical storage partition corresponds to a unique node in the topology graph, possessing a partition identifier. The system extracts these partition identifiers as the target node set for calculating topological distances.
[0134] S4013: Traverse the transmission path from the current node coordinates to the target node coordinates of each logical storage partition.
[0135] Specifically, the system uses the current coordinates of any task node as the starting point of the traversal and the target node coordinates of each logical storage partition node as the ending point. It then uses a graph traversal algorithm to search for a transmission path from the starting point to each ending point in the logical topology graph. The system retains a shortest transmission path for each target logical storage partition and records the sequence of all nodes traversed by that path. For example, the transmission path from task node A to logical storage partition P1 is: TaskNode_A→Switch_S1→Partition_P1.
[0136] S4014: Count the number of intermediate switching nodes or routing nodes traversed in the transmission path, and determine the counted number of nodes as the topology distance.
[0137] Specifically, for each transmission path, the system counts the number of intermediate nodes traversed besides the starting and ending points. These intermediate nodes are all switching or routing devices in the communication link, abstracted as intermediate nodes in the logical topology diagram. The counted number of intermediate nodes is the topology distance corresponding to that transmission path. If any task node is directly connected to a logical storage partition, meaning there are no intermediate switching or routing nodes in the transmission path, the topology distance is 0, indicating the highest logical coupling between them. If there is one switch in the path, the topology distance is 1; if there are two routers connected in series, the topology distance is 2, and so on.
[0138] In a specific example, assume the logical topology diagram has the following connections: Task node B is directly connected to switch SW1, switch SW1 is connected to storage partition P1 and switch SW2, and switch SW2 is connected to storage partition P2. Then, the transmission path from task node B to P1 is "TaskNode_B→SW1→Partition_P1", with one intermediate switch node and a topology distance of 1; the transmission path from task node B to P2 is "TaskNode_B→SW1→SW2→Partition_P2", with two intermediate nodes, SW1 and SW2, and a topology distance of 2. The smaller the topology distance, the fewer network hops are required for data writing and reading, the lower the transmission latency, and the higher the logical coupling between the task node and the storage partition.
[0139] S4015: Generate a set of topology distances with any one of the task nodes as the starting point and each logical storage partition as the ending point, based on the topology distances corresponding to each logical storage partition.
[0140] Specifically, after calculating the topology distances for all target logical storage partitions, the system aggregates the calculation results to generate a topology distance set. This topology distance set uses the node identifier of any task node as the starting point and records the partition identifier of each logical storage partition and its corresponding topology distance in key-value pairs. For example, this topology distance set can be represented as: Distance_Set={(Partition_P1,1),(Partition_P2,2),(Partition_P3,1),(Partition_P4,3)...}; where the key is the partition identifier of the logical storage partition, and the value is the number of intermediate switching nodes or routing nodes traversed from any task node to that partition.
[0141] After generating the topology distance set, the system persistently caches this set as the topology distance calculation result for any task node, so that it can be directly read and used when performing fault domain hierarchical partitioning. Simultaneously, if the logical topology graph changes, the system triggers an update mechanism for the topology distance set, recalculating the topology distances of the affected task nodes to ensure the timeliness and accuracy of the stored mapping relationships.
[0142] In a preferred embodiment, determining the target storage partition corresponding to each fault domain level based on the data volume information of the task execution result of any task node, the task execution quality index, and the current load status of each logical storage partition in each fault domain level includes:
[0143] S4031: Construct or update a logical storage resource tree, wherein the leaf nodes in the logical storage resource tree are mapped one-to-one with the logical storage partitions in the distributed system, and the non-leaf nodes are used to characterize the fault domain range corresponding to their subtrees; wherein, each logical storage partition maintains first metadata representing the available storage space, and each non-leaf node maintains second metadata representing the cumulative available storage space in its subtrees.
[0144] Specifically, the distributed system pre-constructs a corresponding logical storage resource tree for each fault domain level. This logical storage resource tree is a hierarchical data structure used to organize and manage all logical storage partitions under that fault domain level. The root node of the tree represents the overall storage resource pool of that fault domain level, the internal non-leaf nodes represent intermediate groups of storage resources, and the leaf nodes uniquely map to a specific logical storage partition within that fault domain level.
[0145] Each leaf node maintains first metadata, which includes at least the total capacity, used capacity, and remaining available storage space of the logical storage partition. Each non-leaf node maintains second metadata, which represents the maximum available storage space among all its child nodes. In other words, the second metadata is the maximum available storage space of all leaf nodes within the current node's jurisdiction, used to quickly determine whether the entire subtree can accommodate the amount of data to be stored during the retrieval process.
[0146] The system constructs or updates the logical storage resource tree at the following times: (1) after cluster initialization or fault domain hierarchical division, the system constructs the tree for the first time; (2) when the available storage space of any logical storage partition changes due to data writing or deletion, the system updates the metadata of the corresponding leaf node and all its ancestor nodes of the partition from bottom to top; (3) when a logical storage partition is added or removed, the system performs corresponding node insertion or deletion operations on the tree structure. The above update operations are completed incrementally, involving only nodes on the affected path, without requiring a full reconstruction.
[0147] S4032: Based on the confidence index value in the task execution quality index, determine the replica redundancy coefficient for the target fault domain level according to the preset confidence-replica coefficient mapping rule.
[0148] Specifically, the system determines the number of replicas that actually need to be stored in the target fault domain level, i.e., the replica redundancy coefficient, based on the calculated confidence index value Conf_i of any task node and the preset overall replica redundancy strategy.
[0149] In one specific implementation, the replica redundancy factor is determined as follows:
[0150] The system first obtains the preset total number of replicas, N_total. This total number of replicas represents the total number of replicas that need to be stored in the entire distributed system for the execution result of this task node, and can be dynamically determined by the system based on task priority. The system allocates N_total to various fault domain levels according to the position of the target fault domain level in the fault domain hierarchy. For the current target fault domain level, its corresponding replica redundancy coefficient R_layer = ceil(N_total × w_layer), where w_layer is the allocation weight for that level, preset by the system, and ceil is the rounding up function.
[0151] Furthermore, the system also adjusts the replica redundancy coefficient based on the confidence index value Conf_i: when Conf_i is lower than the preset low confidence threshold (e.g., 0.4), the system increases R_layer by one step to increase replica redundancy and compensate for the uncertainty that may exist in the output results of low confidence nodes; when Conf_i is higher than the preset high confidence threshold (e.g., 0.8), the system can keep R_layer unchanged or decrease it by one step to save storage resources.
[0152] S4033: Calculate the total required storage space for the task execution result, and use the product of the data volume information and the replica redundancy coefficient as the total required storage space.
[0153] Specifically, the system obtains the data volume information D_size of the task execution result of any task node and multiplies it by the replica redundancy coefficient R_layer of the target fault domain level to obtain the total storage space requirement D_total = D_size × R_layer required at that level. This total requirement represents the storage capacity required to store R_layer replicas at that level. The system performs a depth-first search starting from the root node of the logical storage resource tree. The goal of the search is to find a combination of logical data blocks that can accommodate the total requirement D_total at that level. During the search, the system visits each node in the tree in a depth-first traversal order.
[0154] S4034: Perform a depth-first search on the logical storage resource tree based on the total required storage space.
[0155] Specifically, the system starts from the root node of the logical storage resource tree and performs a depth-first search. The goal of the search is to find a logical storage partition that can accommodate the total requirement of D_total. During the search, the system visits each node in the tree in a depth-first traversal order.
[0156] During each node access in the depth-first search, the system executes the following judgment logic:
[0157] (1) For the currently accessed non-leaf node, the system reads its second metadata. If the second metadata is less than D_total, it means that the available storage space of any leaf node in the entire subtree governed by the node is insufficient to accommodate data of size D_total. In this case, the system performs a pruning operation, that is, it stops traversing the child nodes of the node and skips the subtree directly, thereby reducing the retrieval space and improving retrieval efficiency.
[0158] (2) For the currently accessed leaf node, the system reads the available storage space in its first metadata. If the available storage space is greater than or equal to D_total, that is, the logical storage partition can independently accommodate all the replica data of the current level, then the leaf node is marked as a candidate node.
[0159] Furthermore, when performing the above judgment, the system filters based on the current load status of each logical storage partition: for leaf nodes where the available storage space meets the requirements, it further checks whether the current IOPS utilization and bandwidth utilization of the logical storage partition are lower than a preset threshold (e.g., lower than 80%). If the load is too high, it will not be marked even if the capacity meets the requirements, in order to avoid writing data to hot partitions.
[0160] S4035: During the retrieval process, if the second metadata of the current non-leaf node is less than the total required storage space, then the subtree of that node is pruned; if the current node is a leaf node and its first metadata is greater than or equal to the total required storage space, then the leaf node is marked as a candidate node.
[0161] Specifically, during the depth-first traversal, the system performs the following operations for each visited node:
[0162] If the current node is a non-leaf node, the system compares the size of its second metadata with that of D_total. If the second metadata < D_total, it means that among all the leaf nodes under this node, the one with the largest available storage space cannot hold D_total alone. Therefore, there is no single leaf node in this subtree that meets the capacity requirement, and the system skips all descendant nodes of this subtree, backtracking to the previous level to continue traversing other branches. If the second metadata ≥ D_total, it means that at least one leaf node in this subtree has sufficient available storage space, and the system continues traversing its child nodes.
[0163] If the current node is a leaf node, the system compares the available storage space in the leaf node's first metadata with the size of D_total. When the available storage space is greater than or equal to D_total, the leaf node meets the capacity requirement, and the system marks it as a candidate node. When the available storage space is less than D_total, the leaf node does not meet the requirement of being accommodated independently, and the system does not mark it.
[0164] Through the pruning and marking mechanisms described above, the system can quickly locate all candidate leaf nodes that meet the capacity requirements in the logical storage resource tree, avoiding the performance overhead caused by full tree traversal.
[0165] S4036: The logical storage partition mapped to the marked candidate node is determined as the target storage partition of the target fault domain level.
[0166] Specifically, the system aggregates all marked candidate nodes to form a candidate set. If the candidate set contains multiple candidate nodes, the system selects the optimal node as the target storage partition. Selection strategies include: selecting the node with the largest available storage space to facilitate subsequent data growth; selecting the node with the lowest current load; or selecting the node with the smallest topological distance to any task node. If the candidate set is empty, meaning that no single logical storage partition has enough available storage space to accommodate D_total, the system triggers a storage resource expansion alarm, or adopts a sharding strategy to distribute the data to multiple logical storage partitions in the same fault domain level, and collectively determines these logical storage partitions as the target storage partition set.
[0167] S50: Based on the data storage mapping relationship, the task execution result is written to the corresponding logical partition of the distributed system to achieve hierarchical data storage.
[0168] Specifically, the distributed system comprises multiple logical partitions, each corresponding to a different storage medium type and / or different storage configuration. Based on the established data storage mapping relationship, the system writes the execution results of each task node to its corresponding logical partition.
[0169] In one specific implementation, the logical partitions of the distributed system are divided into multiple levels according to storage performance and reliability, from high to low. For example, they can be divided into the following three levels:
[0170] Level 1: This level is suitable for storing task execution results with high contribution and low confidence.
[0171] The second level is suitable for storing task execution results with high contribution and high confidence, or task execution results with low contribution but low confidence, that is, data with medium semantic value and reliability requirements.
[0172] The third level is suitable for storing task execution results with low contribution and high confidence, that is, data that has little impact on the semantics of the final query results, and whose output data itself is highly reliable and does not require additional redundancy protection.
[0173] The system searches for logical partitions in the distributed system that match the determined target fault domain hierarchy and replica redundancy policy for each task node. The matching process includes: mapping the target fault domain hierarchy of the task node to the corresponding storage hierarchy; comparing the replica redundancy coefficient of the task node with the pre-configured replica policy of the logical partition; and selecting logical partitions with consistent or at least stringent policies as write targets. The system achieves a complete closed loop based on hierarchical data storage: high-contribution and / or low-confidence task execution results are stored in higher-performance, higher-reliability storage tiers, achieving more sufficient replica redundancy and stricter fault domain isolation guarantees; while low-contribution and / or high-confidence task execution results are stored in lower-cost, lower-reliability storage tiers, effectively saving storage resources while meeting basic availability requirements.
[0174] As an optional embodiment, the method further includes a step of elastic scaling of task instances. This step is preferably performed before traversing and executing each task node in the task flow or in parallel with step S2, so as to dynamically adjust the actual number of concurrent instances of each task node according to global resource constraints and node load conditions. Specifically, it includes:
[0175] S2-1: Monitor the current load status of each logical partition in the distributed system and the task execution quality indicators.
[0176] Specifically, the system periodically collects load status metrics for each logical partition through monitoring agents deployed on each storage node. These load status metrics include: IOPS utilization (the percentage of current actual IOPS to the partition's nominal maximum IOPS), bandwidth utilization (the percentage of current actual read / write bandwidth to the partition's nominal maximum bandwidth), average read / write latency (the average response latency of all I / O operations within the most recent time window), storage media occupancy (the percentage of used capacity to total capacity), and queue depth (the number of I / O requests waiting to be processed).
[0177] The system normalizes the collected metrics and maps them to the interval [0,1]. For IOPS utilization, bandwidth utilization, storage media occupancy, and queue depth, the normalized value is equal to the original percentage value. For average read / write latency, a normal interval [L_low, L_high] is set. When the latency is lower than L_low, the normalized value is 0; when the latency is higher than L_high, the normalized value is 1. If the latency is within the interval, linear interpolation is used. The system performs a weighted summation of the normalized metrics to obtain the comprehensive load value Load_partition for each logical partition. The calculation formula is: Load_partition = w_1·U_IOPS + w_2·U_BW + w_3·U_Lat + w_4·U_Cap + w_5·U_Queue, where w_1 to w_5 are preset weight coefficients that sum to 1.
[0178] S2-2: In response to the detection of data write congestion on high-contribution task nodes or loss of storage replicas on low-confidence task nodes, a dynamic scaling mechanism is triggered.
[0179] Specifically, the system performs real-time analysis on continuously monitored data and activates a dynamic scaling mechanism when any of the following trigger conditions are met:
[0180] Triggering Condition 1: Data write congestion on high-contribution task nodes. The system determines whether the contribution index C_i of a task node exceeds a preset high-contribution threshold, for example, C_i > 0.7. If it does, and the overall load value Load_partition of the target storage partition corresponding to the task node exceeds a preset congestion threshold, for example, Load_partition > 0.8, then the system determines that the data write path of the high-contribution task node is congested. At this time, the system triggers a dynamic scaling mechanism to distribute the write pressure and alleviate congestion by increasing the number of concurrent instances of the task node.
[0181] Triggering Condition 2: Loss of storage replicas of low-confidence task nodes. The system determines whether the confidence index value Conf_i of a task node is lower than a preset low-confidence threshold, for example, Conf_i < 0.4. If it is lower, and the system detects that any storage replica of the task node is lost due to node failure, disk damage, or network partitioning, the system determines that the data reliability of the low-confidence task node is threatened. At this time, the system triggers a dynamic scaling mechanism to regenerate the lost replica data by expanding the execution instance of the task node.
[0182] The two triggering conditions mentioned above can take effect independently; the scaling up / down process will be triggered when either condition is met. The system records the triggering reason and trigger timestamp upon triggering, which will be used for subsequent auditing and cooldown period determination.
[0183] S2-3: Obtain pre-configured global resource pool information, wherein the global resource pool information defines the maximum total number of physical nodes that can be used to execute the task flow.
[0184] Specifically, after the task flow is generated but before actual scheduling and execution, the system reads global resource pool information from the configuration management module or the job configuration submitted by the user. This global resource pool information contains at least one key parameter: the maximum total number of physical nodes, denoted as N_max_physical. This parameter represents the upper limit of the number of physical nodes that the system allows to participate in execution simultaneously throughout the entire task flow. A physical node refers to an actual computing node in the cluster, and each physical node can run one or more task instances.
[0185] This upper limit can be preset by the user based on task priority, total cluster resources, and billing policy, or it can be dynamically calculated by the system based on the current idle resource status of the cluster. For example, if the query task submitted by the user is a high-priority task, N_max_physical can be set to a larger value (such as 64 physical nodes); if it is a low-priority background task, N_max_physical can be set to a smaller value (such as 8 physical nodes). The global resource pool information can also include auxiliary parameters such as scaling step size and cooldown time, but the core constraint is N_max_physical.
[0186] S2-4: Calculate the resource consumption weight of each task node based on the historical resource consumption baseline and the current task execution quality index of each task node in the task flow; based on the resource consumption weight, logically slice the maximum total number of physical nodes proportionally to generate a scheduling quota exclusive to each task node.
[0187] Specifically, the system parses the type information of each task node in the task flow. Different types of task nodes have different resource consumption characteristics during execution. For example, data scanning nodes consume a lot of disk I / O and memory; aggregation calculation nodes consume CPU and memory; connection nodes consume a lot of memory and network bandwidth; and lightweight filtering nodes consume relatively few resources.
[0188] The system pre-configures a resource consumption weight (denoted as W_type) for each node type. This weight is used to quantify the relative resource consumption of that type of node in the cluster. For example, the weight values are set as follows: data scanning node weight is 3, connection node weight is 4, aggregation calculation node weight is 3, filtering node weight is 1, and projection node weight is 1.
[0189] Furthermore, the system adjusts the base weights based on the historical resource consumption baseline of each task node. The historical resource consumption baseline is derived from the statistical average of the actual resources consumed by the task node during historical execution, including average CPU utilization, average memory usage, and average disk I / O throughput. The system calculates the ratio of historical resource consumption to the average consumption level of nodes of the same type, obtaining the correction factor γ_i. The corrected resource consumption weight is: W_eff_i = W_type × γ_i. For task nodes whose contribution index value C_i exceeds the preset high contribution threshold, the system multiplies its resource consumption weight by an amplification factor δ_high greater than 1 to allocate more resources to ensure the execution of high contribution nodes; for task nodes whose contribution index value C_i is lower than the preset low contribution threshold, the system multiplies its resource consumption weight by a reduction factor δ_low less than 1 to appropriately reduce the resource consumption of low contribution nodes.
[0190] Finally, the system iterates through all task nodes in the task flow, calculates the corrected resource consumption weight W_final_i = W_eff_i × δ_i for each node, where δ_i is an amplification or reduction coefficient determined based on the contribution index value, with a default value of 1.0, and calculates the total weight of all nodes Sum_W_final.
[0191] Then, the maximum number of physical nodes, N_max_physical, is allocated in slices according to the proportion of each node's final weight to the total weight. For the i-th task node, its exclusive scheduling quota upper limit, Quota_i = floor((W_final_i / Sum_W_final) × N_max_physical), where floor is a floor function to ensure that the total quota of all nodes does not exceed N_max_physical. The system generates a corresponding scheduling quota segment for each task node, which is represented in interval form. For example, node A gets [0, 20], node B gets [21, 45], node C gets [46, 60], etc., ensuring that the quota segments of all nodes do not overlap and the coverage does not exceed N_max_physical. The lower limit of each quota segment is the quota upper limit of the previous node plus 1, and the upper limit is the cumulative allocation value of the current node, thus forming a continuous and mutually exclusive interval division.
[0192] S2-5: Construct a hash ring space based on virtual nodes, map the scheduling quota to a continuous hash slot interval, and allocate the hash slot interval to each task node.
[0193] Specifically, the system constructs a ring-shaped hash space with a fixed, preset length (e.g., 2^32 or 2^64) to provide a unified addressing space. Multiple virtual nodes are evenly distributed within the ring-shaped hash space, each corresponding to a specific hash slot. The total number of hash slots (denoted as S_total) is significantly larger than the number of task nodes, typically set to more than 10 times N_max_physical to provide sufficient fine-grainedness for resource allocation and load balancing. Based on the scheduling quotas generated for each task node, the system maps the total number of hash slots to consecutive hash slot intervals in the same proportion. Specifically, for the i-th task node, the number of hash slots allocated to it is Slot_i = round((Quota_i / N_max_physical) × S_total). Starting from the initial position, the system allocates consecutive hash slot intervals to each task node sequentially within the ring-shaped hash space. After mapping, the system associates each consecutive hash slot interval with the node identifier of the corresponding task node, forming an allocation mapping table of "hash slot interval → task node".
[0194] S2-6: Broadcast the allocation results of the hash slot interval and the corresponding storage tiering strategy to each execution node; so that each execution node can create or destroy task instances in its respective hash slot interval according to its local computing power, and obtain the unique access address of the instance by preempting the hash slot.
[0195] Specifically, the system broadcasts the hash slot interval allocation results and the storage tiering policies associated with each task node to all execution nodes in the cluster via a distributed message bus or configuration distribution channel. Upon receiving the broadcast message, each execution node parses the message to obtain the list of hash slot intervals corresponding to the task node it is responsible for, as well as the storage tiering policy information for that task node. The storage tiering policy information includes the target fault domain level, replica redundancy coefficient, and target storage partition identifier, among other things.
[0196] Each execution node periodically assesses its local computing power while executing tasks. Quantitative indicators of local computing power include: CPU idle rate, remaining available memory, disk I / O wait time, and the number of currently running instances. The system normalizes these indicators and then comprehensively determines whether local computing power is sufficient or strained.
[0197] If local computing power is sufficient, the execution node preempts unclaimed hash slots within its assigned hash slot range, creating a corresponding task instance for each preempted hash slot. This instance is responsible for processing the data shards corresponding to that hash slot. When creating an instance, the execution node writes the instance's output data to the specified target storage partition according to the storage tiering strategy associated with that task node.
[0198] If local computing power is limited, the execution node proactively destroys some of its own task instances, releasing the corresponding hash slots so that other execution nodes can subsequently preempt them. Before destroying an instance, the execution node must ensure that the intermediate results data completed by that instance have been persistently written to the target storage partition according to the storage tiering strategy to avoid data loss.
[0199] In a preferred embodiment, the step of constructing a hash ring space based on virtual nodes, mapping the scheduling quota to consecutive hash slot intervals, and allocating the hash slot intervals to each task node includes:
[0200] S2-3-1: A ring hash space of a preset length is constructed using a consistent hashing algorithm, and multiple virtual nodes are generated within the ring hash space.
[0201] Specifically, the system employs a consistent hashing algorithm to construct a ring-shaped hash space. The preset length of this ring-shaped hash space is typically set to a sufficiently large integer range, such as 0 to 2^32-1 or 0 to 2^64-1, to ensure an extremely low probability of hash collisions and ample space. Within this ring-shaped hash space, the system generates a large number of virtual nodes. Each virtual node is a logical abstract representation of a physical worker node on the hash ring, and each virtual node has a unique hash value, which is calculated by performing a hash function (such as MD5) on the virtual node identifier.
[0202] S2-3-2: In response to the registration request of each task node, obtain the physical resource identifier and its instance quota segment corresponding to each task node.
[0203] Specifically, after the task flow is generated, each task node initiates a registration request to the system's resource management module. The registration request includes the task node's node identifier and a description of its required computing resources. In response to the registration request, the system allocates a corresponding physical resource identifier to each task node, which points to the worker node or group of worker nodes that actually executes the task node. Simultaneously, the system reads the instance quota corresponding to the task node from the calculation results, i.e., the range of the maximum number of concurrent instances that the node is allowed to occupy.
[0204] S2-3-3: Based on the size of the instance quota segment, determine the number of virtual nodes corresponding to each task node, and establish a mapping relationship between the virtual nodes and the physical resource identifier.
[0205] Specifically, the system determines the number of virtual nodes that a task node should map on the circular hash space proportionally based on the upper limit of its instance quota. The number of virtual nodes is determined as follows: Let the total number of virtual nodes on the circular hash space be V_total, the upper limit of each task node's quota be Quota_i, and the sum of all task node quota limits be Sum_Quota. Then, the number of virtual nodes corresponding to the i-th task node is V_i = round((Quota_i / Sum_Quota) × V_total). Subsequently, the system establishes a mapping relationship between each virtual node of each task node and its physical resource identifier. This mapping relationship is established as follows: for the j-th virtual node of the i-th task node, its virtual node identifier is bound to its physical resource identifier, forming a mapping entry.
[0206] S2-3-4: The annular hash space is evenly divided into a fixed number of hash slots, and the virtual nodes are mapped to the corresponding hash slot intervals, so that each hash slot interval is continuously distributed on the annular hash space.
[0207] Specifically, the system logically and uniformly divides the ring hash space into a fixed number of hash slots. The total number of hash slots is denoted as S_total, which is a preset constant, such as S_total=10000 or S_total=16384. Its value should be large enough to support fine-grained resource allocation, but not too large to avoid excessive metadata overhead. The division method is as follows: divide the total length L of the ring hash space by S_total to obtain the length of each hash slot, L_slot = L / S_total, which are numbered sequentially in a clockwise direction as Slot_0, Slot_1, ..., Slot_{S_total-1}, with each hash slot corresponding to a continuous range of hash values. The system maps virtual nodes to their corresponding hash slots based on their hash value positions in the ring hash space. Specifically, the system calculates the position of each virtual node's hash value in the ring space and determines the first hash slot encountered clockwise from that position as the hash slot to which the virtual node belongs. Since the virtual nodes are evenly distributed in the ring space, each hash slot will be covered by one or more virtual nodes after mapping.
[0208] S2-3-5: Based on the mapping relationship, the data read and write requests carried in the hash slot interval are routed to the corresponding task nodes to complete the allocation of the hash slot interval to each task node.
[0209] Specifically, after the system allocates hash slot intervals to each task node, it establishes a data routing table. This table records the mapping relationship between each hash slot interval and the physical resource identifier of the corresponding task node. When the system needs to execute a computation task corresponding to a task node, it calculates the hash value based on the key value of the data shard, then locates the hash slot interval to which the hash value belongs on the circular hash space, and then queries the routing table to find the task node and physical worker node corresponding to the hash slot interval, routing the data read / write request to the task instance running on that worker node.
[0210] Example 2
[0211] This application provides a hierarchical storage system based on task execution results. The hierarchical storage system includes a task flow construction module, a task flow execution module, a quality assessment module, a mapping relationship construction module, a hierarchical storage execution module, and a distributed system.
[0212] The task flow construction module is configured to obtain query tasks input by users, perform intent recognition and decomposition processing on the query tasks, and generate a task flow consisting of multiple task nodes, wherein the task nodes have data dependencies.
[0213] The task flow execution module is configured to traverse and execute each task node in the task flow, and generate the task execution result corresponding to each task node.
[0214] The quality assessment module is configured to quantitatively evaluate the task execution quality indicators of each task node based on the task execution results of each task node. The task execution quality indicators include at least a contribution index value used to characterize the semantic contribution of the task node to the final query result, and a confidence index value used to characterize the logical consistency of the output of the task node.
[0215] The mapping relationship construction module is configured to construct a data storage mapping relationship in response to the evaluation results of the task execution quality indicators: based on the preset contribution-fault domain level mapping rules, the target fault domain level is determined according to the contribution index value; and the replica redundancy strategy is determined according to the confidence index value.
[0216] The hierarchical storage execution module is configured to write the task execution results into the corresponding logical partition of the distributed system based on the data storage mapping relationship, so as to realize hierarchical data storage.
[0217] Example 3
[0218] According to embodiments of the present invention, the present invention also provides an electronic device, a readable storage medium, and a computer program product.
[0219] Figure 2 A schematic block diagram of an example electronic device 10 that can be used to implement embodiments of the present invention is shown. Electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their links and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0220] like Figure 2 As shown, the electronic device 10 includes a computing unit 101, which can perform various appropriate actions and processes according to a computer program stored in ROM 102 or a computer program loaded into RAM 103 from storage unit 108. RAM 103 may also store various programs and data required for the operation of device 10. The computing unit 101, ROM 102, and RAM 103 are interconnected via bus 104. I / O interface 105 is also linked to bus 104.
[0221] Multiple components in electronic device 10 are linked to I / O interface 105, including: input unit 106, such as keyboard, mouse, etc.; output unit 107, such as various types of displays, speakers, etc.; storage unit 108, such as disk, optical disk, etc.; and communication unit 109, such as network card, modem, wireless transceiver, etc. Communication unit 109 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0222] The computing unit 101 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, CPUs, GPUs, various dedicated AI computing chips, various computing units running machine learning model algorithms, DSPs, and any suitable processors, controllers, microcontrollers, etc. The computing unit 101 performs the various methods and processes described above, such as a hierarchical storage method based on task execution results. For example, in some embodiments, a hierarchical storage method based on task execution results can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed on device 10 via ROM 102 and / or communication unit 109. When the computer program is loaded into RAM 103 and executed by the computing unit 101, one or more of the hierarchical storage methods based on task execution results described above can be executed. Alternatively, in other embodiments, the computing unit 101 can be configured to perform a hierarchical storage method based on task execution results by any other suitable means (e.g., by means of firmware).
[0223] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs, ASICs, ASSPs, SOCs, CPLDs, computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0224] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.
[0225] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical links based on one or more wires, portable computer disks, hard disks, RAM, ROM, EPROM or flash memory, optical fibers, CD-ROMs, optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0226] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback; and input from the user can be received in any form.
[0227] The systems and technologies described herein can be implemented in computing systems that include backend components, middleware components, frontend components, or any combination thereof. The components of the system can be interconnected via digital data communication of any form or medium. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0228] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.
[0229] It should be understood that the various processes described above can be reordered, added, or deleted. For example, the processes described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.
[0230] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Therefore, any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.
Claims
1. A hierarchical storage method based on a task execution result, characterized by, include: The system obtains a query task input by the user, performs intent recognition and decomposition on the query task, and generates a task flow consisting of multiple task nodes, wherein the task nodes have data dependencies on each other. Traverse and execute each task node in the task flow, and generate the task execution result corresponding to each task node; Based on the task execution results of each task node, the task execution quality indicators of each task node are quantitatively evaluated. The task execution quality indicators include at least a contribution index value used to characterize the degree of semantic contribution of the task node to the final query result, and a confidence index value used to characterize the logical consistency of the output of the task node. In response to the evaluation results of the task execution quality indicators, a data storage mapping relationship is constructed: based on the preset contribution-fault domain hierarchy mapping rules, the target fault domain hierarchy is determined according to the contribution index value; and a replica redundancy strategy is determined according to the confidence index value. Based on the data storage mapping relationship, the task execution result is written into the logical data block of the corresponding logical storage partition of the distributed system to achieve hierarchical data storage.
2. The hierarchical storage method based on task execution results according to claim 1, characterized in that, The process of constructing a data storage mapping relationship in response to the evaluation results of the task execution quality indicators includes: Obtain the node position of any task node in the task flow in the logical topology graph, and calculate the topological distance between the node position and each logical storage partition. According to the mapping relationship between topological distance and fault domain level, the multiple logical storage partitions are divided into multiple fault domain levels. The logical storage partitions within the same fault domain level have the same logical coupling degree with any of the task nodes. The logical coupling degree increases as the topological distance decreases. Based on the data volume information of the task execution result of any task node, the task execution quality index, and the current load status of each logical storage partition in each fault domain level, determine the target storage partition corresponding to each fault domain level. Based on the target storage partition corresponding to each fault domain level, a replica redundancy strategy for the output data of any task node is generated to construct the data storage mapping relationship.
3. The hierarchical storage method based on task execution results according to claim 2, characterized in that, The step of obtaining the node position of any task node in the task flow in the logical topology graph and calculating the topological distance between the node position and each logical storage partition includes: Parse a pre-constructed logical topology graph, which consists of task nodes, logical storage partitions, and communication links connecting the two. Locate the current node coordinates of any task node in the logical topology graph, and extract the partition identifier of the target logical storage partition; Traverse the transmission path from the current node coordinates to the target node coordinates where each of the logical storage partitions is located; The number of intermediate switching nodes or routing nodes traversed in the transmission path is counted, and the counted number of nodes is determined as the topology distance. Based on the topological distances corresponding to each logical storage partition, a set of topological distances is generated, starting from any one of the task nodes and ending at each logical storage partition.
4. The hierarchical storage method based on task execution results according to claim 2, characterized in that, The step of determining the target storage partition corresponding to each fault domain level based on the data volume information of the task execution result of any task node, the task execution quality indicators, and the current load status of each logical storage partition in each fault domain level includes: Construct or update a logical storage resource tree, wherein the leaf nodes in the logical storage resource tree are mapped one-to-one with the logical storage partitions in the distributed system, and the non-leaf nodes are used to characterize the fault domain range corresponding to their subtrees; wherein, each logical storage partition maintains first metadata representing the available storage space, and each non-leaf node maintains second metadata representing the cumulative available storage space in its subtrees. Based on the confidence index value in the task execution quality index, the replica redundancy coefficient for the target fault domain level is determined according to the preset confidence-replica coefficient mapping rule. Calculate the total required storage space for the task execution result, and use the product of the data volume information and the replica redundancy coefficient as the total required storage space; A depth-first search is performed on the logical storage resource tree based on the total required storage space. During the retrieval process, if the second metadata of the current non-leaf node is less than the total required storage space, then the subtree of that node is pruned; if the current node is a leaf node and its first metadata is greater than or equal to the total required storage space, then the leaf node is marked as a candidate node. The logical storage partition mapped to the marked candidate node is determined as the target storage partition of the target fault domain level.
5. The hierarchical storage method based on task execution results according to claim 1, characterized in that, The quantitative evaluation of task execution quality indicators for each task node based on the task execution results of each task node includes: The first overall execution result is generated by aggregating the output data of all task nodes, and a unique logical reference identifier is attached to the first overall execution result. The logical reference identifier is used to associate the full task node index generated by the first overall execution result. Simulate removing the output data of the target task node, and aggregate the output data of the remaining task nodes to generate a second overall execution result; The first overall execution result and the second overall execution result are input into a preset semantic similarity calculation model to calculate the semantic difference between the two, and the semantic difference is quantified into the contribution index value of the target task node. Retrieve the runtime log data of the target task node and extract the task execution error rate and average response latency within a preset time window; The error rate and the average response delay are normalized to obtain a dimensionlessly consistent error rate score and delay score; a reliability score is calculated based on a preset weighted scoring function. The reliability score is mapped to a confidence index value within the interval [0,1]; and the contribution index value is combined with the confidence index value to generate a task execution quality index that includes contribution and confidence.
6. The hierarchical storage method based on task execution results according to claim 5, characterized in that, The step of inputting the first overall execution result and the second overall execution result into a preset semantic similarity calculation model to calculate the semantic difference between the two includes: Extract the logical reference identifier of the first overall execution result and construct an attention weight matrix consisting of the mapping relationship between the result fragment and the output data of each task node; wherein, the values in the attention weight matrix are contribution weight values, which are used to characterize the semantic influence probability of the output data of each task node on a specific result fragment; Based on the attention weight matrix, the weight vector corresponding to the target segment is located, and the weight vector contains the contribution weight value of each task node to the target segment; Filter task nodes whose contribution weight values are greater than a preset threshold and identify them as key nodes that affect the target segment; Based on the index information in the weight vector, feature vectors that are semantically aligned with the target fragment are extracted from the output data of the key node. Calculate the feature vector missing rate in the second overall execution result, and calculate the semantic difference based on the feature vector missing rate.
7. The hierarchical storage method based on task execution results according to claim 1, characterized in that, The quantitative evaluation of the task execution quality indicators for each task node also includes determining the confidence index value, including: For the target task node, a preset self-reflection logic is invoked. Based on the task execution result and execution trajectory of the target task node, a reflection text is generated that represents the node's verification of its own output. The reflection text includes logical consistency verification information and anomaly attribution information. Semantic analysis is performed on the reflected text to extract positive feature words representing execution determinism and negative feature words representing execution uncertainty; Based on the preset confidence mapping rules, the confidence index value of the target task node is calculated according to the weighted proportion of the positive and negative feature words. The confidence index value is positively correlated with the weight of the positive feature words, and is used to quantify the reliability of the task execution result of the target task node.
8. The hierarchical storage method based on task execution results according to claim 1, characterized in that, Also includes: Monitor the current load status of each logical partition in the distributed system and the task execution quality indicators; In response to the detection of data write congestion on high-contribution task nodes or loss of storage replicas on low-confidence task nodes, a dynamic scaling mechanism is triggered. Obtain pre-configured global resource pool information, wherein the global resource pool information defines the maximum total number of physical nodes that can be used to execute the task flow; Based on the historical resource consumption baseline and current task execution quality indicators of each task node in the task flow, calculate the resource consumption weight of each task node; based on the resource consumption weight, logically slice the maximum total number of physical nodes proportionally to generate a scheduling quota exclusive to each task node. Construct a hash ring space based on virtual nodes, map the scheduling quota to a continuous hash slot interval, and allocate the hash slot interval to each task node; The allocation results of the hash slot intervals and the corresponding storage tiering strategies are broadcast to each execution node, so that each execution node can create or destroy task instances within its respective hash slot interval based on its local computing power, and obtain the unique access address of the instance by preempting the hash slot.
9. A hierarchical storage method based on task execution results according to claim 8, characterized in that, The construction of a hash ring space based on virtual nodes, mapping the scheduling quota to consecutive hash slot intervals, and allocating the hash slot intervals to each task node includes: A ring hash space of a preset length is constructed using a consistent hashing algorithm, and multiple virtual nodes are generated within the ring hash space; In response to the registration requests from each task node, obtain the physical resource identifier and instance quota corresponding to each task node; Based on the instance quota, determine the number of virtual nodes corresponding to each task node, and establish a mapping relationship between the virtual nodes and the physical resource identifier; The ring hash space is evenly divided into a fixed number of hash slots, and the virtual nodes are mapped to the corresponding hash slot intervals, so that each hash slot interval is continuously distributed on the ring hash space. Based on the mapping relationship, data read and write requests carried in the hash slot interval are routed to the corresponding task nodes to complete the allocation of the hash slot interval to each task node.
10. A hierarchical storage system based on task execution results, characterized in that, include: The task flow construction module is configured to obtain query tasks input by users, perform intent recognition and decomposition processing on the query tasks, and generate a task flow consisting of multiple task nodes, wherein the task nodes have data dependencies. The task flow execution module is configured to traverse and execute each task node in the task flow, and generate the task execution result corresponding to each task node. The quality assessment module is configured to quantitatively evaluate the task execution quality indicators of each task node based on the task execution results of each task node. The task execution quality indicators include at least a contribution index value used to characterize the semantic contribution of the task node to the final query result, and a confidence index value used to characterize the logical consistency of the output of the task node. The mapping relationship construction module is configured to construct a data storage mapping relationship in response to the evaluation results of the task execution quality indicators: based on the preset contribution-fault domain level mapping rules, the target fault domain level is determined according to the contribution index value; and the replica redundancy strategy is determined according to the confidence index value. The hierarchical storage execution module is configured to write the task execution results into the corresponding logical partition of the distributed system based on the data storage mapping relationship, so as to realize hierarchical data storage.
