A computing cluster storage intelligent scheduling method

Abstract

This invention relates to the field of storage scheduling technology, specifically to an intelligent storage scheduling method for computing clusters, comprising the following steps: acquiring link topology and status data through an interface, generating a dynamic link status set, inputting OPTICS clustering to identify high-load nodes, calculating load correlation strength, outputting high-load node clusters, querying routing paths and generating adaptive paths through multi-objective optimization, calculating heat gradients to trigger migration or archiving instructions, generating a policy adjustment table, inputting it into a scheduling engine to judge the load mitigation effect, and outputting a dynamic scheduling policy. In this invention, real-time physical link topology modeling is used, load intensity is calculated through access frequency and I / O queue changes, high-load nodes are located using density clustering, balanced paths are generated by combining global routing and multi-objective optimization, and data is dynamically migrated based on heat gradients to achieve self-optimization of storage resources, improve load balancing and scheduling response capabilities, and reduce maintenance costs.

Description

Technical Field

[0001] This application belongs to the field of storage scheduling technology, and specifically relates to an intelligent storage scheduling method for computing clusters. Background Technology

[0002] Storage scheduling technology mainly involves the allocation, management and scheduling of storage resources in high-performance computing clusters. Its core aspects include dynamic allocation of storage resources, selection of data access paths, load balancing of storage nodes and maintenance of data consistency under multi-user and multi-task conditions. It also systematically covers how to efficiently utilize storage resources to meet computing needs in a large-scale parallel computing environment.

[0003] Traditional intelligent scheduling methods for computing cluster storage refer to allocating and scheduling storage nodes in a cluster environment based on data distribution, task load, and access requests through preset rules or static strategies. These methods rely on manually set allocation strategies and pre-determined scheduling schemes, or simple methods such as round-robin or minimum load priority to allocate storage resources and schedule tasks.

[0004] Traditional methods allocate storage resources using fixed rules or static attributes. Their scheduling process does not integrate physical link status and real-time performance indicators. Relying on manually preset priorities makes it difficult to match dynamically changing I / O queue depth and bandwidth utilization. The polling mechanism is prone to producing suboptimal solutions for path selection and resource allocation. File popularity assessment ignores the temporal fluctuation characteristics of access frequency, resulting in an imbalance between hot and cold data distribution and storage node load, exacerbating data access latency in multi-tasking scenarios. Static consistency maintenance strategies cannot match high-concurrency write requests, leading to storage resource fragmentation and decreased utilization. Summary of the Invention

[0005] To address the aforementioned problems, this application provides a method for intelligent scheduling of storage in a computing cluster, comprising:

[0006] Step S1: Obtain the physical link topology between the source node and the storage node through the storage node interface, and collect the data block access request frequency, I / O queue depth and real-time storage bandwidth utilization to generate a dynamic state set of the storage link;

[0007] Step S2: Input the dynamic state set of the storage link into the OPTICS clustering algorithm to calculate the load correlation strength between storage nodes, filter the cluster groups whose average strength exceeds the cluster strength judgment threshold, and obtain a set of high-load storage node clusters.

[0008] Step S3: Based on the start and end storage node identifiers of the high-load storage node cluster set, query candidate paths from the global routing table; obtain the node hop order, link transmission bandwidth value, and average disk response latency of the candidate paths, and calculate their remaining bandwidth capacity and cumulative response latency; set weight coefficients for the remaining bandwidth capacity and cumulative response latency for weighted calculation, sort the candidate paths according to the obtained evaluation values, and select the candidate path with the highest evaluation value as the load balancing adaptation routing path;

[0009] Step S4: Extract the access request frequency of adjacent storage nodes according to the load balancing adaptation routing path, calculate the access heat gradient value, if the access heat gradient value of two consecutive storage nodes exceeds the upper limit threshold, trigger the data replica migration instruction, if it is lower than the lower limit threshold, trigger the cold data archiving instruction, and generate a storage policy adjustment table.

[0010] Step S5: Input the storage policy adjustment table into the storage scheduling engine for execution, and after execution, verify whether the decrease in I / O queue depth exceeds the storage load mitigation threshold and whether the bandwidth utilization fluctuation is within the bandwidth stability tolerance range. If the verification results are both yes, then generate a dynamic storage scheduling policy.

[0011] Preferably, the dynamic state set of the storage link includes the time window access frequency distribution, I / O queue depth increment, and bandwidth utilization standard deviation.

[0012] Preferably, the high-load storage node cluster set includes storage node topology identifiers, load correlation coefficient matrix, and time-series rate of change standard deviation.

[0013] Preferably, the load balancing adaptation routing path includes the path's remaining bandwidth capacity, the storage node latency percentile, and the multi-objective optimization weight coefficients used for weighted calculation.

[0014] Preferably, in step S1, when collecting data block access request frequency, I / O queue depth and real-time storage bandwidth utilization, the collection interval is adjusted based on the load and traffic changes of the storage node.

[0015] Preferably, step S1 specifically includes:

[0016] The physical link topology between the source node and the storage node is obtained through the storage node interface. The link connection table in the storage node interface is called to identify the storage node identifier, interface ID, hop count and bandwidth configuration value, and generate a link path structure dataset.

[0017] Based on the storage node pairs in the aforementioned link path structure dataset, the data block access request frequency, I / O queue depth, and real-time storage bandwidth utilization are collected, and a storage node link status matching data table is generated by aligning with timestamps.

[0018] The storage node link status matching data table is invoked, and the access frequency fluctuation value, bandwidth change magnitude and load fluctuation rate are calculated based on the time window slice to generate the dynamic status set of the storage link.

[0019] Preferably, in step S2, the specific method for calculating the load association strength between storage nodes includes: based on the dynamic state set of the storage link, calculating the mean and standard deviation of the data block access request frequency and the mean and standard deviation of the I / O queue depth of the storage nodes in a continuous time period, generating a load state change coefficient after Z-Score normalization of the calculation results, and performing cluster analysis based on the load state change coefficient to generate a dense cluster set of storage node states, and using the standardized average distance of the storage nodes in the state mapping as the load association strength value.

[0020] Preferably, in step S3, the formula for calculating the evaluation value is:

[0021] Evaluation value = W1 × -W2× ;

[0022] in, The standard bandwidth indicator is calculated using the following formula: , The standard latency index is calculated using the following formula: , This represents the remaining bandwidth capacity of the current path. and These represent the maximum and minimum remaining bandwidth capacity among all candidate paths, respectively. This represents the cumulative response time for the current path. and W1 and W2 are the maximum and minimum cumulative response times among all candidate paths, respectively. W1 and W2 are preset dimensionless weight coefficients, and W1 + W2 = 1.

[0023] Preferably, in step S4, the formula used to calculate the access request frequency of adjacent storage nodes is:

[0024] ;

[0025] in, Represents storage node With storage nodes Access frequency values ​​between Representing the Storage node in this record With storage nodes The number of communications between them Represents storage node With storage nodes The average number of communications. Represents storage node With storage nodes Total number of access records Represents storage node With storage nodes Logical distance.

[0026] Preferably, in step S4, both the upper and lower limits are determined based on the percentiles of the heat gradient values ​​of all storage nodes in the path.

[0027] Preferably, the dynamic storage scheduling strategy includes a queue depth decline rate, a bandwidth fluctuation tolerance range, and strategy effectiveness verification conditions.

[0028] The advantages of this application include: it enables autonomous optimization of storage resource allocation and data distribution, breaks through the bottleneck of static strategy response to link state fluctuations and load changes, improves the load balance between storage nodes, reduces disk queue congestion caused by local hotspots, enhances the real-time response capability of storage resource scheduling in high-concurrency scenarios, and reduces the cost of manual strategy maintenance. Attached Figure Description

[0029] Figure 1 A schematic diagram of a computing cluster storage intelligent scheduling method;

[0030] Figure 2 A detailed flowchart of step S1 in the intelligent scheduling method for computing cluster storage;

[0031] Figure 3 A detailed flowchart of step S2 in the intelligent scheduling method for computing cluster storage;

[0032] Figure 4 A detailed flowchart of step S3 in the intelligent scheduling method for computing cluster storage;

[0033] Figure 5 A detailed flowchart of step S4 in the intelligent scheduling method for computing cluster storage;

[0034] Figure 6 A detailed flowchart of step S5 in the intelligent scheduling method for computing cluster storage. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are only some, not all, of the embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0036] See Figure 1 This application provides a method for intelligent scheduling of storage in a computing cluster, comprising:

[0037] Step S1: Obtain the physical link topology between the source node and the storage node through the storage node interface, and collect the data block access request frequency, I / O queue depth and real-time storage bandwidth utilization to generate a dynamic state set of the storage link;

[0038] Step S2: Input the dynamic state set of the storage link into the OPTICS clustering algorithm to calculate the load correlation strength between storage nodes, filter the cluster groups whose average strength exceeds the cluster strength judgment threshold, and obtain a set of high-load storage node clusters.

[0039] Step S3: Based on the start and end storage node identifiers of the high-load storage node cluster set, query candidate paths from the global routing table; obtain the node hop order, link transmission bandwidth value, and average disk response latency of the candidate paths, and calculate their remaining bandwidth capacity and cumulative response latency; set weight coefficients for the remaining bandwidth capacity and cumulative response latency for weighted calculation, sort the candidate paths according to the obtained evaluation values, and select the candidate path with the highest evaluation value as the load balancing adaptation routing path;

[0040] Step S4: Extract the access request frequency of adjacent storage nodes according to the load balancing adaptation routing path, calculate the access heat gradient value, if the access heat gradient value of two consecutive storage nodes exceeds the upper limit threshold, trigger the data replica migration instruction, if it is lower than the lower limit threshold, trigger the cold data archiving instruction, and generate a storage policy adjustment table.

[0041] Step S5: Input the storage policy adjustment table into the storage scheduling engine for execution, and after execution, verify whether the decrease in I / O queue depth exceeds the storage load mitigation threshold and whether the bandwidth utilization fluctuation is within the bandwidth stability tolerance range. If the verification results are both yes, then generate a dynamic storage scheduling policy.

[0042] The dynamic state set of storage links includes access frequency fluctuation values, bandwidth change magnitude, and load volatility. The high-load storage node cluster set includes storage node topology identifiers, load correlation coefficient matrix, and time-series change rate standard deviation. The load balancing adaptation routing path includes path remaining bandwidth capacity, storage node latency percentile, and multi-objective optimization weight coefficient. The storage policy adjustment table includes a replica migration target list, cold data archiving paths, and access heat gradient thresholds. The dynamic storage scheduling policy includes queue depth decline rate, bandwidth fluctuation tolerance range, and policy effectiveness verification conditions.

[0043] Specifically, such as Figure 2 As shown, the specific steps of S1 are as follows:

[0044] S101: Obtain the link connection structure between the source node and the storage node through the storage node interface, call the link connection table in the storage node interface, identify the storage node identifier, interface ID, hop count and bandwidth configuration value, filter the matching storage node pairs and map the communication path, and generate the link path structure dataset.

[0045] In the process of obtaining the link connection structure between the source node and the storage node through the storage node interface, it is first necessary to call the storage node interface from the network topology management module of the cluster scheduling platform. This interface has a built-in link connection table. The table structure contains the source node number, destination storage node number, interface ID, hop count information, and corresponding bandwidth configuration value. The scheduling module will traverse the storage node interfaces in a round-robin manner, read the structured field data in the link connection table, and cache it in the task scheduling queue.

[0046] Taking a real-time computing cluster as an example, suppose there are three link paths between source node A and storage node B. The interface IDs of the three link paths are labeled eth1, eth2, and ib0, with hop counts of 2, 3, and 1, and bandwidth configurations of 10Gbps, 5Gbps, and 40Gbps, respectively. We need to read the corresponding interface fields, extract the hop count and bandwidth values, and construct a set of connection pairs using (node ​​A, interface eth1, hop count 2, bandwidth 10Gbps) as a preliminary path tuple. .

[0047] When performing path filtering, paths with hop count ≤ 3 and bandwidth ≥ 5Gbps are selected as candidate paths. If multiple paths meet the criteria, path priority is ranked. The specific filtering process involves processing the set... Each path ,when At that time, then Write to candidate path set ,in Indicates the number of jumps. This represents the bandwidth value. For example, in the example above, all three paths meet the conditions and are selected into the candidate path set. .

[0048] Next, the candidate path set needs to be analyzed. Perform bandwidth priority sorting and define path priority vectors. Each element , This reflects the bandwidth corresponding to the number of hops. In the example above, the priority of the path on interface eth1 is 5, eth2 is 1.67, and ib0 is 40. Therefore, the final priority order is ib0>eth1>eth2, and the corresponding path identifier is (ib0, eth1, eth2).

[0049] After filtering, the logical communication paths between the source nodes and storage nodes are mapped sequentially according to the sorting order. By mapping node interface identifiers to path IDs, a link path structure dataset is constructed. This ultimately forms the physical link topology data structure, which is used for subsequent state matching operations.

[0050] Table 1 lists the structural attributes of available links between a pair of storage nodes in the cluster. This type of structure is used to form the basic path dataset after being collected in real time through the node interface table. This data provides fundamental support for subsequent node path optimization and dynamic link scheduling. It also facilitates the quantification of path performance metrics for ranking and filtering.

[0051] Table 1: Example Data Table of Link Connections

[0052] Source node target node Interface ID Hop count Bandwidth (Gbps) A B eth1 2 10 A B eth2 3 5 A B ib0 1 40

[0053] S102: Based on the storage node pairs in the link path structure dataset, the monitoring unit adaptively adjusts the collection interval according to the load and traffic changes of the storage node pairs, and collects the data block access request frequency, I / O queue depth, and real-time storage bandwidth utilization. The real-time storage bandwidth utilization (hereinafter referred to as bandwidth utilization) and the data block access request frequency are referred to as access frequency.

[0054] Align data based on timestamp field, unify time window, and generate storage node link status matching data table;

[0055] Based on the storage node pairs in the link path structure dataset, the set of paths to be monitored within the current task cycle is first determined. Then, the source node, target storage node, and interface ID triplet corresponding to each path are read sequentially. The monitoring interface provided by the cluster monitoring unit is called, and the data collection interval is adaptively adjusted according to real-time changes in the access frequency or bandwidth utilization of the path storage node pairs. The access frequency, I / O queue depth, and bandwidth utilization data for each link path within a specified time window are retrieved. The access frequency represents the number of requests per unit time, and the I / O queue depth represents the currently pending I / O requests for that link. / O request count, bandwidth utilization rate represents the ratio of real-time bandwidth usage to configured bandwidth value. During data collection, if a sudden change in load or bandwidth utilization is detected on a certain path, the collection interval is automatically shortened; if the status is stable, the collection interval is appropriately extended, thereby improving monitoring efficiency and reducing system load. After data acquisition, the timestamp format must first be standardized and aligned to the second level. Assuming a time window of 5 minutes (300 seconds), each type of parameter data is mapped to an array of 300 sampling points. For each path group, assuming its interface ID is ib0 and bandwidth configuration is 40Gbps, the access frequency array within the sampling time period is:

[0056] ;

[0057] The I / O queue depth array is as follows: ;

[0058] Bandwidth utilization array is ;

[0059] Each type of parameter sampling point needs to be normalized to unify the units of measurement, with the access frequency normalized based on the maximum value of 200. The I / O queue depth is normalized to a maximum value of 20. Bandwidth utilization data is a proportional value and does not require normalization. After normalization, a three-dimensional time series matrix is ​​built for each type of parameter. ,in This represents 300 observations within a time window, which are then aligned dimensionally using a unified timestamp. This involves merging and recording the values ​​of the three parameters every second to construct a path state record set. Each record unit is in the form of... ,in For the first The timestamp in seconds ultimately integrates the path status records of node pairs into a unified data table. Each record includes the source node, target node, interface ID, timestamp, access frequency normalized value, I / O queue depth normalized value, and bandwidth utilization. Example data might be a path in... The status record at each second is (A, B, ib0, 120, 0.92, 0.55, 0.48). The normalization values ​​of the above parameters can be set based on the original cluster data. If the 99th percentile of a node's access frequency in the past 30 days is 200 times / s, then 200 is used as the normalization benchmark. Similarly, the I / O depth is set as the maximum queue capacity. Finally, a node link status matching data table is formed, which is used for subsequent load volatility calculation and dynamic status expression.

[0060] S103: Call the node link status matching data table, extract access frequency and bandwidth data, calculate access frequency fluctuation value and bandwidth change amplitude based on time window slice, calculate load fluctuation rate, summarize the three types of data to express the status between node links, and generate a dynamic status set of storage links.

[0061] After calling the node link status matching data table, the access frequency and bandwidth utilization of each path are first sliced ​​within a unified time window. Assuming the time window is 300 seconds, with each 30-second segment as a sub-window, a total of 10 time slices are generated. For each time slice, the access frequency array... With bandwidth utilization array The standard deviation and the magnitude of change need to be calculated separately, and the bandwidth variation within each segment is defined as follows: The access frequency fluctuation value is:

[0062] ;

[0063] in For the first The average access frequency per segment, for example, within the third time slice, the access frequency is... Bandwidth utilization rate ,but , , After calculation, the results for each segment are stored in the fluctuation record set. Then, the load volatility within the overall time window is calculated and defined as follows:

[0064] ;

[0065] in To determine the number of segments (10 in this example), divide the 10 segments... and The final load volatility value is obtained by adding and averaging the results. Define the path as a stable state, if The path is defined as a slightly fluctuating state. Define a path as a high-volatility state. Based on this criterion, the dynamic states of a link can be classified and a set of dynamic states can be formed. For example, the weighted average volatility of 10 segments of a certain path is... If so, the path is marked as a slightly fluctuating state, and the state records of the path constitute the dynamic state set of the storage link.

[0066] Specifically, such as Figure 3 As shown, the specific steps of S2 are as follows:

[0067] S201: Based on the dynamic state set of the storage link, calculate the request frequency (the request frequency is the data block access request frequency) and I / O queue depth of the node (for simplicity, the storage node will be referred to as the node below) in a continuous time period, normalize the calculation results and reorganize them into a feature set to generate the load state change coefficient.

[0068] Based on the time-series data recorded in the dynamic state centralized path of the storage link, the original sequence of request frequency and I / O queue depth recorded by each node at the second level within a continuous 5-minute time window is extracted sequentially. For node A... arrive The request frequency array per second is:

[0069] ;

[0070] The I / O queue depth array is as follows: ;

[0071] Calculate the mean of the request frequency array separately. Standard deviation and the average I / O queue depth Standard deviation Then, for each time t, calculate its Z-score normalized value and request the normalized value of the frequency. The formula is:

[0072] ;

[0073] Where i = t; when Seconds, request frequency When the value is 188, we get Similarly, the I / O queue depth is... If the time interval is 13 seconds, then the normalized value of the I / O queue depth is... After normalization, the two sets of standardized sequences are combined into a node load feature matrix. Then, the eigenvalue pairs between adjacent time points are traversed, the load state change coefficient between consecutive time points is calculated, and the adjacent state points are calculated using Euclidean distance. and The formula for the change in time and the coefficient of change is:

[0074] ;

[0075] by and Taking seconds as an example, let's assume... , ,but:

[0076] ;

[0077] The sequence of 299 load state change coefficients for node A within a 300-second time window was obtained sequentially. , such as in to The coefficients of change between them are as follows:

[0078] ;

[0079] The above set of change coefficients is the load state change description data of node A, which is used for subsequent distance calculation and analysis.

[0080] S202: Call the load state change coefficient, calculate the Euclidean distance between adjacent state points between nodes and generate an reachability ranking graph, determine the cluster boundary based on local reachability fluctuations, extract node combinations, and generate a set of dense clusters for storing node states;

[0081] After calling the node load state change coefficient sequence, a two-dimensional vector space of adjacent state points between nodes is constructed, with each node having its own... Standardized eigenvalues ​​at a time of seconds As coordinate points, and perform pairwise Euclidean distance calculations on the set of nodes, with node A as the coordinate point. Seconds and node B at Taking seconds as an example, if the coordinates of its feature points are respectively and Then the Euclidean distance is:

[0082] ;

[0083] Construct a distance matrix from the distances between node pairs, and traverse the distance vectors centered on each node. Calculate the mean and standard deviation of the distances from each node to its five nearest neighbors. Let the distances from node A to its five nearest neighbors be... The mean is The standard deviation is Then, the fluctuation of the local reachability of each node is recorded, and the local reachability fluctuation of the node is defined as the difference between the maximum and minimum values ​​of these 5 distances, i.e. To determine whether local reachability fluctuations exceed a threshold, let the local fluctuation threshold be... If so, the node is determined to be a boundary node.

[0084] During boundary expansion, nodes whose distance is less than the current node mean plus twice the standard deviation (i.e., 0.172 + 2 × 0.024 = 0.22) are included in the cluster. Finally, node combinations from multiple clusters are extracted to form a set of node state dense clusters.

[0085] One example of a dense cluster is a node. The distances between them are all between 0.15 and 0.20, and this structure can be used to determine the load association of downstream applications.

[0086] S203: Based on the dense cluster set of storage node states, calculate the standardized average distance of nodes in the state mapping as the load association strength value, and filter clusters whose average strength exceeds the cluster strength judgment threshold to obtain a high-load storage node cluster set.

[0087] Based on the obtained set of dense clusters of node states, pairwise distance calculation and averaging are performed on the node combinations within each cluster. The average distance within the cluster is recorded as the load association strength value of that cluster. For example, if a cluster contains nodes... There are a total of 6 groups of four pairs. The distance of each group is calculated using the aforementioned method. Let the result be... The average distance is:

[0088] ;

[0089] This average value represents the correlation strength of the cluster. Then, the intensity values ​​of the clusters are statistically analyzed, and the overall mean and standard deviation of the set are calculated. Let the intensity values ​​of the 10 clusters be:

[0090] (0.175, 0.160, 0.182, 0.198, 0.144, 0.192, 0.122, 0.185, 0.170, 0.188);

[0091] Calculate the average intensity value of the above 10 clusters. The standard deviation is approximately Let the threshold for determining aggregation intensity be the average value plus one standard deviation, i.e. Clusters with an association strength value exceeding 0.1929 are selected. After selection, only cluster 4 (with a strength value of 0.198) meets the criteria, and cluster 4 is ultimately included in the high-load node cluster set. The cluster structure and its corresponding strength value are recorded to form the final high-load storage node cluster set.

[0092] Table 2 shows the composition of the selected high-load node clusters and their load intensity values.

[0093] Table 2: Information on High-Load Node Clusters

[0094] Cluster number Node composition Correlation strength value Judgment Result Cluster 4 A, B, F, H 0.198 High-load clusters

[0095] Specifically, such as Figure 4 As shown, the specific steps of S3 are as follows:

[0096] S301: Based on the start and end node identifiers of the high-load storage node cluster set, identify the corresponding path entries, extract the node jump order, transmission bandwidth value and disk device average response latency value from the path entries, and generate a node link information set;

[0097] Specifically, the first step is to retrieve path entries containing the identifiers of the start and end nodes from the routing topology database. Taking the start node A (number N1) to the end node D (number N4) as an example, the retrieved path entries are: N1→N2→N3→N4. The intermediate hops (such as N2 and N3) in each path entry are parsed one by one, and the hop order information between nodes is extracted, recording their physical connection order. Simultaneously, the transmission bandwidth value of each link segment is extracted from the node monitoring system or network status table; for example, the bandwidth value of the N1-N2 segment is 400Mbps, the bandwidth value of the N2-N3 segment is 600Mbps, and the bandwidth value of the N3-N4 segment is 450Mbps.

[0098] In addition, the average response latency of the disk devices in each storage node was queried. The average response latency of the disk devices in node N1 was 12ms, node N2 was 14ms, node N3 was 10ms, and node N4 was 16ms. The average response latency of the disk devices was obtained by statistical analysis of node load records or device raw response logs.

[0099] Finally, a node link information set is constructed: the jump order, bandwidth value, and response latency are encapsulated into triples. An example is shown below:

[0100] (N1-N2, 400Mbps, 12ms), (N2-N3, 600Mbps, 14ms), (N3-N4, 450Mbps, 10ms), this set is used for subsequent path performance analysis.

[0101] S302: Call the node link information set, use the Floyd-Worsche algorithm to calculate the remaining bandwidth capacity of the node pair, filter the path information with response latency lower than the benchmark value, and obtain the path performance filter value group;

[0102] The node link information set is used to construct a bandwidth matrix and a delay matrix. The elements of the bandwidth matrix represent the remaining bandwidth of the links between nodes, and the elements of the delay matrix represent the transmission delay of the links between nodes. First, the jump link pairs are traversed, and the corresponding bandwidth values ​​are extracted from all jump link pairs. If a link connection exists between nodes (e.g., N1-N2, N2-N3, N3-N4), the bandwidth value is filled into the matrix; otherwise, it is assigned a value of 0 or ∞.

[0103] The matrix is ​​updated according to the remaining bandwidth calculation rules (remaining bandwidth = original link bandwidth B - current transmission load L). For example, if the N1-N2 link bandwidth is 400Mbps and the current load is 120Mbps, then the remaining bandwidth is 280Mbps. The remaining bandwidth matrix is ​​obtained by performing the calculation for each link in the path.

[0104] During path selection, the cumulative response latency of each complete path is calculated by adding the latency of each link along the path. For example, the cumulative latency of path N1→N2→N3→N4 is 12+14+10+16=52ms. A path is retained if it meets the system's preset benchmark of 55ms; otherwise, it is discarded if the total latency of a path is 68ms. Finally, a set of path performance screening values ​​that meet the criteria is generated, and its data structure includes four indicators: start and end nodes, path sequence, remaining bandwidth, and cumulative latency.

[0105] S303: Based on the path performance screening value group, extract the remaining bandwidth and cumulative response delay parameters of the path, set weighting coefficients to normalize and weight the parameters, sort the summary results, select the optimal path number, and generate a load-balanced adaptive routing path.

[0106] Based on the path performance filtering value group, extract two parameters for the candidate paths: remaining bandwidth (B) and cumulative response latency (T). Assume there are 3 candidate paths in the current query, with the following parameters:

[0107] Candidate path P1: B=280Mbps, T=52ms;

[0108] Candidate path P2: B=300Mbps, T=47ms;

[0109] Candidate path P3: B=200Mbps, T=60ms.

[0110] Based on this, the extreme values ​​of each parameter are determined: maximum bandwidth B max =300Mbps, minimum bandwidth B min =200Mbps; Maximum latency T max =60ms, minimum delay T min =47ms.

[0111] The parameters of the three paths are then normalized for extreme values ​​in sequence:

[0112] For candidate path P1:

[0113] Standard bandwidth specifications Standard latency index .

[0114] For candidate path P2: Standard bandwidth metrics Standard latency index .

[0115] For candidate path P3: Standard bandwidth metrics Standard latency index .

[0116] Set dimensionless weighting coefficients, assuming bandwidth weight W1 = 0.6 and latency weight W2 = 0.4, and perform a weighted scoring calculation (evaluation value calculation formula: score = W1 × standard bandwidth index - W2 × standard latency index):

[0117] Score of candidate path P1 ;

[0118] Score of candidate path P2 ;

[0119] Score of candidate path P3 .

[0120] The path score results are sorted in descending order, and the path number with the highest score (i.e., P2) is selected as the load balancing adaptation route path for the current node cluster. Finally, the optimized route path number group is obtained for the scheduling module to execute the path distribution task.

[0121] As shown in Table 3, path P2 was ultimately selected as the suitable route path for the current task because its score of 0.600 was greater than that of candidate paths P1 and P2 (0.326 and -0.400, respectively).

[0122] Table 3: Routing Path Performance Parameters

[0123] Starting point node End point Path number Minimum remaining bandwidth (Mbps) Cumulative response time (ms) Path rating N1 N4 P1 280 52 0.326 N1 N4 P2 300 47 0.600 N1 N4 P3 200 60 -0.400

[0124] Specifically, such as Figure 5 As shown, the specific steps of S4 are as follows:

[0125] S401: Obtain access request records of adjacent node pairs under the load balancing adaptation routing path, i.e., the access frequency of adjacent storage nodes; collect the number of communication between node pairs, extract the access frequency value and arrange it in the path order, and generate an access heat gradient value distribution sequence based on the frequency change amplitude and path length fitting heat change ratio coefficient.

[0126] First, identify the node sequence of the current load path in the network topology. Taking path P={A, B, C, D, E} as an example, its set of adjacent node pairs is (A, B), (B, C), (C, D), (D, E).

[0127] The node network monitoring module retrieves internal access logs and queries communication records between nodes pair by pair. Each record includes fields such as source node, destination node, timestamp, and number of communications. For example, for node pair (A, B), two communication records were collected within the statistical period: record 1 showed 4 communications, and record 2 showed 2 communications. Based on this, an array of communication counts for node pair (i, j) is constructed. First, we need to extract the average number of communications between node pairs. The calculation method is as follows:

[0128] ;

[0129] Where n is the total number of records. Let the number of communications between node pair (A, B) be... ,Right now ,but:

[0130] ;

[0131] Calculate access frequency value The formula used is:

[0132] ;

[0133] in, Represents nodes in the path With nodes The access frequency value, measured in times per hop, serves as the fundamental parameter for calculating the access heat gradient, reflecting the coupling relationship between node access density and topology distribution. Representing the Nodes in this record With nodes The number of communications between them, in units of times. Representative node With nodes The average number of communications in the communication records, expressed in times, is a unitless count. Represents nodes in the path With nodes The logical distance, in units of jumps.

[0134] Example Calculation: Let the logical hop count be... Then we have:

[0135] First term (sum of absolute deviations):

[0136] ;

[0137] The second term (root of the variance term):

[0138] ;

[0139] Denominator term: ;

[0140] therefore, jump;

[0141] Repeat the above operation for all node pairs in path P, and arrange them in path order to generate an access frequency array in path order: , , , .

[0142] The amplitude of frequency sequence changes is then analyzed, and the heat gradient value for each node pair is calculated. For the frequency sequence, a weighted average of the frequency differences between the node pair and its adjacent node pairs is calculated (one-sided differences are used for edge nodes). For example:

[0143] For edge node pair AB:

[0144] ;

[0145] For intermediate node pair BC:

[0146] ;

[0147] This process is repeated to form a heat gradient distribution sequence, which is then recorded in the access heat gradient array, as shown in Table 4.

[0148] Table 4: Heat Gradient Distribution Table

[0149] Node pairs Communication frequency (times / hop) adjacent frequency difference Heat gradient value AB 0.8535 (Only the latter item) 0.2535 0.2535 BC 0.6000 0.2535,0.4000 0.3268 CD 0.2000 0.4000,0.8000 0.6000 DE 1.0000 (First item only) 0.8000 0.8000

[0150] As shown in Table 4, a preliminary heat gradient value can be formed based on the difference between the communication frequency and the adjacent frequency. This gradient value will serve as an important basis for the division of hot and cold zones.

[0151] S402: Based on the distribution sequence of access popularity gradient values, judge the popularity gradient values ​​of adjacent node pairs in the path, mark the path segment where the popularity values ​​of two consecutive nodes exceed the upper threshold as a hot spot, and mark the path segment where the popularity values ​​are below the lower threshold as a cold spot, and obtain the list of path segment popularity type identifiers.

[0152] Based on the distribution sequence of access popularity gradient values, the popularity gradient values ​​of adjacent node pairs in the path are judged one by one. First, the quantile method is used to set the judgment threshold, where the upper threshold is set to the 90th quantile of the node pair gradient value and the lower threshold is set to the 20th quantile, so as to avoid the bias caused by the averaging method under extreme values.

[0153] For example, suppose the gradient array is sorted in ascending order as follows:

[0154] ;

[0155] Calculated using the quantile method:

[0156] The lower threshold (20th percentile) is calculated using linear interpolation, falling between 0.2535 and 0.3268, resulting in 0.2535 + (0.3268 - 0.2535) × 0.2 = 0.2682.

[0157] Upper limit threshold (90th percentile): Using linear interpolation, it lies between 0.6000 and 0.8000, and is calculated as 0.6000 + (0.8000 - 0.6000) × 0.9 = 0.7800.

[0158] The heat gradient values ​​are classified and judged sequentially. If the gradient value is ≥0.7800, it is marked as a "hot spot"; if the gradient value is ≤0.2682, it is marked as a "cold spot"; and the middle value is marked as "neutral". The specific judgment results are as follows: AB (gradient 0.2535<0.2682), cold spot; BC (gradient 0.3268), neutral; CD (gradient 0.6000), neutral; DE (gradient 0.8000>0.7800), hot spot.

[0159] The final list of path segment popularity type identifiers is: [Cold spot, Neutral, Neutral, Hot spot].

[0160] S403: Call the path segment heat type identifier list, identify hot node pairs, perform replica migration operation to suboptimal path nodes, identify cold node pairs, archive to storage area, and establish storage policy adjustment table;

[0161] Perform data replica migration for hot path segments: Taking the hot marked path segment DE as an example, call the suboptimal path P2={A, B, X, E} in the path candidate set to check whether there is a resource conflict at node X. If there is no conflict, call the migration logic to migrate the original service resources to node X.

[0162] For cold path segments, data archiving is performed: Taking the cold-marked path segment AB as an example, the session request data involving node A to node B is retrieved from the main processing module and sent to the cold storage area via an archiving command. For example, the archived file for path segment AB is packaged as AB_cold_0713.bin. Finally, based on the processing status of hot and cold path segments, migration record tables and archiving record tables are generated respectively, and a path popularity storage strategy adjustment table is constructed. The record fields in this table include: path segment identifier, current popularity value, threshold type, processing action, and processing time, serving as the basis for strategy scheduling.

[0163] Specifically, such as Figure 6 As shown, the specific steps of S5 are as follows:

[0164] S501: Input the storage policy adjustment table into the storage scheduling engine, parse the data replica migration instruction and cold data archiving instruction, extract the target disk identifier, data block number and instruction type, mark disk behavior according to access attributes, construct a disk behavior data structure set, count the distribution quantity and type composition of access behavior, and obtain instruction behavior feature quantities.

[0165] When inputting the storage policy adjustment table into the storage scheduling engine, policies should be imported sequentially according to entry number. The instruction type field is parsed one by one, and the policies are categorized by reading the field identifier code (e.g., a CMD_CODE field set to "MIGR" indicates replica migration, and "ARCH" indicates archiving).

[0166] During this process, the following extraction operations are performed:

[0167] Extract the target disk identifier: such as DISK_ID="D003", which corresponds to the physical disk number or RAID logical volume number. Determine the corresponding device by comparing it with the disk list or device mapping table in the scheduling engine.

[0168] Extracting the data block number: For example, BLOCK_ID="B11234", it is necessary to combine the mapping relationship table to locate the offset or logical page number of the data block in the physical sector to form the specific operation target.

[0169] Extract instruction type: such as CMD_TYPE="WRITE_BACK", which indicates the execution form of disk behavior.

[0170] Subsequently, disk behavior is tagged using the access attribute table. Access attributes (recent access frequency, write cycle, read / write latency, etc.) are obtained through raw IO log statistics. For example, if a data block has been read 3 times, written 0 times, and last accessed 8 minutes ago in the past 10 minutes, its access frequency is calculated to be 0.3 times / minute, and its latency is 480 seconds. If the access frequency is less than 0.5 times / minute and the latency is greater than 300 seconds, it is marked as cold data access behavior.

[0171] The tagged disk behaviors are categorized and stored according to data block numbers, constructing a set of disk behavior data structures. Each structure contains: disk ID, data block ID, instruction type, access attribute type flag (hot / cold data), and operation timestamp.

[0172] Finally, the distribution characteristics of statistical access behavior are analyzed:

[0173] Behavior count: Counts the number of replica migrations, cold data archiving, and other behaviors (such as read optimizations). For example, in a certain period, there were 28 replica migrations, 41 archivings, and 13 other behaviors.

[0174] Cold and hot data classification: Based on access frequency aggregation, cold data (41), hot data (21), and fuzzy data (20, with a frequency between 0.4 and 0.6 times / minute) are statistically analyzed. The judgment criteria are: frequency < 0.5 is cold, frequency > 1.0 is hot, and frequency between 0.5 and 1.0 is fuzzy.

[0175] The above statistical items and their proportions are summarized as the characteristic quantities of instruction behavior after this round of analysis.

[0176] S502: Based on instruction behavior characteristics, collect I / O queue depth and bandwidth utilization, calculate the change range of queue depth, compare the bandwidth fluctuation value with the stable tolerance range, mark the disk status that simultaneously meets the conditions of queue depth decrease exceeding the storage load mitigation threshold and bandwidth fluctuation within the tolerance range, and obtain the scheduling trigger matching disk quantity.

[0177] Based on instruction behavior characteristics, disk status data is polled every 5 seconds. The current disk I / O request queue length, historical average, and current bandwidth utilization are obtained through system API calls.

[0178] Queue depth change magnitude: For example, the current queue length of disk D003 is 17, and the historical average over the past 5 minutes is 25, with a calculated decrease of 0.32.

[0179] Bandwidth fluctuation value: For example, disk D003 currently has a bandwidth utilization of 70%. Retrieve the past 10 polling records:

[0180] [68%, 70%, 72%, 73%, 69%, 71%, 74%, 72%, 70%, 71%], with a standard deviation σ of 2%.

[0181] Conditional Judgment: Bandwidth Stability Judgment: The bandwidth fluctuation tolerance range is set to ±3σ (i.e., 68% to 74%). If the current value is within this range 70% of the time, the bandwidth fluctuation is determined to meet the stability requirements.

[0182] Load mitigation assessment: The storage load mitigation threshold is set to 30%. The current queue depth decreases by 32% (>30%), which meets the condition.

[0183] If both of the above conditions are met simultaneously (queue depth reduction > threshold and bandwidth fluctuation < tolerance), then the disk is marked as a "scheduling-triggered matching disk".

[0184] For example, if the current monitored disk list is {D001,D002,D003,D004,D005}, and only D002 and D003 meet the criteria, then the scheduling trigger will match 2 disks. This value will be output and used for the next stage of scheduling policy generation.

[0185] As shown in Table 5, two disks (D002 and D003) meet the conditions that the queue drop is greater than 30% and the bandwidth fluctuation is within the set tolerance range. Therefore, they are determined to be the disks to be matched for scheduling trigger and enter the policy entry generation process.

[0186] Table 5: Monitoring Table of Disk Matching for Scheduling

[0187] Disk Number Current I / O queue depth Original average depth Depth of descent Current bandwidth utilization stable bandwidth range Does it match trigger? D001 23 24 4.17% 75% 68%-74% no D002 15 25 40% 72% 68%-74% yes D003 17 25 32% 70% 68%-74% yes D004 28 30 6.67% 76% 68%-74% no D005 22 24 8.33% 69% 68%-74% no

[0188] S503: Based on the scheduling trigger, match the disk quantity, match the target disk identifier and data migration behavior structure set, integrate the disk instruction combination sequence and the current queue load status, summarize the policy entries in the form of disk identifier, instruction sequence and load status combination, and generate a dynamic storage scheduling policy.

[0189] Based on the number of disks matched by the scheduling trigger, the disk IDs marked as matched in the scheduling engine are traversed. Taking matched disks D002 and D003 as an example, the structure records with disk ID fields D002 and D003 are filtered out one by one from the aforementioned data structure set, and the instruction type and data block number contained therein are read.

[0190] By sorting the timestamp field in the structure, a sequence of disk instruction combinations is constructed. For example, the instruction sequence corresponding to D002 is:

[0191] [{B11432, WRITE_BACK}, {B11435, MIGR}, {B11436, ARCH}], whose corresponding queue status is: I / O depth 12 rows, write-intensive ratio 70%, cold data ratio 40%.

[0192] The information is packaged into policy entries, in the format {Disk ID, Command Sequence, Current Load Status}. Disk policy entries that match the criteria are then aggregated and organized to generate the dynamic storage scheduling policy for the current period. For example, this might contain two policy entries, corresponding to disks D002 and D003 respectively.

[0193] Disk D002 policy: Instruction sequence [{B11432,WRITE_BACK},...], queue state with I / O depth of 12...

[0194] Disk D003 policy: The instruction sequence is [{B11540, MIGR}, {B11542, ARCH}], the queue status is I / O depth of 14 lines, cold data ratio of 65%, and average access latency of 520ms.

[0195] Finally, the policy entries are combined into a unified policy file and output to the scheduling system execution module.

[0196] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for intelligent scheduling of storage in a computing cluster, characterized in that, include: Step S1: Obtain the physical link topology between the source node and the storage node through the storage node interface, and collect the data block access request frequency, I / O queue depth and real-time storage bandwidth utilization to generate a dynamic state set of the storage link; Step S2: Input the dynamic state set of the storage link into the OPTICS clustering algorithm to calculate the load correlation strength between storage nodes, filter the cluster groups whose average strength exceeds the cluster strength judgment threshold, and obtain a set of high-load storage node clusters. Step S3: Based on the start and end storage node identifiers of the high-load storage node cluster set, query the candidate path from the global routing table, obtain the node hop order, link transmission bandwidth value and average disk response latency of the candidate path, and calculate its remaining bandwidth capacity and cumulative response latency; set weight coefficients for the remaining bandwidth capacity and cumulative response latency for weighted calculation, sort the candidate paths according to the obtained evaluation values, and select the candidate path with the highest evaluation value as the load balancing adaptation routing path; Step S4: Extract the access request frequency of adjacent storage nodes according to the load balancing adaptation routing path, calculate the access heat gradient value, if the access heat gradient value of two consecutive storage nodes exceeds the upper limit threshold, trigger the data replica migration instruction, if it is lower than the lower limit threshold, trigger the cold data archiving instruction, and generate a storage policy adjustment table. Step S5: Input the storage policy adjustment table into the storage scheduling engine for execution, and after execution, verify whether the decrease in I / O queue depth exceeds the storage load mitigation threshold and whether the bandwidth utilization fluctuation is within the bandwidth stability tolerance range. If the verification results are both yes, then generate a dynamic storage scheduling policy.

2. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, The dynamic state set of the storage link includes the time window access frequency distribution, I / O queue depth increment, and bandwidth utilization standard deviation.

3. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, The set of high-load storage node clusters includes storage node topology identifiers, load correlation coefficient matrices, and time-series rate of change standard deviations.

4. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, The load balancing adaptation routing path includes the remaining bandwidth capacity of the path, the percentile of the storage node latency, and the multi-objective optimization weight coefficients used for weighted calculation.

5. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, In step S1, when collecting data block access request frequency, I / O queue depth and real-time storage bandwidth utilization, the collection interval is adjusted based on the load and traffic changes of the storage nodes.

6. The intelligent scheduling method for computing cluster storage according to claim 1 or 5, characterized in that, Step S1 specifically includes: The physical link topology between the source node and the storage node is obtained through the storage node interface. The link connection table in the storage node interface is called to identify the storage node identifier, interface ID, hop count and bandwidth configuration value, and generate a link path structure dataset. Based on the storage node pairs in the aforementioned link path structure dataset, the data block access request frequency, I / O queue depth, and real-time storage bandwidth utilization are collected, and a storage node link status matching data table is generated by aligning with timestamps. The storage node link status matching data table is invoked, and the access frequency fluctuation value, bandwidth change magnitude and load fluctuation rate are calculated based on the time window slice to generate the dynamic status set of the storage link.

7. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, In step S2, the specific method for calculating the load association strength between storage nodes includes: based on the dynamic state set of the storage link, calculating the mean and standard deviation of the data block access request frequency and the mean and standard deviation of the I / O queue depth of the storage nodes in a continuous time period, generating load state change coefficients after Z-Score normalization of the calculation results, and performing cluster analysis based on the load state change coefficients to generate a set of dense clusters of storage node states, and using the standardized average distance of the storage nodes in the state mapping as the load association strength value.

8. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, In step S3, the formula for calculating the evaluation value is: Evaluation value = W1 × -W2× ; in, The standard bandwidth indicator is calculated using the following formula: , The standard latency index is calculated using the following formula: , This represents the remaining bandwidth capacity of the current path. and These represent the maximum and minimum remaining bandwidth capacity among all candidate paths, respectively. This represents the cumulative response time for the current path. and W1 and W2 are the maximum and minimum cumulative response times among all candidate paths, respectively. W1 and W2 are preset dimensionless weight coefficients, and W1 + W2 = 1.

9. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, In step S4, the formula used to calculate the access request frequency of adjacent storage nodes is: ； in, Represents storage node With storage nodes Access frequency values ​​between Representing the Storage node in this record With storage nodes The number of communications between them Represents storage node With storage nodes The average number of communications. Represents storage node With storage nodes Total number of access records Represents storage node With storage nodes Logical distance.

10. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, In step S4, both the upper and lower thresholds are determined based on the percentiles of the heat gradient values ​​of all storage nodes in the path.

11. The intelligent scheduling method for computing cluster storage as described in claim 1, characterized in that, The dynamic storage scheduling strategy includes queue depth decline rate, bandwidth fluctuation tolerance range, and strategy effectiveness verification conditions.

Images