A storage replica scheduling method and device based on failure probability prediction

Abstract

The application discloses a storage copy scheduling method and device based on failure probability prediction, and relates to the technical field of distributed storage, which comprises the following steps: constructing a candidate rack set of a data unit according to a rack dispersion rule; determining the device-level failure risk probability of the storage devices of the candidate rack set; constructing the rack-level failure risk probability by aggregating the device-level failure risk probability; eliminating the candidate racks with the rack-level failure risk probability greater than a rack risk threshold and arranging the candidate racks in a risk ascending order to determine a priority candidate rack sequence; when the number of racks in the priority candidate rack sequence is greater than or equal to the number of copies to be scheduled of the data unit, selecting target racks in the order of the number of copies to be scheduled; determining the device comprehensive score of the storage devices in the normal state and meeting the device resource feasibility in each target rack; and writing the storage copy into the storage device with the minimum device comprehensive score in each target rack. Based on the above scheme, the reliability of copy scheduling is improved.

Description

Technical Field

[0001] This invention relates to the field of distributed storage technology, and in particular to a storage replica scheduling method and apparatus based on failure probability prediction. Background Technology

[0002] Distributed storage systems typically achieve data redundancy through multiple replicas or erasure coding mechanisms, and combine storage topology to distribute different replicas (or coded fragments) of the same data across multiple nodes to reduce the risk of data unavailability due to correlation failures. Existing replica scheduling schemes are usually designed around "resource utilization and performance balancing," mainly relying on static topology relationships and simple threshold elimination based on the current resource status to complete replica placement. They are difficult to proactively avoid potentially high-failure-risk locations and cannot flexibly adapt to dynamic changes in data and systems, thus urgently requiring improvements in replica scheduling reliability. Summary of the Invention

[0003] This invention provides a storage replica scheduling method and apparatus based on failure probability prediction, which is used to improve the reliability of replica scheduling.

[0004] The first aspect of this invention provides a storage replica scheduling method based on failure probability prediction, comprising:

[0005] For any data unit, construct a candidate rack set for the data unit according to the rack distribution rules;

[0006] For any storage device in the candidate rack set, predict and calibrate the device-level failure risk probability within the prediction time window based on the current device failure-related data of the storage device;

[0007] Based on the topology attribution information, the device-level failure risk probabilities of each device are aggregated to construct the rack-level failure risk probabilities of each candidate rack.

[0008] Remove candidate racks from the candidate rack set whose rack-level failure risk probability is greater than the rack risk threshold and sort them in ascending order of risk to determine the priority candidate rack sequence;

[0009] When the number of racks in the priority candidate rack sequence is greater than or equal to the number of replicas of the data unit to be scheduled, priority candidate racks are selected as target racks in order of the number of replicas to be scheduled.

[0010] For storage devices in each target rack that are in normal condition and meet the equipment resource feasibility requirements, a comprehensive equipment score is determined based on the device-level failure risk probability, rack-level failure risk probability, and equipment resource stress.

[0011] The storage devices with the smallest comprehensive device score in each of the target racks are combined into a target placement device set, and written into the storage copy of the data unit.

[0012] Furthermore, the process of determining the probability of equipment-level failure risk includes:

[0013] The current device failure-related data of the storage device is input into a preset machine learning model to predict the risk within the prediction time window, and the original device failure risk probability is output.

[0014] The original device failure risk probability is calibrated and mapped using a preset calibration mapping function to determine the corresponding device-level failure risk probability.

[0015] Furthermore, the process of determining the feasibility of the equipment resources includes:

[0016] ;

[0017] In the formula, The index representing the storage copy of a data unit. Indicates the storage device index. Indicates the feasibility of equipment resources. Indicates an indicator function, Indicates the available capacity of the equipment. Indicates the copy size. This indicates the required space for the equipment. This indicates the equipment load index. Indicates the device load threshold. This indicates the available bandwidth of the device. This indicates the device bandwidth threshold.

[0018] Furthermore, it also includes:

[0019] When the number of racks in the priority candidate rack sequence is less than the number of replicas of the data unit to be scheduled, a candidate node set for the data unit is constructed according to the node distribution rule.

[0020] Based on the topology attribution information, the device-level failure risk probabilities of each candidate node are aggregated to construct the node-level failure risk probabilities of each candidate node.

[0021] Candidate nodes with a node-level failure risk probability greater than the node risk threshold are removed from the candidate node set and sorted in ascending order of risk to determine the priority candidate node sequence;

[0022] When the number of nodes in the priority candidate node sequence is greater than or equal to the number of replicas to be scheduled, priority candidate nodes are selected as target nodes in order of the number of replicas to be scheduled.

[0023] For storage devices in each of the target nodes that are in normal condition and meet the device resource feasibility requirements, a comprehensive device score is determined based on the device-level failure risk probability, the node-level failure risk probability, and the device resource stress.

[0024] The storage devices with the lowest overall device score among the target nodes are grouped into a target placement device set, and written into the storage copy of the data unit.

[0025] Furthermore, it also includes:

[0026] When the number of nodes in the priority candidate node sequence is less than the number of replicas of the data unit to be scheduled, a candidate device set for the data unit is constructed according to the device distribution rule;

[0027] From the set of candidate devices, candidate devices with a device-level failure risk probability less than or equal to the device risk threshold, in normal condition, and meeting the device resource feasibility requirements are selected as priority candidate devices.

[0028] When the number of priority candidate devices is greater than or equal to the number of replicas to be scheduled, the comprehensive score of the devices is determined based on the device-level failure risk probability and the device resource tension, and priority candidate devices are selected in order of the number of replicas to be scheduled to form a target placement device set and write it into the storage replica of the data unit.

[0029] When the number of priority candidate devices is less than the number of replicas to be scheduled, the number of temporary devices is determined by the difference between the number of replicas to be scheduled and the number of priority candidate devices.

[0030] From the storage devices in the candidate device set whose device-level failure risk probability is greater than the device risk threshold, are in normal condition, and meet the device resource feasibility requirements, the storage devices with the lowest device-level failure risk probability are selected in order of the number of temporary devices to form a temporary placement device set.

[0031] The preferred candidate devices and the set of temporary placement devices are combined to form a target placement device set, which is then written into a storage copy of the data unit.

[0032] Furthermore, it also includes:

[0033] After the data unit is written to a storage copy, any storage device in the target placement device set is monitored in real time. If the failure risk probability meets the risk triggering condition, is in an abnormal state, or / and does not meet the device resource feasibility, then it is determined that the storage adjustment triggering condition is met.

[0034] Add the storage replicas on the storage devices that meet the storage adjustment trigger conditions to the queue to be adjusted, and then proceed to execute the step of constructing a candidate rack set for any data unit according to the rack distribution rules, so as to adjust the storage replicas in the queue to be adjusted to the new storage devices.

[0035] A second aspect of the present invention provides a storage replica scheduling apparatus based on failure probability prediction, applied to implement the storage replica scheduling method based on failure probability prediction as described in any one of the first aspects of the present invention, comprising:

[0036] The candidate rack determination module is used to construct a candidate rack set for any data unit according to the rack distribution rules.

[0037] The device risk determination module is used to predict and calibrate the device-level failure risk probability within the prediction time window for any storage device in the candidate rack set based on the current device failure-related data of the storage device.

[0038] The rack risk aggregation module is used to aggregate the failure risk probabilities of each device according to the topology attribution information and construct the rack-level failure risk probability of each candidate rack.

[0039] The rack screening module is used to remove candidate racks from the candidate rack set whose rack-level failure risk probability is greater than the rack risk threshold and sort them in ascending order of risk to determine the priority candidate rack sequence.

[0040] The rack selection module is used to select priority candidate racks as target racks in order of the number of replicas to be scheduled when the number of racks in the priority candidate rack sequence is greater than or equal to the number of replicas to be scheduled for the data unit.

[0041] The equipment scoring and determination module is used to determine the comprehensive equipment score for storage devices in each target rack that are in normal condition and meet the equipment resource feasibility requirements, based on the equipment-level failure risk probability, rack-level failure risk probability, and equipment resource stress.

[0042] The copy writing module is used to assemble the storage devices with the smallest comprehensive device score in each of the target racks into a target placement device set, and write it into the storage copy of the data unit.

[0043] A computer device provided in a third aspect of the present invention includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the memory replica scheduling method based on failure probability prediction as described in any of the preceding claims.

[0044] The fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the storage copy scheduling method based on failure probability prediction as described in any of the preceding claims.

[0045] The fifth aspect of the present invention provides a computer program product comprising a computer program / instruction, wherein when the computer program / instruction is executed by a processor, it implements the memory replica scheduling method based on failure probability prediction as described in any of the preceding claims.

[0046] As can be seen from the above technical solutions, the present invention has the following advantages:

[0047] The above-described solution of the present invention provides a storage replica scheduling method based on failure probability prediction, comprising: for any data unit, constructing a candidate rack set for the data unit according to rack distribution rules; for any storage device in the candidate rack set, predicting and calibrating the device-level failure risk probability within the prediction time window based on the current device failure-related data of the storage device; aggregating each device-level failure risk probability according to topology attribution information to construct the rack-level failure risk probability of each candidate rack; removing candidate racks with rack-level failure risk probabilities greater than the rack risk threshold from the candidate rack set and sorting them in ascending order of risk to determine a priority candidate rack sequence; when the number of racks in the priority candidate rack sequence is greater than or equal to the number of replicas to be scheduled for the data unit, selecting priority candidate racks as target racks according to the order of the number of replicas to be scheduled; for storage devices in each target rack that are in normal condition and meet the device resource feasibility requirements, determining a comprehensive device score based on the device-level failure risk probability, rack-level failure risk probability, and device resource tension; forming a target placement device set by combining the storage devices with the lowest comprehensive device scores in each target rack and writing them into the storage replicas of the data unit. Based on the above scheme, the device-level failure risk probability within the predicted time window is obtained through calibration, making the risk index comparable, sortable, and thresholdable. The high-risk clustering is reduced based on the constraint of the dispersion rule, and risk threshold filtering and risk sorting are performed on the candidate set for selection. Under the premise of meeting resource constraints, the target device set for storage replicas is determined by joint scoring of risk and resources, which helps to reduce the risk of replica failure and improve the reliability of replica scheduling. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1The flowchart illustrates the steps of a storage replica scheduling method based on failure probability prediction provided in Embodiment 1 of the present invention.

[0050] Figure 2 The overall framework for storage replica scheduling provided in Embodiment 1 of the present invention Figure 1 ;

[0051] Figure 3 The overall framework for storage replica scheduling provided in Embodiment 1 of the present invention Figure 2 ;

[0052] Figure 4 This is a flowchart of the online triggering migration relocation and execution control provided in Embodiment 1 of the present invention;

[0053] Figure 5 This is a schematic diagram of the fault domain hierarchy and risk aggregation provided in Embodiment 1 of the present invention;

[0054] Figure 6 This is a structural block diagram of a storage replica scheduling device based on failure probability prediction provided in Embodiment 2 of the present invention. Detailed Implementation

[0055] This invention provides a storage replica scheduling method and apparatus based on failure probability prediction to improve replica scheduling reliability.

[0056] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0057] It should be noted that in distributed storage systems, data is typically divided into multiple data units (for example, data can be organized into stripe groups, and when using erasure coding schemes, a stripe group can be encoded into multiple code slices) for storage and management. This embodiment addresses the scenario of replica placement and online adjustment of data units in a distributed storage system. To reduce the unavailability risk caused by correlated failures, the physical topology is abstracted into multi-level fault domains. Fault domains include, but are not limited to, topology levels such as storage devices, storage nodes, and racks, and at least one storage device is present. Belongs to storage node Storage nodes Belongs to rack The hierarchy relationship is used to determine the topology path of any storage device; where storage replicas... As a replica object of a data unit, the replica size can be associated. The replica information includes the expected number of replicas or encoding parameters, the existing set of placement locations, and the required distribution and consistency rules; storage devices (e.g., hard disk drives or solid-state drives) are used to hold the replica data, and for any storage device... It can associate device information such as device identifier, device attribute information, device health information, device node information, and device resource status; a storage node (such as a physical server or storage server) is used to host one or more storage devices, and for any storage node... It can associate node information such as node identifier, node resource status, and rack information; while the rack, as a typical physical isolation boundary, is used to characterize the fault correlation under the same power supply, heat dissipation, and network switching environment, for any rack It can be associated with rack information such as rack identifier, rack availability status, and rack maintenance status;

[0058] During replica scheduling, the input dataset required for scheduling decisions is obtained in real time from the configuration management and monitoring system (such as monitoring system, configuration management database, metadata service, operation and maintenance system, or log system). This embodiment does not limit the specific source or collection frequency. Then, structured normalization and consistency verification are performed on the input dataset to obtain a standardized input dataset that can be directly used for candidate generation, risk calculation, and constraint judgment. Specifically, it can be collected according to data needs, including but not limited to:

[0059] 1) Hard drive attribute information: including but not limited to hard drive identification, model, batch, capacity, media type, firmware version, service life, historical bad block count, SMART health information (SMART is a built-in health monitoring tool for hard drives, used to record the operating status of hard drives) and maintenance / isolation status, etc.;

[0060] 2) Topology attribution information: including but not limited to one or more of the following: the node identifier to which the hard drive belongs, the rack identifier to which it belongs, and the higher-level fault domain identifier to which it belongs;

[0061] 3) Risk source data (historical risk sources or synthetic risk sources): including but not limited to historical failure events, maintenance and replacement records, batch risk labels and environmental / temperature anomaly records, or synthetic risk events formed by fusing information from multiple sources;

[0062] 4) Resource status: including but not limited to capacity utilization, remaining available capacity, inbound / outbound network bandwidth usage, input / output load or CPU load and other operating indicators, background migration task usage and concurrent quota, and corresponding threshold policies or reservation policies.

[0063] Please see Figures 1 to 3The present invention provides a storage replica scheduling method based on failure probability prediction, taking a three-level fault domain topology of rack-storage node-storage device as an example, including:

[0064] Step 101: For any data unit, construct a candidate rack set for the data unit according to the rack distribution rules.

[0065] Rack distribution rules refer to rules that require different storage copies of the same data unit to be distributed across different racks.

[0066] A candidate rack set refers to a group of racks that meet the rack distribution rules for a certain data unit, and includes multiple candidate racks.

[0067] It should be noted that when making a placement decision for the storage copy of any data unit, priority is given to ensuring that the rack distribution rule is met in the highest-level rack, thereby obtaining the set of candidate racks to which the data unit belongs.

[0068] Step 102: For any storage device in the candidate rack set, predict and calibrate the device-level failure risk probability within the prediction time window based on the current device failure-related data of the storage device.

[0069] In one specific embodiment of this example, the process of determining the probability of device-level failure risk includes:

[0070] Input the current device failure-related data of the storage device into the preset machine learning model to predict the risk within the prediction time window, and output the original device failure risk probability;

[0071] The original equipment failure risk probability is calibrated and mapped using a preset calibration mapping function to determine the corresponding equipment-level failure risk probability.

[0072] Equipment failure-related data refers to operational observation data that is correlated with equipment failure and can reflect the evolution of storage devices from "health" to "degradation" and then to "failure" to a certain extent. Examples include historical bad block counts, SMART health information, and risk source data. This data can be unimodal or multimodal; this embodiment does not impose any restrictions, as long as it can be used for storage device failure estimation. "Current" refers to the current observation time. .

[0073] The prediction time window refers to a continuous time interval extending into the future from the current observation time during the failure risk prediction process. , Indicates the length of the window.

[0074] The original device failure risk probability refers to the raw quantitative indicator of the likelihood of a storage device failing within a prediction time window. It is the raw output obtained by predicting failure risk based on device failure-related data.

[0075] Device-level failure risk probability refers to a unified quantitative indicator of the likelihood of a storage device experiencing a failure event within a predicted time window. A failure event refers to the storage device's inability to provide normal data read and write services as expected. For example, it can be understood as including at least one of the following: the storage device enters an unavailable state, an unrecoverable media error occurs resulting in data being unreadable, or the device is determined to be in an unrecoverable state that requires it to be taken offline or isolated.

[0076] It should be noted that the current device failure-related data, aligned to a unified time granularity on the storage device side, is used as the prediction input, and the original device failure risk probability oriented towards the prediction time window is used as the prediction output to characterize the probability of a storage device failure event occurring within the prediction time window. Regarding the prediction time window, this embodiment sets at least one prediction time window. When it is necessary to simultaneously cover short-term risk mitigation and medium-to-long-term planning, this embodiment can define multiple long sets of time windows simultaneously, and output the corresponding original device failure risk probability set for the same storage device at the same observation time. The prediction output can be updated on a rolling basis according to the sampling period, keeping the risk characterization synchronized with changes in device status, thereby ensuring that the decision uses the effective risk quantification results at the current moment. In specific implementation, compared to relying solely on static configuration or human experience, machine learning models obtained from learning historical failure samples can be used to quantify and assess device risk at a higher frequency and lower cost, and can output more continuous risk measurements and support failures targeting specific time windows.

[0077] Meanwhile, since different storage devices may have prediction outputs from different sources and with different dimensions, this embodiment uses the original device failure risk probability as the risk output to be calibrated, and maps it to the calibrated output, i.e., the device-level failure risk probability, through a preset calibration mapping function. This process can be expressed as: , Indicates the storage device index. Indicates storage device In window length The probability of original equipment failure risk within the predicted time window. Represents the calibration mapping function. Indicates storage device In window length The device-level failure risk probability within the prediction time window is comparable (i.e., the magnitude can be directly compared between different hard drives) and consistent within the time window (i.e., the probability under the same prediction time window can be used for unified decision-making). This allows for comparisons with the same caliber across different devices, models, and operating stages, and ensures that threshold strategies and scoring strategies have a consistent basis. It can be understood that the calibration mapping satisfies monotonicity constraints to maintain stable risk ranking, and achieves consistent meaning across models, batches, and scenarios through grouping parameters. That is, different calibration parameters are set for different storage device groups, so that the same post-calibration probability expresses a similar risk level in different groups. At the same time, the calibration parameters can be updated according to a preset period to adapt to the drift of operating data distribution and maintain the stability of probability semantics. In addition, this embodiment does not limit the specific implementation method of the calibration mapping function, and can refer to the existing technology for understanding and selection.

[0078] Step 103: Aggregate the failure risk probabilities of each device according to the topology attribution information, and construct the rack-level failure risk probabilities of each candidate rack.

[0079] Rack-level failure risk probability refers to a unified quantitative indicator of the likelihood that a rack will experience a failure event within a predicted time window.

[0080] It should be noted that the topology attribution information at least describes the hierarchical relationship between storage devices, storage nodes, and racks. Since storage devices within the same rack may have related failure characteristics, using only device-level failure risk probabilities may lead to a situation where replicas are "distributed across different devices but still concentrated in high-risk racks." Therefore, this embodiment aggregates device-level failure risk probabilities to the rack level according to topology attribution to obtain rack-level failure risk probabilities. This process can be expressed as: , Indicates rack index, Represents aggregate functions, Indicates rack In window length The probability of rack-level failure risk within the predicted time window;

[0081] It is understood that this embodiment does not restrict the form of the aggregation function, as long as the aggregation result can reflect the risk level of the corresponding node or rack and can be used for sorting or threshold judgment.

[0082] Step 104: Remove candidate racks from the candidate rack set whose rack-level failure risk probability is greater than the rack risk threshold, and sort them in ascending order of risk to determine the priority candidate rack sequence.

[0083] The priority candidate rack sequence refers to an ordered list of candidate racks sorted according to rack-level failure risk probability, including multiple priority candidate racks.

[0084] Rack risk threshold refers to the probability boundary value for filtering rack-level failure risks.

[0085] It should be noted that high-risk filtering is performed on candidate racks in the candidate rack set based on rack risk thresholds to avoid risk clustering. Then, the remaining candidate racks are sorted in ascending order of their rack-level failure risk probability to form a priority candidate rack sequence.

[0086] Step 105: When the number of racks in the priority candidate rack sequence is greater than or equal to the number of data unit replicas to be scheduled, select the priority candidate racks as the target racks according to the order of the number of replicas to be scheduled.

[0087] The number of racks refers to the total number of racks.

[0088] The number of replicas to be scheduled refers to the number of storage replicas that need to be placed in a certain data unit.

[0089] The target rack refers to the rack that is ultimately selected to house the storage copies.

[0090] It should be noted that if the number of racks in the priority candidate rack sequence is greater than or equal to the number of replicas to be scheduled for the data unit, the rack distribution rule can be satisfied. Therefore, multiple priority candidate racks with the number of replicas to be scheduled are selected sequentially from the priority candidate rack sequence as the target racks.

[0091] Step 106: For storage devices in each target rack that are in normal condition and meet the equipment resource feasibility requirements, determine the comprehensive equipment score based on the device-level failure risk probability, rack-level failure risk probability, and equipment resource stress.

[0092] Normal state refers to a non-faulty, non-isolated, and non-maintenance state; in contrast, abnormal state refers to a faulty, isolated, or maintenance state.

[0093] Device resource feasibility refers to whether the storage device has sufficient basic resources to support the storage replicas. For example, it may include capacity constraints, bandwidth constraints, and load constraints. Capacity constraints are usually represented by the available capacity of the device, bandwidth constraints are usually reflected in network bandwidth or disk rebuild bandwidth, and load constraints are used to avoid placing new replicas in high-load locations, which would lead to increased tail latency.

[0094] Device resource stress refers to a quantitative indicator that measures the current resource consumption of storage devices and the degree to which performance bottlenecks are approaching. For example, it may include capacity utilization, load rate and bandwidth utilization. Generally, the higher the value, the more stressful the situation.

[0095] The overall device rating refers to a numerical value obtained by comprehensively quantifying the risks and resources of storage devices from multiple dimensions.

[0096] It should be noted that, for any storage device in a target rack, storage devices in a faulty, isolated, or maintenance state are removed. Resource feasibility screening is performed on the remaining storage devices to obtain a set of resource-feasible candidate devices that meet the resource requirements for placing storage replicas. For the storage devices in the resource-feasible candidate device set, a comparable comprehensive device score is generated based on the associated device-level failure risk probability, rack-level failure risk probability, and device resource stress. It is understood that this embodiment does not limit the comprehensive calculation method of the comprehensive device score, such as weighted summation or multi-objective optimization.

[0097] In one specific embodiment of this example, the process of determining the feasibility of equipment resources includes:

[0098] ;

[0099] In the formula, The index representing the storage copy of a data unit. Indicates the storage device index. Indicates the feasibility of equipment resources. This indicates an indicator function (returns 1 if the condition is met, otherwise returns 0). Indicates the available capacity of the equipment. Indicates the copy size. This indicates the required space for the equipment. This indicates the equipment load index. Indicates the device load threshold. This indicates the available bandwidth of the device. This indicates the device bandwidth threshold.

[0100] For example, the available bandwidth of the device can be represented by indicators such as available network bandwidth during migration / repair, cross-rack bandwidth budget, and link congestion level. The device load can be measured by indicators such as device input / output busyness, node CPU / memory pressure, queue depth, or latency. This embodiment does not limit these.

[0101] It should be noted that the above process defines a resource feasibility judgment function based on three hard constraints: capacity constraint, bandwidth constraint, and load constraint. The function returns 1 (i.e., the device resource feasibility is satisfied) only when all conditions are met simultaneously; otherwise, it returns 0 (i.e., the device resource feasibility is not satisfied).

[0102] Step 107: Combine the storage devices with the smallest comprehensive equipment score in each target rack into a target placement device set and write it into the storage copy of the data unit.

[0103] The target placement device set refers to the set of storage devices that are ultimately selected to place storage copies.

[0104] It should be noted that a lower overall device score indicates better performance. Screening based on the overall device score allows for the use of device-level failure risk probability to mitigate high single-disk risks. Furthermore, when device risks are similar, locations with lower rack risk can be prioritized to reduce the risk of simultaneous replica loss due to related failures. The storage devices with the lowest overall device score in each target rack are selected as the target placement device set, serving as the final destination for replicas. The target placement devices are then written to the metadata service, generating placement / adjustment task entries. Subsequently, the tasks are sent to the task queue or scheduler to enter the execution phase for writing to the storage replicas. Finally, the task execution results, final destination, execution time, and latest resource status are written back to the metadata service. For example, the metadata content may include replica object identifiers, target placement device identifiers and their topology paths, policy version numbers, prediction time window lengths, threshold and weight parameter snapshots, rollback type identifiers (if a rollback occurs), and task status fields.

[0105] In one specific embodiment of this example, it further includes:

[0106] When the number of racks in the priority candidate rack sequence is less than the number of replicas of the data unit to be scheduled, a candidate node set for the data unit is constructed according to the node distribution rule;

[0107] Based on the topology attribution information, the failure risk probabilities of each device are aggregated to construct the node-level failure risk probabilities of each candidate node;

[0108] Remove candidate nodes whose node-level failure risk probability is greater than the node risk threshold from the candidate node set and sort them in ascending order of risk to determine the priority candidate node sequence;

[0109] When the number of nodes in the priority candidate node sequence is greater than or equal to the number of replicas to be scheduled, the priority candidate nodes are selected as the target nodes in order of the number of replicas to be scheduled.

[0110] For storage devices in each target node that are in normal condition and meet the feasibility of device resources, a comprehensive device score is determined based on the device-level failure risk probability, the node-level failure risk probability, and the device resource stress.

[0111] The storage devices with the lowest overall device score among the target nodes are grouped into a target placement device set and written into the storage copy of the data unit.

[0112] Node distribution rules refer to rules that require different storage replicas of the same data unit to be distributed across different storage nodes.

[0113] A candidate node set refers to a group of racks that meet the node distribution rules for a certain data unit, including multiple candidate nodes.

[0114] Node-level failure risk probability refers to a unified quantitative indicator of the likelihood of a storage node experiencing a failure event within a predicted time window. Its aggregation process can be represented as: , Indicates the node index. Represents a node In window length The probability of node-level failure risk within the predicted time window.

[0115] The node risk threshold refers to the probability boundary value used to filter node-level failure risks.

[0116] The priority candidate node sequence refers to an ordered list obtained by sorting candidate nodes according to the node-level failure risk probability, which includes multiple priority candidate nodes.

[0117] The number of nodes refers to the number of storage nodes.

[0118] The target node refers to the storage node that is ultimately selected to place the storage replica.

[0119] In a more specific implementation of this embodiment, it also includes:

[0120] When the number of nodes in the priority candidate node sequence is less than the number of replicas of the data unit to be scheduled, a candidate device set for the data unit is constructed according to the device distribution rules.

[0121] From the candidate equipment set, candidate equipment with a probability of failure of less than or equal to the equipment risk threshold, in normal condition and meeting the equipment resource feasibility requirements are selected as priority candidate equipment.

[0122] When the number of priority candidate devices is greater than or equal to the number of replicas to be scheduled, the comprehensive score of the devices is determined based on the probability of device failure risk and the tension of device resources, and priority candidate devices are selected in order of the number of replicas to be scheduled to form the storage replicas of the target placement device set written to the data unit.

[0123] When the number of priority candidate devices is less than the number of replicas to be scheduled, the number of temporary devices is determined by the difference between the number of replicas to be scheduled and the number of priority candidate devices.

[0124] From the candidate device set, storage devices with the lowest device-level failure risk probability that are in normal condition and meet the device resource feasibility requirements are selected according to the number of temporary devices to form a temporary placement device set.

[0125] The target placement device set is composed of a set of priority candidate devices and a set of temporary placement devices, and is written into a storage copy of the data unit.

[0126] Device distribution rules refer to rules that require different storage copies of the same data unit to be distributed across different storage devices.

[0127] The candidate device set refers to a group of devices that meet the device distribution rules for a certain data unit, including multiple candidate devices.

[0128] Equipment risk threshold refers to the probability boundary value for filtering equipment-level failure risks.

[0129] The number of devices refers to the number of storage devices.

[0130] The number of temporary devices refers to the number of storage devices that are temporarily used to store storage copies.

[0131] A temporary storage device set refers to a collection of storage devices for temporarily storing copies of storage data.

[0132] It should be noted that the distribution rules in this embodiment include rack distribution rules, node distribution rules, and device distribution rules. When prioritizing rack distribution rules, if a target rack with a sufficient number of replicas to be scheduled cannot be selected, rack-level fallback is performed, and a decision is made based on satisfying node distribution rules. If a target node with a sufficient number of replicas to be scheduled cannot be selected, node-level fallback is performed, and device selection is made based on satisfying device distribution rules. If the number of priority candidate devices that meet the requirements is insufficient to meet the number of replicas to be scheduled, the number of temporary devices is determined based on the difference between the number of replicas to be scheduled and the number of priority candidate devices. According to the number of temporary devices, multiple candidate devices that are in normal condition and meet the requirements of device resource feasibility and have a lower probability of device-level failure risk are selected from the candidate devices that do not meet the three requirements to form a temporary placement device set. Finally, the original priority candidate devices and the temporary placement device set are combined to form the target placement device set for writing to replicas. An adjustment mark can be added to the corresponding replica of the temporary placement device set.

[0133] In one specific embodiment of this example, it further includes:

[0134] After the data unit's storage copy is written, any storage device in the target placement device set is monitored in real time. If the failure risk probability meets the risk triggering condition, is in an abnormal state, or / and does not meet the device resource feasibility, then the storage adjustment triggering condition is determined to be met.

[0135] Add the storage replicas on the storage devices that meet the storage adjustment trigger conditions to the queue to be adjusted, and then proceed to execute the step of constructing a candidate rack set for any data unit according to the rack distribution rules, so as to adjust the storage replicas in the queue to the new storage device.

[0136] Risk triggering conditions refer to exceeding a risk threshold or reaching a preset risk increase within a risk period.

[0137] Storage adjustment trigger conditions refer to the conditions that trigger the migration, relocation, or other storage location adjustments of already placed storage copies.

[0138] The queue to be adjusted refers to the queue of storage replicas that currently need to have their storage locations adjusted.

[0139] It should be noted that, for reference Figure 4 After the storage replicas are placed, continuously update the operating status information, resource status information, device-level failure risk probability, node-level failure risk probability, and rack-level failure risk probability of the storage device where the placed storage replicas are located. If the device-level failure risk probability corresponding to the placed storage replicas exceeds the corresponding risk threshold, or / and the node-level failure risk probability or rack-level failure probability exceeds the corresponding risk threshold or reaches the preset risk increase, or / and the resource status does not meet the device resource feasibility (e.g., capacity level exceeds the threshold, load continuously exceeds the threshold, bandwidth budget is insufficient, etc.), or / and the operation and maintenance policy triggers the storage device to be in an abnormal state (e.g., maintenance, isolation, batch blacklisting, etc.). If the corresponding storage replica is added to the adjustment queue, steps 101 to 107 are re-executed to migrate or reposition the corresponding replica, thus supporting the dynamic adjustment capability triggered by online updates. It can be understood that during the replica placement process, concurrency limits can be set for nodes and racks respectively to strictly limit the number of data migrations that occur simultaneously within the same domain. Bandwidth limits and priority scheduling are implemented for migration traffic to ensure that front-end services (online services that directly face end users and are highly sensitive to latency and performance) are given priority. At the same time, the placement results, migration execution results, failure reasons, and new resource status can be written back to the metadata service or state database for subsequent decision iteration and audit traceability.

[0140] Understandably, based on the three-tier fault domain topology of rack-storage node-storage device, to ensure compatibility with the deployment of different systems, please refer to [reference needed]. Figure 5 It can also further include a first fault domain. Second fault domain It is used to characterize related failures over a larger range, such as across racks, network partitions, and power domains. The first-level fault domain is used to represent an isolation boundary one level higher than the rack, where resources (racks / nodes / hard drives) may have stronger correlations. The second-level fault domain is used to represent an even higher isolation boundary and is usually composed of multiple first-level fault domains.

[0141] In this embodiment of the invention, by calibrating to obtain the device-level failure risk probability within the predicted time window, the risk index is made comparable, sortable, and thresholdable. Based on the constraints of the dispersion rule, the high-risk clustering is reduced. Risk threshold filtering and risk ranking are performed on the candidate set for optimization. Under the premise of meeting resource constraints, the target device set for storage replicas is determined by joint scoring of risk and resources. During operation, the risk and resource status are continuously updated to dynamically trigger replica migration / replacement decisions, which helps to reduce replica failure risk, improve replica scheduling reliability, and also improve the availability and stability of the distributed storage system.

[0142] Please see Figure 6 The second embodiment of the present invention provides a storage replica scheduling device based on failure probability prediction, comprising:

[0143] The candidate rack determination module 601 is used to construct a candidate rack set for any data unit according to the rack distribution rules.

[0144] The device risk determination module 602 is used to predict and calibrate the device-level failure risk probability within the prediction time window for any storage device in the candidate rack set based on the current device failure-related data of the storage device.

[0145] The rack risk aggregation module 603 is used to aggregate the failure risk probabilities of each device according to the topology attribution information and construct the rack-level failure risk probabilities of each candidate rack.

[0146] The rack screening module 604 is used to remove candidate racks from the candidate rack set whose rack-level failure risk probability is greater than the rack risk threshold and sort them in ascending order of risk to determine the priority candidate rack sequence.

[0147] The rack selection module 605 is used to select the priority candidate rack as the target rack according to the order of the number of replicas to be scheduled when the number of racks in the priority candidate rack sequence is greater than or equal to the number of replicas to be scheduled for the data unit.

[0148] The equipment scoring and determination module 606 is used to determine the comprehensive equipment score for storage devices in each target rack that are in normal condition and meet the equipment resource feasibility requirements, based on the equipment-level failure risk probability, rack-level failure risk probability, and equipment resource stress.

[0149] The copy writing module 607 is used to assemble the storage devices with the smallest comprehensive device score in each target rack into a target placement device set and write the data unit's storage copy.

[0150] In one specific embodiment of this example, a node-level scheduling module is also included:

[0151] When the number of racks in the priority candidate rack sequence is less than the number of replicas of the data unit to be scheduled, a candidate node set for the data unit is constructed according to the node distribution rule;

[0152] Based on the topology attribution information, the failure risk probabilities of each device are aggregated to construct the node-level failure risk probabilities of each candidate node;

[0153] Remove candidate nodes whose node-level failure risk probability is greater than the node risk threshold from the candidate node set and sort them in ascending order of risk to determine the priority candidate node sequence;

[0154] When the number of nodes in the priority candidate node sequence is greater than or equal to the number of replicas to be scheduled, the priority candidate nodes are selected as the target nodes in order of the number of replicas to be scheduled.

[0155] For storage devices in each target node that are in normal condition and meet the feasibility of device resources, a comprehensive device score is determined based on the device-level failure risk probability, the node-level failure risk probability, and the device resource stress.

[0156] The storage devices with the lowest overall device score among the target nodes are grouped into a target placement device set and written into the storage copy of the data unit.

[0157] In a more specific implementation of this embodiment, a device-level scheduling module is also included:

[0158] When the number of nodes in the priority candidate node sequence is less than the number of replicas of the data unit to be scheduled, a candidate device set for the data unit is constructed according to the device distribution rules.

[0159] From the candidate equipment set, candidate equipment with a probability of failure of less than or equal to the equipment risk threshold, in normal condition and meeting the equipment resource feasibility requirements are selected as priority candidate equipment.

[0160] When the number of priority candidate devices is greater than or equal to the number of replicas to be scheduled, the comprehensive score of the devices is determined based on the probability of device failure risk and the tension of device resources, and priority candidate devices are selected in order of the number of replicas to be scheduled to form the storage replicas of the target placement device set written to the data unit.

[0161] When the number of priority candidate devices is less than the number of replicas to be scheduled, the number of temporary devices is determined by the difference between the number of replicas to be scheduled and the number of priority candidate devices.

[0162] From the candidate device set, storage devices with the lowest device-level failure risk probability that are in normal condition and meet the device resource feasibility requirements are selected according to the number of temporary devices to form a temporary placement device set.

[0163] The target placement device set is composed of a set of priority candidate devices and a set of temporary placement devices, and is written into a storage copy of the data unit.

[0164] In one specific embodiment of this example, an online scheduling module is also included:

[0165] After the data unit's storage copy is written, any storage device in the target placement device set is monitored in real time. If the failure risk probability meets the risk triggering condition, is in an abnormal state, or / and does not meet the device resource feasibility, then the storage adjustment triggering condition is determined to be met.

[0166] Add the storage replicas on the storage devices that meet the storage adjustment trigger conditions to the queue to be adjusted, and then proceed to execute the step of constructing a candidate rack set for any data unit according to the rack distribution rules, so as to adjust the storage replicas in the queue to the new storage device.

[0167] Embodiment 3 of the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the storage copy scheduling method based on failure probability prediction as described in Embodiment 1 of the present invention.

[0168] Embodiment 4 of the present invention also provides a computer-readable storage medium storing a computer program / instruction thereon, wherein when the computer program / instruction is executed by a processor, it implements the steps of the storage replica scheduling method based on failure probability prediction as described in Embodiment 1 of the present invention.

[0169] Embodiment 5 of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the storage replica scheduling method based on failure probability prediction as described in Embodiment 1 of the present invention.

[0170] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0171] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0172] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0173] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0174] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0175] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A storage replica scheduling method based on failure probability prediction, characterized in that, include: For any data unit, construct a candidate rack set for the data unit according to the rack distribution rules; For any storage device in the candidate rack set, predict and calibrate the device-level failure risk probability within the prediction time window based on the current device failure-related data of the storage device; Based on the topology attribution information, the device-level failure risk probabilities of each device are aggregated to construct the rack-level failure risk probabilities of each candidate rack. Remove candidate racks from the candidate rack set whose rack-level failure risk probability is greater than the rack risk threshold and sort them in ascending order of risk to determine the priority candidate rack sequence; When the number of racks in the priority candidate rack sequence is greater than or equal to the number of replicas of the data unit to be scheduled, priority candidate racks are selected as target racks in order of the number of replicas to be scheduled. For storage devices in each target rack that are in normal condition and meet the equipment resource feasibility requirements, a comprehensive equipment score is determined based on the device-level failure risk probability, rack-level failure risk probability, and equipment resource stress. The storage devices with the smallest comprehensive device score in each of the target racks are combined into a target placement device set, and written into the storage copy of the data unit.

2. The storage replica scheduling method based on failure probability prediction according to claim 1, characterized in that, The process of determining the probability of equipment-level failure risk includes: The current device failure-related data of the storage device is input into a preset machine learning model to predict the risk within the prediction time window, and the original device failure risk probability is output. The original device failure risk probability is calibrated and mapped using a preset calibration mapping function to determine the corresponding device-level failure risk probability.

3. The storage replica scheduling method based on failure probability prediction according to claim 1, characterized in that, The process of determining the feasibility of the equipment resources includes: ； In the formula, The index representing the storage copy of a data unit. Indicates the storage device index. Indicates the feasibility of equipment resources. Indicates an indicator function, Indicates the available capacity of the equipment. Indicates the copy size. This indicates the required space for the equipment. This indicates the equipment load index. Indicates the device load threshold. This indicates the available bandwidth of the device. This indicates the device bandwidth threshold.

4. The storage replica scheduling method based on failure probability prediction according to claim 1, characterized in that, Also includes: When the number of racks in the priority candidate rack sequence is less than the number of replicas of the data unit to be scheduled, a candidate node set for the data unit is constructed according to the node distribution rule. Based on the topology attribution information, the device-level failure risk probabilities of each candidate node are aggregated to construct the node-level failure risk probabilities of each candidate node. Candidate nodes with a node-level failure risk probability greater than the node risk threshold are removed from the candidate node set and sorted in ascending order of risk to determine the priority candidate node sequence; When the number of nodes in the priority candidate node sequence is greater than or equal to the number of replicas to be scheduled, priority candidate nodes are selected as target nodes in order of the number of replicas to be scheduled. For storage devices in each of the target nodes that are in normal condition and meet the device resource feasibility requirements, a comprehensive device score is determined based on the device-level failure risk probability, the node-level failure risk probability, and the device resource stress. The storage devices with the lowest overall device score among the target nodes are grouped into a target placement device set, and written into the storage copy of the data unit.

5. The storage replica scheduling method based on failure probability prediction according to claim 4, characterized in that, Also includes: When the number of nodes in the priority candidate node sequence is less than the number of replicas of the data unit to be scheduled, a candidate device set for the data unit is constructed according to the device distribution rule; From the set of candidate devices, candidate devices with a device-level failure risk probability less than or equal to the device risk threshold, in normal condition, and meeting the device resource feasibility requirements are selected as priority candidate devices. When the number of priority candidate devices is greater than or equal to the number of replicas to be scheduled, the comprehensive score of the devices is determined based on the device-level failure risk probability and the device resource tension, and priority candidate devices are selected in order of the number of replicas to be scheduled to form a target placement device set and write it into the storage replica of the data unit. When the number of priority candidate devices is less than the number of replicas to be scheduled, the number of temporary devices is determined by the difference between the number of replicas to be scheduled and the number of priority candidate devices. From the storage devices in the candidate device set whose device-level failure risk probability is greater than the device risk threshold, are in normal condition, and meet the device resource feasibility requirements, the storage devices with the lowest device-level failure risk probability are selected in order of the number of temporary devices to form a temporary placement device set. The preferred candidate devices and the set of temporary placement devices are combined to form a target placement device set, which is then written into a storage copy of the data unit.

6. The storage replica scheduling method based on failure probability prediction according to any one of claims 1-5, characterized in that, Also includes: After the data unit is written to a storage copy, any storage device in the target placement device set is monitored in real time. If the failure risk probability meets the risk triggering condition, is in an abnormal state, or / and does not meet the device resource feasibility, then it is determined that the storage adjustment triggering condition is met. Add the storage replicas on the storage devices that meet the storage adjustment trigger conditions to the queue to be adjusted, and then proceed to execute the step of constructing a candidate rack set for any data unit according to the rack distribution rules, so as to adjust the storage replicas in the queue to be adjusted to the new storage devices.

7. A storage replica scheduling device based on failure probability prediction, characterized in that, The apparatus is used to implement the storage replica scheduling method based on failure probability prediction as described in any one of claims 1-6, the apparatus comprising: The candidate rack determination module is used to construct a candidate rack set for any data unit according to the rack distribution rules. The device risk determination module is used to predict and calibrate the device-level failure risk probability within the prediction time window for any storage device in the candidate rack set based on the current device failure-related data of the storage device. The rack risk aggregation module is used to aggregate the failure risk probabilities of each device according to the topology attribution information and construct the rack-level failure risk probability of each candidate rack. The rack screening module is used to remove candidate racks from the candidate rack set whose rack-level failure risk probability is greater than the rack risk threshold and sort them in ascending order of risk to determine the priority candidate rack sequence. The rack selection module is used to select priority candidate racks as target racks in order of the number of replicas to be scheduled when the number of racks in the priority candidate rack sequence is greater than or equal to the number of replicas to be scheduled for the data unit. The equipment scoring and determination module is used to determine the comprehensive equipment score for storage devices in each target rack that are in normal condition and meet the equipment resource feasibility requirements, based on the equipment-level failure risk probability, rack-level failure risk probability, and equipment resource stress. The copy writing module is used to assemble the storage devices with the smallest comprehensive device score in each of the target racks into a target placement device set, and write it into the storage copy of the data unit.

8. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor causes the processor to perform the steps of the storage replica scheduling method based on failure probability prediction as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the storage replica scheduling method based on failure probability prediction as described in any one of claims 1-6.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the storage replica scheduling method based on failure probability prediction as described in any one of claims 1-6.

Images