Data scheduling method and device of distributed storage system, electronic equipment, storage medium and program product

Abstract

The invention provides a data scheduling method and device of a distributed storage system, electronic equipment, a storage medium and a program product, and relates to the technical field of big data management. The method comprises the following steps: predicting a state change trend of each node based on multi-source state data of the nodes of the distributed storage system; determining an initial allocation scheme and a reward score of the target task to determine a task allocation scheme; and scheduling the data blocks of the distributed storage system based on the task allocation scheme. According to the method, the state change trend of the node is predicted according to the multi-source state data, the state data of the node in multiple different dimensions are fully considered, and accurate prediction of the current state and the future state of the node is realized. The task allocation scheme is determined through the state change trend and the features of the target task, the task allocation scheme is made in advance in combination with the future resource demand change of the node and the features of the to-be-allocated task, and the utilization rate of the node at the current moment and in the future time period can be improved.

Description

Technical Field

[0001] This application relates to the field of big data management technology, and in particular to a data scheduling method, apparatus, electronic device, storage medium and program product for a distributed storage system. Background Technology

[0002] With the rapid development of technologies such as cloud computing, big data, artificial intelligence, and the Internet of Things, the total amount of global data is exploding at an unprecedented rate. Massive, heterogeneous, and highly dynamic data poses severe challenges to the capacity, performance, and reliability of storage systems. Traditional monolithic storage architectures can no longer meet the needs of modern applications. Distributed storage systems, due to their superior scalability, high availability, and cost-effectiveness, have become a critical infrastructure supporting upper-layer applications. In distributed storage systems, data is distributed across multiple physical nodes in the form of data blocks, and data reliability is ensured through a replication mechanism. However, how to scientifically and efficiently schedule tasks such as data access, data migration, and replica repair within the system directly determines the overall system's Quality of Service (QoS), including response latency, throughput, and resource utilization.

[0003] To meet the aforementioned technical requirements, various scheduling methods for distributed storage systems have been proposed in the prior art. These methods are typically based on different optimization objectives and information dimensions. A common method is block-based scheduling, which mainly makes decisions based on static attributes such as the location, size, and access frequency of data blocks. The aim is to schedule computational tasks to the nodes closest to the data source to reduce network transmission overhead.

[0004] However, existing distributed storage systems mostly use linear models or simple threshold-based rules to predict future loads for data scheduling. But the behavior of distributed systems is highly complex and nonlinear, and these traditional prediction models cannot fully capture their inherent laws, resulting in inaccurate predictions of future resource demands, which in turn leads to inaccurate data scheduling in distributed storage systems. Summary of the Invention

[0005] This application provides a data scheduling method, apparatus, electronic device, storage medium, and program product for a distributed storage system, which addresses the shortcomings of inaccurate data scheduling in existing distributed storage systems and improves the accuracy of data scheduling in distributed storage systems.

[0006] This application provides a data scheduling method for a distributed storage system, including: Based on multi-source state data of at least one node in a distributed storage system, predict the state change trend of each node at the current moment and in future time periods. The multi-source state data includes state data of multiple different dimensions. Based on the state change trends of each node and the characteristics of the target task, determine at least one initial allocation scheme for the target task and its reward score. The initial allocation scheme with the highest reward score will be used as the task allocation scheme. The data blocks of the distributed storage system are scheduled based on the task allocation scheme.

[0007] In one embodiment, the reward score for the initial allocation scheme is determined based on the following method: Obtain the execution node for the initial allocation scheme; In the past allocation process of the target task, obtain the actual completion time of the target task for the execution node in the past; The first score is determined based on the error between the actual completion time in the past and the expected completion time of the target task; Obtain the resource utilization rate of each node after the target task is assigned to the execution node; The second score is determined based on the resource utilization rate of each node and the total number of nodes; The first and second scores are weighted and summed to obtain the reward score for the initial allocation scheme.

[0008] In one embodiment, based on multi-source state data from at least one node of a distributed storage system, predicting the state change trend of each node at the current moment and within a future time period includes: Principal component analysis is performed on the multi-source state data to reduce the dimensionality of the multi-source state data and obtain the key features of each node; Input each key feature into the prediction model to obtain the state change trend of each node output by the prediction model; The prediction model is used to extract features from each key feature, obtain the state feature set of each node, perform weighted summation on each state feature set, and correct and activate each weighted summation result to obtain the state change trend of each node.

[0009] In one embodiment, the multi-source state data of a node is obtained in the following manner: The remote direct memory access module of the control node collects various performance indicators of the node at different time points to obtain the node's collected data; Identify abnormal data in the collected data to obtain the identified collected data; The collected data after identification is standardized and abnormal data is repaired to obtain the repaired collected data. The performance indicators of different types of collected data after repair are normalized to obtain multi-source state data of the nodes.

[0010] In one embodiment, scheduling data blocks in a distributed storage system based on a task allocation scheme includes: The distribution configuration information for each data block is determined based on the task allocation scheme. The distribution configuration information includes the number of replicas, placement location, and priority of each data block. Based on the past access records of each data block, obtain the previously popular data blocks, the previously unpopular data blocks, and the access patterns of each data block in future time periods; Based on past hot data blocks, past cold data blocks, and the access patterns and distribution configuration information of each data block in future time periods, each data block is scheduled.

[0011] In one embodiment, after scheduling the data blocks of the distributed storage system based on the task allocation scheme, the method further includes: Obtain the data distribution scheme table of the distributed storage system. The data distribution scheme table includes the placement location of each data block, the processing node, and the expected access frequency. Based on the data distribution scheme table, determine the monitoring indicators and their normal ranges; Collect the values ​​of monitoring indicators for each node, and evaluate the operating status of each node based on the values ​​and normal ranges of the monitoring indicators to obtain the health score of each node. When a node's health score falls below a set score, an alarm message is generated for the node.

[0012] Secondly, this application also provides a data scheduling device for a distributed storage system, characterized in that it includes: The prediction module is used to predict the state change trend of each node at the current time and in the future time period based on the multi-source state data of at least one node in the distributed storage system. The multi-source state data includes state data of multiple different dimensions. The allocation module is used to determine at least one initial allocation scheme and its reward score for the target task based on the state change trend of each node and the characteristics of the target task. The allocation module is also used to select the initial allocation scheme with the highest reward score as the task allocation scheme; The scheduling module is used to schedule the data blocks of the distributed storage system based on the task allocation scheme.

[0013] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a data scheduling method for any of the distributed storage systems described above.

[0014] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a data scheduling method as described above for any of the distributed storage systems.

[0015] This application also provides a computer program product, including a computer program that, when executed by a processor, implements a data scheduling method for any of the distributed storage systems described above.

[0016] This application provides a data scheduling method, apparatus, electronic device, storage medium, and program product for a distributed storage system. Based on multi-source state data of at least one node in the distributed storage system, it predicts the state change trends of each node at the current moment and in future time periods. The multi-source state data includes state data from multiple different dimensions. Based on the state change trends of each node and the characteristics of the target task, it determines at least one initial allocation scheme and its reward score for the target task. The initial allocation scheme with the highest reward score is used as the task allocation scheme. Data blocks in the distributed storage system are scheduled based on the task allocation scheme. This application predicts the state change trends of nodes based on multi-source state data, fully considering the state data of nodes in multiple different dimensions, and achieves accurate prediction of the current and future states of nodes. By determining the task allocation scheme through state change trends and the characteristics of the target task, and combining the future resource demand changes of nodes with the characteristics of the tasks to be allocated, the task allocation scheme can be made in advance, improving the utilization rate of nodes at the current moment and in future time periods. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating the data scheduling method for the distributed storage system provided in this application.

[0019] Figure 2 This is a schematic diagram of the data scheduling device of the distributed storage system provided in this application.

[0020] Figure 3 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] The following is combined with Figures 1-3 This application describes a data scheduling method, apparatus, and electronic device for a distributed storage system.

[0023] Figure 1 This is a flowchart illustrating the data scheduling method for the distributed storage system provided in this application, as shown below. Figure 1 As shown, the data scheduling method of the distributed storage system includes steps S100 to S400, each of which is detailed below.

[0024] S100: Based on multi-source state data of at least one node in a distributed storage system, predict the state change trend of each node at the current moment and in future time periods.

[0025] Multi-source state data includes state data from multiple different dimensions.

[0026] The execution entity of this application embodiment includes a distributed storage system. A distributed storage system refers to a cluster composed of multiple independent computers (called nodes) connected via a network. These nodes work collaboratively to provide a single, unified logical view for storing and accessing data, while their underlying physical storage resources are distributed across the nodes. A distributed storage system has a complex set of distributed software. This software is responsible for managing all nodes and handling all data operations. In a distributed storage system, the functions of the distributed software include data distribution strategy formulation, data redundancy strategy formulation, metadata management, fault detection and recovery, and task scheduling and concurrency control. Data distribution strategy formulation includes determining how many data blocks a newly written data should be divided into and on which nodes these data blocks should be stored. Data redundancy strategy formulation includes determining whether data is guaranteed to be reliable using multiple replicas or erasure coding. Metadata management includes recording information such as the location, version, and replica of each data block. Fault detection and recovery includes continuously monitoring the health status of all nodes and data; once a node failure or data corruption is detected, it can automatically perform data recovery or migration without affecting upper-layer services. Task scheduling and concurrency control are used to efficiently handle concurrent read and write requests from a large number of clients. Distributed storage systems use software to organize a bunch of independent, ordinary computers, and through complex scheduling and management mechanisms, they work together to achieve a storage service that is more powerful, more reliable, and more scalable than any single computer.

[0027] A distributed storage system comprises at least one node. A node includes compute nodes and storage nodes. Compute nodes are used to perform calculations on the data. Storage nodes are used to store the data. In a distributed storage system, a node is the smallest unit of work that carries data and participates in cluster collaboration. A node can be a physical server, or a storage process or service instance running on a server; multiple nodes are interconnected through a network to collaboratively complete data sharding, replication, location, read / write operations, and fault recovery.

[0028] Multi-source state data is obtained by collecting different types of performance metrics from various nodes of a distributed storage system from multiple components or multiple dimensions.

[0029] Multi-source state data includes Central Processing Unit (CPU) utilization, memory usage, Graphics Processing Unit (GPU) status, process load, storage utilization, data block access frequency, and replica status.

[0030] The state change trend includes the current state of each node and the expected future state change trend. Node state includes data such as node load, hardware health (e.g., CPU / memory utilization), and replica distribution. Based on the multi-source state data of each node in the distributed storage system, the state change trend of each node is predicted.

[0031] S200: Based on the state change trends of each node and the characteristics of the target task, determine at least one initial allocation scheme for the target task and its reward score.

[0032] The target task includes the tasks to be assigned. Characteristics of the target task include its type, size, access locality, and location constraints. The target task may involve storing, copying, or restoring data.

[0033] Based on the state change trends of each node and the characteristics of the target task, a comprehensive state vector is constructed. At least one possible task allocation action is defined based on this comprehensive state vector. Each task allocation action corresponds to an initial allocation scheme. For example, the initial allocation scheme is to allocate the target task to computing node B for execution.

[0034] Based on the comprehensive state vector, at least one possible task allocation action is defined. Optionally, at least one valid node capable of executing the target task is selected from the distributed storage system based on the comprehensive state vector. Each valid node is traversed, and the target task is sequentially allocated to the traversed valid nodes to obtain at least one initial allocation scheme.

[0035] Define a reward function for the initial allocation scheme, and calculate a reward score for each initial allocation scheme based on the reward function. The reward score is used to characterize the quality of the initial allocation scheme; the higher the reward score, the better the initial allocation scheme.

[0036] S300: Use the initial allocation scheme with the highest reward score as the task allocation scheme.

[0037] The initial allocation scheme that yields the highest reward score is used as the task allocation scheme. The task allocation scheme specifies the matching execution nodes for the target task. The target task is then assigned to the matching execution nodes according to the task allocation scheme.

[0038] S400: Schedules data blocks in a distributed storage system based on a task allocation scheme.

[0039] In a distributed storage system, large files are divided into fixed-size data blocks. The task allocation scheme determines the execution nodes for the target tasks. The data blocks are the processing objects of the target tasks.

[0040] The task allocation scheme is used to schedule data blocks in a distributed storage system, including determining the placement location, access route, migration, and number of replicas of the data blocks according to the task allocation scheme.

[0041] For example, the target task is to read data block A. The task allocation scheme is for node 2 to execute the reading of data block A. Then, according to the task allocation scheme, data block A will be scheduled to node 2, or a location near node 2.

[0042] The data scheduling method for a distributed storage system provided in this application predicts the state change trends of each node in the current and future time periods based on multi-source state data of at least one node in the distributed storage system. The multi-source state data includes state data from multiple different dimensions. Based on the state change trends of each node and the characteristics of the target task, at least one initial allocation scheme and its reward score for the target task are determined. The initial allocation scheme with the highest reward score is used as the task allocation scheme. Data blocks in the distributed storage system are scheduled based on the task allocation scheme. This application predicts the state change trends of nodes based on multi-source state data, fully considering the state data of nodes in multiple different dimensions, and achieves accurate prediction of the current and future states of nodes. By determining the task allocation scheme through state change trends and the characteristics of the target task, and combining the future resource demand changes of nodes with the characteristics of the tasks to be allocated, the task allocation scheme can be made in advance, improving the utilization rate of nodes in the current and future time periods.

[0043] Based on the above embodiments, the reward score of the initial allocation scheme is determined in the following manner: Obtain the execution node for the initial allocation scheme; In the past allocation process of the target task, obtain the actual completion time of the target task for the execution node in the past; The first score is determined based on the error between the actual completion time in the past and the expected completion time of the target task; Obtain the resource utilization rate of each node after the target task is assigned to the execution node; The second score is determined based on the resource utilization rate of each node and the total number of nodes; The first and second scores are weighted and summed to obtain the reward score for the initial allocation scheme.

[0044] The execution node is the node used to execute the target task in the initial allocation scheme. Obtain the execution nodes for each initial allocation scheme. Query the previous allocation records of the target task to determine if there is a record of assigning the target task to this execution node. If there is a previous record of assigning the target task to this execution node, obtain the execution node's previous actual completion time for the target task. For example, in the previous period, the target task was assigned to execution node 1. In the previous period, execution node 1's actual completion time for the target task was 1 minute. In the current period, the target task is also assigned to execution node 1, so execution node 1's previous actual completion time for the target task is 1 minute. Determine the first score based on the error between the previous actual completion time and the expected completion time of the target task.

[0045] Optionally, query the target task's past assignment records. If no record exists of assigning the target task to this execution node, then the first score is the set score.

[0046] Node resource utilization refers to the degree to which various resources are effectively used by a node within a specific time window. Optionally, node resource utilization can be calculated as a percentage or quantile of the node's actual resource usage (e.g., storage resources) relative to the total available resources. Optionally, node resource utilization includes the comprehensive usage of at least one key resource such as node capacity, computing resources, network resources, and input / output resources. The resource utilization of each node is obtained after the target task is assigned to the execution nodes. A second score is determined based on the resource utilization of all nodes in the distributed storage system and the total number of nodes. The first and second scores are weighted and summed to obtain the reward score for the initial allocation scheme. The formula for calculating the reward score for the initial allocation scheme is as follows.

[0047] ; in, The comprehensive state vector (determined by the state change trends of each node and the characteristics of the target task) is: The initial allocation scheme is as follows: Reward points, The weight of the first score, The first score, As the weight of the second score, The actual time taken to complete the target task in the past. The expected completion time for the target task. For the first Resource utilization rate of each node The total number of nodes. The second score, This is the time sensitivity coefficient.

[0048] Optionally, the initial allocation scheme can be for a single target task or for multiple target tasks. When the initial allocation scheme is for multiple target tasks, the reward score for the allocation scheme for each target task is calculated. The total reward score of the initial allocation scheme is determined by summing the reward scores of all target task allocation schemes. The initial allocation scheme with the highest total reward score is selected as the final task allocation scheme.

[0049] This application embodiment combines the error between the actual completion time and the expected completion time in the past, as well as the resource utilization rate of the nodes to determine the reward score. It fully considers the past task completion status of the execution nodes and the current usage status of each node, and realizes the accurate calculation of the reward score.

[0050] Based on the above embodiments, the multi-source state data of the nodes is obtained in the following manner: The remote direct memory access module of the control node collects various performance indicators of the node at different time points to obtain the node's collected data; Identify abnormal data in the collected data to obtain the identified collected data; The collected data after identification is standardized and abnormal data is repaired to obtain the repaired collected data. The performance indicators of different types of collected data after repair are normalized to obtain multi-source state data of the nodes.

[0051] Install and start a Remote Direct Memory Access (DMA) module on each node of the distributed storage system. The DMA module collects various performance metrics of the node at different points in time from the operating system level, such as CPU utilization, memory usage, GPU status, process load, storage space utilization, data block access frequency, replica status, etc., to obtain the node's collected data.

[0052] The collected data is transmitted to the central management system of the distributed storage system via an encrypted communication protocol to obtain the collected data. The collected data reflects the overall health status of the nodes at different points in time. The encrypted communication protocol includes Transport Layer Security (TLS).

[0053] Abnormal data in the collected data is identified, resulting in identified collected data. The collected data is then imported into the preprocessing module for preliminary inspection, identifying abnormal data and obtaining identified collected data. Abnormal data includes data with obvious errors.

[0054] The collected data after labeling is standardized. A Z-score is calculated for each performance metric value. The Z-score represents the distance between a single performance metric value and its mean. A Z-score greater than 0 indicates the performance metric value is above the mean. A Z-score equal to 0 indicates the performance metric value is exactly the same as the mean. A Z-score less than 0 indicates the performance metric value is below the mean. The absolute value of the Z-score for each performance metric value is compared to a set threshold. If the absolute value of the Z-score for a performance metric value is greater than the set threshold, the performance metric value deviates significantly from the mean, and this performance metric value is identified as outlier data.

[0055] Anomaly repair is performed on the standardized collected data to obtain repaired collected data. Anomaly repair includes using time series interpolation. Time series interpolation utilizes the temporal continuity of the standardized collected data and the correlation of nearby observations to estimate reasonable values ​​for the locations of anomalies using an interpolation model, thereby restoring the continuity of the standardized collected data.

[0056] The different performance metrics of the repaired collected data are normalized to obtain the node state data. Normalization includes min-max normalization. Each performance metric value is linearly scaled to a uniform range, for example, [0, 1], based on its minimum and maximum values. This eliminates dimensional differences between different performance metrics, making them more comparable and avoiding interference from certain large-range performance metrics during subsequent unified processing of multi-source state data. The specific calculation formula for min-max normalization is as follows.

[0057] ; in, These are the normalized performance index values. This represents the minimum value of the performance metric. The maximum value of the performance index. These are performance index values.

[0058] This application achieves automated collection of node performance indicators through a remote direct memory access module. By performing abnormal data identification, standardization, repair, and normalization on the collected data, multi-source state data of the nodes is obtained, realizing the continuity, uniformity, and comparability of multi-source state data, which is conducive to improving the accuracy of subsequent prediction of node state change trends based on multi-source state data.

[0059] Based on the above embodiments, predicting the state change trend of each node in the current time and future time periods based on multi-source state data of at least one node in a distributed storage system includes the following steps: Principal component analysis is performed on the multi-source state data to reduce the dimensionality of the multi-source state data and obtain the key features of each node; Input each key feature into the prediction model to obtain the state change trend of each node output by the prediction model; The prediction model is used to extract features from each key feature, obtain the state feature set of each node, perform weighted summation on each state feature set, and correct and activate each weighted summation result to obtain the state change trend of each node.

[0060] Principal component analysis (PCA) uses orthogonal linear transformations to rotate the high-dimensional features of the original data onto a new set of coordinate axes (principal components). These axes are ordered from largest to smallest variance, so that the projected data retains as much information as possible in as few dimensions as possible.

[0061] Principal component analysis (PCA) is performed on the multi-source state data to reduce its dimensionality and obtain the key features of each node. The specific calculation formula for PCA is as follows.

[0062] ; in, These are the key features of the nodes obtained after dimensionality reduction. The matrix consists of principal components. This refers to the multi-source state data of the nodes. This is the mean vector of the collected data from the nodes.

[0063] The prediction model is trained based on a neural network model using deep learning algorithms, using the key features of node samples and the labels of the node sample state change trends.

[0064] The prediction model consists of an input layer, multiple hidden layers, and an output layer. Each hidden layer contains a certain number of neurons. The prediction model introduces a nonlinear transformation using an activation function (e.g., the ReLU activation function) and incorporates a dropout layer to randomly discard some neurons.

[0065] The prediction model extracts key features from each node to obtain a state feature set for each node. It then performs a weighted summation of the state features in each node's state feature set to obtain the weighted summation result for each node. The weighted summation results for each node are then corrected and activated to obtain the state change trend of each node. The formula for calculating the state change trend of a node is as follows.

[0066] ; in, This represents the trend of node state changes. The activation function of the output layer. This is the weight matrix of the output layer. As a key feature, This is the set of state features after processing through L hidden layers. This is the bias term for the output layer, used to correct the weighted summation result.

[0067] This application utilizes principal component analysis (PCA) on multi-source state data to extract key features, eliminating the complexity and redundancy of the data. This allows subsequent prediction models to focus feature extraction on representative key features, improving the accuracy of state change trend prediction. Furthermore, by extracting, weighting, and correcting / activating key features, this application achieves automated prediction of node state change trends, avoiding errors inherent in human prediction.

[0068] Based on the above embodiments, scheduling each data block of the distributed storage system according to the task allocation scheme includes the following steps: The distribution configuration information for each data block is determined based on the task allocation scheme. The distribution configuration information includes the number of replicas, placement location, and priority of each data block. Based on the past access records of each data block, obtain the previously popular data blocks, the previously unpopular data blocks, and the access patterns of each data block in future time periods; Based on past hot data blocks, past cold data blocks, and the access patterns and distribution configuration information of each data block in future time periods, each data block is scheduled.

[0069] A task allocation scheme can be a final allocation scheme for a single target task or a final allocation scheme for multiple target tasks. The task allocation scheme lists the specific execution nodes to which the target tasks should be assigned and their expected completion times.

[0070] The distribution configuration information for each data block is determined based on the task allocation scheme. This configuration information includes the number of replicas, placement location, and priority of each data block. For example, the priority of the data blocks corresponding to the target tasks is set according to their expected completion times. The number of replicas and their placement location (on or near the execution node) of the data blocks are determined based on the execution node to which the target tasks are assigned.

[0071] Identify historically frequently accessed and historically infrequently accessed data blocks in a distributed storage system. Use clustering algorithms to cluster data blocks based on their historical access frequency, thus identifying historically frequently accessed and infrequently accessed data blocks. Historically frequently accessed data blocks include data that was accessed frequently within a historical time period. Historically infrequently accessed data blocks include data that was accessed less frequently within a historical time period.

[0072] By using a time series model, the past access records of each data block are processed using Long Short-Term Memory (LSTM) to predict the access patterns of each data block in the distributed storage system over future time periods. Past access records include the access time, frequency, and access nodes of the data block in previous time periods.

[0073] Based on historically frequently accessed and infrequently accessed data blocks, as well as the access patterns and distribution configurations of each data block over future time periods, data blocks are scheduled. The access patterns of each data block over future time periods are analyzed to construct a heatmap reflecting the access frequency of data blocks on each node. The heatmap includes the access frequency value of data blocks on each node. The formula for calculating the access frequency value is as follows.

[0074] ; in, For the first The first node at the nth node The access frequency value of each data block. As a weighting factor for access frequency, In order to be in Time of day The first node at the nth node The access frequency of each data block for Time of day The first node at the nth node The time interval at which each data block is accessed. This is the weighting factor for the time interval.

[0075] Access popularity values ​​are used to determine the access frequency of data blocks in future time periods. A popularity threshold is set, and data blocks with high or low access frequency in future time periods are identified based on this threshold. For example, if the access popularity value of a data block is greater than or equal to the threshold, the data block is identified as high-frequency data in the future time period. If the access popularity value of a data block is less than the threshold, the data block is identified as low-frequency data in the future time period.

[0076] Based on historically frequently accessed and less frequently accessed data blocks, as well as future periods of frequently accessed and less frequently accessed data, and the distribution configuration information of data blocks, data blocks in a distributed storage system are scheduled. For example, historically frequently accessed data blocks and future periods of frequently accessed data are placed closer to the nodes or users accessing that data to reduce data transmission distance, lower network latency, and thus improve data block access efficiency.

[0077] Optionally, the scheduling of data blocks also includes: placing previously popular data blocks and future popular data on nodes with lower load and lower network latency. For previously unpopular data blocks and future unpopular data, a uniform distribution strategy is adopted to distribute the unpopular data evenly among all nodes.

[0078] Optionally, the scheduling data block may also include: for hot data in future time periods, loading it into the cache in advance during idle periods to improve response speed.

[0079] Optionally, scheduling data blocks also includes: adjusting the number and location of data replicas in a timely manner based on node health status and load conditions to ensure high availability and fault tolerance of data blocks.

[0080] After scheduling the data blocks, a data distribution scheme table for all data blocks and their corresponding nodes is output. The data distribution scheme table details the specific nodes where each data block should be placed, as well as the expected access frequency.

[0081] This application determines the distribution configuration information of data blocks by combining a task allocation scheme with the allocation status of target tasks and expected completion times, thus fully considering the allocation requirements of target tasks. This application schedules data blocks based on historically popular and unpopular data blocks, future access patterns of data blocks, and their distribution configuration information. This approach considers not only the past access frequency of data blocks but also their future access frequency, achieving accurate scheduling of data blocks.

[0082] Based on the above embodiments, after scheduling the data blocks of the distributed storage system according to the task allocation scheme, the following steps are also included: Obtain the data distribution scheme table of the distributed storage system. The data distribution scheme table includes the placement location of each data block, the processing node, and the expected access frequency. Based on the data distribution scheme table, determine the monitoring indicators and their normal ranges; Collect the values ​​of monitoring indicators for each node, and evaluate the operating status of each node based on the values ​​and normal ranges of the monitoring indicators to obtain the health score of each node. When a node's health score falls below a set score, an alarm message is generated for the node.

[0083] After scheduling data blocks in the distributed storage system, a data distribution scheme table for all data blocks is obtained. This table includes the placement location of each data block, the processing node, and the expected access frequency.

[0084] Based on the data distribution scheme table, a real-time monitoring and feedback adjustment mechanism is adopted to track the operational status of each node. The monitoring system is configured according to the data distribution scheme table, defining the monitoring indicators to be monitored. Optionally, monitoring indicators include core operational metrics such as node load rate, data transmission latency, task completion efficiency, and data access frequency. The normal range for each monitoring indicator is determined based on the data distribution scheme table. Furthermore, the monitoring frequency and data collection cycle are determined based on the data distribution scheme table.

[0085] According to the data collection cycle, the values ​​of monitoring indicators for each node are collected. Based on the values ​​of the monitoring indicators and their normal ranges, the operating status of each node is evaluated, and a health score is obtained for each node. Optionally, when there are multiple monitoring indicators, the value of each monitoring indicator is scored according to its normal range. The scores of all monitoring indicators for a node are weighted and summed to obtain the health score of that node.

[0086] Optionally, a machine learning model or a random forest model can be used to evaluate the operational status of each node in the current distributed storage system. The monitoring metric values ​​and their normal ranges for each node are input into the machine learning model to obtain the health score for each node. The formula for calculating the node's health score is as follows.

[0087] ; in, For the first Health score of each node For the first The values ​​of the monitoring metrics for each node. It is a machine learning model.

[0088] When a node's health score falls below a set threshold, an anomaly is considered to have occurred, triggering an alarm mechanism. The administrator is notified, and a warning message is sent via automated tools. Simultaneously, the time, location, and relevant parameters of the anomaly are recorded. When a node's health score is greater than or equal to the set threshold, the node is considered to be operating normally.

[0089] The data scheduling device for the distributed storage system provided in this application is described below. The data scheduling device for the distributed storage system described below can be referred to in correspondence with the data scheduling method for the distributed storage system described above.

[0090] like Figure 2 As shown, this application also provides a data scheduling device for a distributed storage system, comprising: Prediction module 201 is used to predict the state change trend of each node at the current time and in the future time period based on the multi-source state data of at least one node in the distributed storage system. The multi-source state data includes state data of multiple different dimensions. The allocation module 202 is used to determine at least one initial allocation scheme and its reward score for the target task based on the state change trend of each node and the characteristics of the target task. The allocation module 202 is also used to select the initial allocation scheme with the highest reward score as the task allocation scheme; The scheduling module 203 is used to schedule the data blocks of the distributed storage system based on the task allocation scheme.

[0091] The data scheduling device for a distributed storage system provided in this application predicts the state change trends of each node in the current and future time periods based on multi-source state data of at least one node in the distributed storage system. The multi-source state data includes state data from multiple different dimensions. Based on the state change trends of each node and the characteristics of the target task, at least one initial allocation scheme and its reward score for the target task are determined. The initial allocation scheme with the highest reward score is used as the task allocation scheme. Data blocks in the distributed storage system are scheduled based on the task allocation scheme. This application predicts the state change trends of nodes based on multi-source state data, fully considering the state data of nodes in multiple different dimensions, and achieves accurate prediction of the current and future states of nodes. By determining the task allocation scheme through state change trends and the characteristics of the target task, and combining the future resource demand changes of nodes with the characteristics of the tasks to be allocated, the task allocation scheme can be made in advance, thereby improving the utilization rate of nodes in the current and future time periods.

[0092] In one embodiment, the allocation module 202 is configured to: obtain the execution nodes of the initial allocation scheme; obtain the actual completion time of the execution nodes for the target task in the past allocation process of the target task; determine a first score based on the error between the actual completion time and the expected completion time of the target task; obtain the resource utilization rate of each node after the target task is allocated to the execution nodes; determine a second score based on the resource utilization rate of each node and the total number of nodes; and perform a weighted summation of the first score and the second score to obtain the reward score of the initial allocation scheme.

[0093] In one embodiment, the allocation module 202 is used to: perform principal component analysis on each multi-source state data to reduce the dimensionality of each multi-source state data and obtain the key features of each node; input each key feature into the prediction model to obtain the state change trend of each node output by the prediction model; wherein, the prediction model is used to extract features from each key feature to obtain the state feature set of each node, perform weighted summation on each state feature in each state feature set, and perform correction and activation on each weighted summation result to obtain the state change trend of each node.

[0094] In one embodiment, the prediction module 201 is further configured to: control the remote direct memory access module of the node to collect various performance indicators of the node at different time points to obtain the node's collected data; identify abnormal data in the collected data to obtain identified collected data; perform standardization processing and abnormal data repair on the identified collected data to obtain repaired collected data; and perform normalization processing on different performance indicators of the repaired collected data to obtain multi-source state data of the node.

[0095] In one embodiment, the scheduling module 203 is further configured to: determine the distribution configuration information of each data block based on the task allocation scheme, the distribution configuration information including the number of replicas, placement location and priority of each data block; obtain previously popular data blocks, previously unpopular data blocks and the access patterns of each data block in the future time period based on the previous access records of each data block; and schedule each data block based on the previously popular data blocks, previously unpopular data blocks, the access patterns of each data block in the future time period and the distribution configuration information.

[0096] In one embodiment, the data scheduling device of the distributed storage system further includes a monitoring module, which is used to: obtain a data distribution scheme table of the distributed storage system, the data distribution scheme table including the placement location, processing node, and expected access frequency of each data block; determine monitoring indicators and their normal ranges based on the data distribution scheme table; collect the values ​​of monitoring indicators of each node; evaluate the operating status of each node based on the values ​​and normal ranges of the monitoring indicators to obtain a health score for each node; and generate an alarm message for the node when the health score of a node is lower than a set score.

[0097] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, communications interface 320, and memory 330 communicate with each other via the communication bus 340. The processor 310 can invoke logical instructions in the memory 330 to execute a data scheduling method for the distributed storage system. This method includes: predicting the state change trends of each node in the current time and future time periods based on multi-source state data of at least one node in the distributed storage system, wherein the multi-source state data includes state data of multiple different dimensions; determining at least one initial allocation scheme and its reward score for the target task based on the state change trends of each node and the characteristics of the target task; selecting the initial allocation scheme with the highest reward score as the task allocation scheme; and scheduling the data blocks of the distributed storage system based on the task allocation scheme.

[0098] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the data scheduling method for the distributed storage system provided by the above methods. The method includes: predicting the state change trend of each node in the current time and future time period based on multi-source state data of at least one node in the distributed storage system, wherein the multi-source state data includes state data of multiple different dimensions; determining at least one initial allocation scheme and its reward score for the target task based on the state change trend of each node and the characteristics of the target task; taking the initial allocation scheme with the largest reward score as the task allocation scheme; and scheduling each data block of the distributed storage system based on the task allocation scheme.

[0100] Furthermore, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the data scheduling method for the distributed storage system provided by the above methods. The method includes: predicting the state change trend of each node in the current time and future time period based on multi-source state data of at least one node in the distributed storage system, wherein the multi-source state data includes state data of multiple different dimensions; determining at least one initial allocation scheme and its reward score for the target task based on the state change trend of each node and the characteristics of the target task; using the initial allocation scheme with the largest reward score as the task allocation scheme; and scheduling each data block of the distributed storage system based on the task allocation scheme.

[0101] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0102] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data scheduling method for a distributed storage system, characterized in that, include: Based on multi-source state data of at least one node in a distributed storage system, predict the state change trend of each node at the current time and in a future time period. The multi-source state data includes state data of multiple different dimensions. Based on the state change trends of each node and the characteristics of the target task, at least one initial allocation scheme and its reward score for the target task are determined. The initial allocation scheme with the highest reward score will be used as the task allocation scheme. The task allocation scheme is used to schedule the data blocks of the distributed storage system.

2. The data scheduling method for a distributed storage system according to claim 1, characterized in that, The reward points for the initial allocation scheme are determined based on the following method: Obtain the execution node of the initial allocation scheme; In the past allocation process of the target task, the actual completion time of the execution node for the target task in the past is obtained; A first score is determined based on the error between the actual completion time in the past and the expected completion time of the target task. Obtain the resource utilization rate of each node after the target task is assigned to the execution node; The second score is determined based on the resource utilization rate of each node and the total number of nodes. The first score and the second score are weighted and summed to obtain the reward score of the initial allocation scheme.

3. The data scheduling method for a distributed storage system according to claim 1, characterized in that, The prediction of the state change trend of each node in the current time and future time period based on the multi-source state data of at least one node in the distributed storage system includes: Principal component analysis is performed on each of the multi-source state data to reduce the dimensionality of each of the multi-source state data and obtain the key features of each node; Input each of the key features into the prediction model and obtain the state change trend of each node output by the prediction model; The prediction model is used to extract features from each of the key features, obtain the state feature set of each node, perform weighted summation on each state feature in each state feature set, and correct and activate each weighted summation result to obtain the state change trend of each node.

4. The data scheduling method for a distributed storage system according to claim 1, characterized in that, The multi-source state data of the nodes is obtained in the following way: The remote direct memory access module controlling the node collects various performance indicators of the node at different time points to obtain the collected data of the node; Abnormal data in the collected data is identified to obtain identified collected data; The collected data after the identification is standardized and abnormal data is repaired to obtain the repaired collected data; The performance indicators of different types of the repaired collected data are normalized to obtain the multi-source state data of the node.

5. The data scheduling method for a distributed storage system according to claim 1, characterized in that, The scheduling of data blocks in the distributed storage system based on the task allocation scheme includes: Based on the task allocation scheme, the distribution configuration information of each data block is determined, and the distribution configuration information includes the number of replicas, placement location, and priority of each data block; Based on the previous access records of each data block, obtain the previously popular data blocks, the previously unpopular data blocks, and the access patterns of each data block in future time periods; Based on the previously popular data blocks, the previously unpopular data blocks, the access patterns of each data block in future time periods, and the distribution configuration information, each data block is scheduled.

6. The data scheduling method for a distributed storage system according to claim 1, characterized in that, After scheduling the data blocks of the distributed storage system based on the task allocation scheme, the process further includes: Obtain the data distribution scheme table of the distributed storage system, which includes the placement location, processing node, and expected access frequency of each data block; Based on the data distribution scheme table, determine the monitoring indicators and their normal ranges; The values ​​of monitoring indicators for each node are collected, and the operating status of each node is evaluated based on the values ​​of the monitoring indicators and the normal range to obtain a health score for each node. When the health score of the node is lower than the set score, an alarm message is generated for the node.

7. A data scheduling device for a distributed storage system, characterized in that, include: The prediction module is used to predict the state change trend of each node at the current moment and in a future time period based on the multi-source state data of at least one node in the distributed storage system. The multi-source state data includes state data of multiple different dimensions. The allocation module is used to determine at least one initial allocation scheme and its reward score for the target task based on the state change trend of each node and the characteristics of the target task. The allocation module is also used to select the initial allocation scheme with the highest reward score as the task allocation scheme. The scheduling module is used to schedule the data blocks of the distributed storage system based on the task allocation scheme.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the data scheduling method of the distributed storage system as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the data scheduling method of the distributed storage system as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the data scheduling method of the distributed storage system as described in any one of claims 1 to 6.

Images