Storage node load balancing method and device, equipment and storage medium

Abstract

The invention discloses a storage node load balancing method and device, equipment and a storage medium, and relates to the technical field of load balancing, and the method comprises the steps: carrying out the real-time monitoring of the state of each storage node, and collecting the real-time load information; predicting the load trend of each storage node based on the real-time load information and the historical load data of each storage node to obtain a corresponding prediction result; and adjusting each storage node and a corresponding data access strategy according to the real-time load information and the prediction result so as to respond to the related data request based on the data copy in the adjusted storage node. Therefore, the storage nodes and the data access strategy can be adjusted in combination with the real-time load information of each storage node and the prediction result of the corresponding load trend; the load change trend of each storage node can be accurately mastered, the storage nodes and access strategies are dynamically optimized, the load balancing capacity of the storage nodes can be improved, and diversified service requirements are met.

Description

Technical Field

[0001] The present invention relates to the technical field of load balancing, and particularly relates to a method, device, equipment and storage medium for load balancing of storage nodes. Background Art

[0002] Currently, commonly used distributed storage systems (such as financial transactions, log processing, and video streaming) usually adopt static or simple dynamic load balancing mechanisms, such as consistent hashing, random distribution, polling, etc. to distribute data or access requests to each storage node. However, these methods cannot perceive the load changes of nodes in real time, lack an effective response mechanism for dynamic load fluctuations in high-throughput scenarios, and are prone to causing some nodes to be overloaded due to frequent hot access, bandwidth exhaustion, or I / O (Input / Output) saturation, while other nodes have low loads, resulting in unbalanced utilization of system resources. In addition, although some systems support manual data migration or policy adjustment to alleviate load problems, such operations have untimely responses and high costs, and cannot meet the requirements of high-real-time services.

[0003] Therefore, how to improve the load balancing effect of storage nodes is a problem to be solved in this field. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a method, device, equipment and storage medium for load balancing of storage nodes, which can accurately grasp the load change trend of each storage node, dynamically optimize the storage nodes and access policies, improve the load balancing ability of the storage nodes, and meet the diverse business needs. The specific solutions are as follows:

[0005] In a first aspect, the present application provides a method for load balancing of storage nodes, including:

[0006] Real-time monitoring of the states of each storage node, and collecting real-time load information;

[0007] Based on the real-time load information and the historical load data of each storage node, predicting the load trend of each storage node to obtain corresponding prediction results;

[0008] Adjusting each storage node and the corresponding data access policy according to the real-time load information and the prediction results, so as to respond to relevant data requests based on the data replicas in the adjusted storage nodes.

[0009] Optionally, the real-time monitoring of the states of each storage node and collecting real-time load information includes:

[0010] Monitor the performance usage status, storage space status, network status, data hotspots, and health status of each storage node in real time, and collect the real-time load information corresponding to each of the storage nodes.

[0011] Optionally, predicting the load trends of each of the storage nodes based on the real-time load information and the historical load data of each of the storage nodes to obtain corresponding prediction results, including:

[0012] Predict the load trends of the storage nodes according to a preset multi-model using the real-time load information and the historical load data of the storage nodes to obtain a corresponding number of initial prediction results;

[0013] Perform cross-validation on each of the initial prediction results to finally obtain the target prediction results corresponding to the storage nodes.

[0014] Optionally, adjusting each of the storage nodes and the corresponding data access policies according to the real-time load information and the prediction results, including:

[0015] Adjust the number of each of the storage nodes and the data replicas on each of the storage nodes according to the real-time load information and the prediction results;

[0016] Adjust the data access policies for each of the storage nodes according to the real-time load information and the prediction results; the data access policy characterizes the priority of the data access request and the relevant access path;

[0017] Among them, adjusting the number of each of the storage nodes and the data replicas on each of the storage nodes according to the real-time load information and the prediction results includes:

[0018] If the real-time load information and the prediction results indicate that the load trends of each of the storage nodes meet the preset high-load conditions, then construct several new nodes;

[0019] Based on the load trends of each of the storage nodes, migrate the relevant data replicas to the new nodes;

[0020] If the load trend of a single storage node meets the preset high-load conditions, then migrate the data replicas on the single storage node to other nodes.

[0021] Optionally, after collecting the real-time load information, it further includes:

[0022] Identify whether there are faults in each of the storage nodes according to the real-time load information;

[0023] Isolate the faulty node and enable the storage nodes corresponding to the faulty node that contain the same data replicas to respond to relevant data requests.

[0024] Optionally, before responding to the relevant data requests, it further includes:

[0025] Based on the access frequency and data life cycle, write several data replicas that meet the preset popularity conditions in each adjusted storage node into the cache, so as to use the data replicas in the cache to respond to relevant data requests.

[0026] Optionally, the method further includes:

[0027] Split the file with a data volume greater than the preset data volume threshold, and parallelly transmit each split data block through several transmission links to store the file in the corresponding storage node or use the file to respond to relevant data requests.

[0028] In a second aspect, the present application provides a storage node load balancing device, including:

[0029] A monitoring module for real-time monitoring of the status of each storage node and collecting real-time load information;

[0030] A prediction module for predicting the load trend of each storage node based on the real-time load information and the historical load data of each storage node to obtain corresponding prediction results;

[0031] An adjustment module for adjusting each storage node and the corresponding data access policy according to the real-time load information and the prediction results, so as to respond to relevant data requests based on the data replicas in the adjusted storage nodes.

[0032] In a third aspect, the present application provides an electronic device, including:

[0033] A memory for storing a computer program;

[0034] A processor for executing the computer program to implement the storage node load balancing method as described above.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, and when the computer program is executed by a processor, it implements the storage node load balancing method as described above.

[0036] It can be seen that the present application can monitor the status of each storage node in real time and collect real-time load information; then, based on the real-time load information and the historical load data of each storage node, predict the load trend of each storage node to obtain corresponding prediction results; thereafter, adjust each storage node and the corresponding data access policy according to the real-time load information and the prediction results, so as to respond to relevant data requests based on the data replicas in the adjusted storage nodes. In this way, the present application can combine the real-time load information of each storage node and the prediction results of the corresponding load trends to adjust the storage nodes and the data access policy; it can accurately grasp the load change trend of each storage node, dynamically optimize the storage nodes and the access policy, improve the load balancing ability of the storage nodes, and meet diverse business requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention, and for those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0038] Figure 1 It is a flowchart of a method for load balancing of storage nodes disclosed in the present application;

[0039] Figure 2 It is a flowchart of a specific method for load balancing of storage nodes disclosed in the present application;

[0040] Figure 3 It is a schematic structural diagram of a device for load balancing of storage nodes disclosed in the present application;

[0041] Figure 4 It is a structural diagram of an electronic device disclosed in the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0043] See Figure 1 As shown, an embodiment of the present invention discloses a method for load balancing of storage nodes, including:

[0044] Step S11: Monitor the status of each storage node in real time and collect the real-time load information.

[0045] In this application, in order to grasp the operating status and resource usage of each storage node in the distributed storage system, it is necessary to monitor the status of each storage node in real time; in a specific embodiment, a lightweight load monitoring module can be deployed inside each storage node for status monitoring; in this way, the real-time load information corresponding to each storage node can be collected.

[0046] In a specific embodiment, the step of monitoring the status of each storage node in real time and collecting the real-time load information may include: monitoring the performance usage status, storage space status, network status, data hot zone, and health status of each storage node in real time, and collecting the real-time load information corresponding to each of the storage nodes respectively. Specifically, when monitoring a storage node, its performance usage status, storage space status, network status, data hot zone, and health status, etc. can be monitored, such as the processor usage rate, memory occupancy rate, storage space utilization, read / write throughput, network bandwidth and connection status, node health and fault status, data access status, etc., and the real-time load information of each storage node can be collected. In a specific embodiment, the collected information can be locally recorded according to a time period and reported to the central control module; and a threshold can be set locally, and when a monitored metric information exceeds the threshold, a local alarm can be triggered for quick response.

[0047] In a specific embodiment, the real-time monitoring process may include monitoring the CPU (Central Processing Unit) utilization rate of storage nodes. This information can reflect the ability of nodes to process compute-intensive requests (such as data compression, encryption, protocol conversion, etc.), and identify whether there are bottlenecks in CPU resources; monitoring the memory usage of nodes, including metrics such as total memory, used memory, free memory, cache, and buffer occupancy, to evaluate the ability of nodes to process requests and cache hot data, and avoid performance degradation or process anomalies caused by insufficient memory; monitoring the total storage capacity, used space, remaining space of storage nodes, as well as the usage of each logical volume or data pool, to prevent write failures caused by insufficient disk space, and at the same time provide a space basis for dynamic data replica adjustment. Further, for monitoring the read / write throughput status of each storage node, detailed I / O performance metrics such as IOPS (Input / Output Operations Per Second), bandwidth, average latency, I / O queue length, etc. of each node can be collected to identify the storage pressure and processing ability of the current node, and evaluate whether there is an I / O saturation situation; in the process of monitoring the network bandwidth and connection of nodes, specifically, the uplink and downlink bandwidth occupancy, packet loss rate, latency and other metrics of the node network interface can be monitored in real time to evaluate the data transmission ability between the node and the client and other storage nodes, and identify the problem of access performance degradation caused by network bottlenecks. Still further, for the health and fault status of storage nodes, it may include disk errors, hardware failures, process anomalies, system load anomalies, etc., to promptly discover abnormal nodes and prevent service unavailability caused by request scheduling to faulty nodes. Correspondingly, for the monitoring of data access status, it may include analyzing information such as the access frequency, request type (read, write, delete, etc.), request distribution of the data stored on the node, to assist in identifying hot data and access hotspots, which is an important basis for load balancing decisions. It can be understood that through the above multi-dimensional load monitoring, it is possible to comprehensively, accurately and real-time master the resource usage status of each node in the storage system, identify possible performance bottlenecks and risks, provide detailed data support for subsequent load balancing, and thus achieve the efficient and stable operation of the overall system.

[0048] In another specific embodiment, after obtaining the real-time load information, it may further include: identifying whether each of the storage nodes has a fault according to the real-time load information; isolating the faulty node and enabling the storage node corresponding to the faulty node that contains the same data replica, so as to respond to relevant data requests. Specifically, if it is identified through the collected real-time load information that a certain storage node has a fault, the faulty node can be isolated to prevent the faulty storage node from affecting the overall data reading and writing performance; moreover, other storage nodes containing the same data replica can be enabled to ensure that the reading and writing of the data replica corresponding to the faulty storage node will not be affected.

[0049] Step S12: Based on the real-time load information and the historical load data of each storage node, predict the load trend of each storage node to obtain corresponding prediction results.

[0050] In this application, the corresponding real-time load information of each storage node is monitored through the above steps; on this basis, combined with the historical load data of each storage node, the node load level in the short term in the future can be predicted. In a specific embodiment, time series modeling can be performed on the collected load information to mine the temporal characteristics and correlation patterns of node load changes; algorithms such as long short-term memory networks, one-dimensional convolutional neural networks, and weighted moving averages can be used to accurately predict the node load change trend to determine the load change trend of each node and possible access hotspot data in a period of time in the future.

[0051] In a specific embodiment, the predicting the load trend of each storage node based on the real-time load information and the historical load data of each storage node to obtain corresponding prediction results may include: predicting the load trend of the storage node according to a preset multi-model using the real-time load information and the historical load data of the storage node to obtain a corresponding number of initial prediction results; performing cross-validation on each of the initial prediction results to finally obtain the target prediction result corresponding to the storage node. Specifically, in order to avoid the situation where the prediction result is ambiguous in some special cases by a single model, multiple models can be combined for joint decision-making; by integrating multiple different models (such as rule-based threshold models, machine learning models, and deep learning models), cross-validation is performed on the prediction results, and finally a more robust and reliable load trend prediction result is output.

[0052] Step S13: Adjust each storage node and the corresponding data access policy according to the real-time load information and the prediction results, so as to respond to relevant data requests based on the data replicas in the adjusted storage nodes.

[0053] In this application, the real-time load information of each storage node and the prediction results including the load change trends of each storage node can be obtained through the above steps; on this basis, the status of each storage node and the corresponding data access policies at the current moment can be adjusted so that the adjusted storage nodes and the corresponding data access policies can cope with the load changes in the future for a period of time, and the data requests are responded to by using the data replicas in the adjusted storage nodes.

[0054] In a specific embodiment, before responding to relevant data requests, it may further include: based on the access frequency and data life cycle, writing several data replicas that meet the preset popularity conditions in each adjusted storage node into the cache so as to respond to relevant data requests by using the data replicas in the cache. Specifically, in order to reduce the load of the storage system and speed up data access, the data with high-frequency access can be cached to reduce the frequency of accessing the disk and shorten the access latency; specifically, factors such as the access frequency (i.e., data popularity) of the data and the data life cycle can be considered, and the data that may be accessed can be written into the cache in advance, and then the relevant data requests can be directly responded to based on the data in the cache to improve the response efficiency. Further, in a specific embodiment, the data that may be accessed can be pre-loaded into the cache in combination with the prediction results of each storage node to avoid the performance problem of cache misses caused by subsequent sudden access requests.

[0055] In another specific embodiment, it may further include: splitting a file with a data volume greater than a preset data volume threshold, and parallelly transmitting each split data block through several transmission links to store the file in the corresponding storage node or respond to relevant data requests by using the file. Specifically, for the reading and writing processes of large data volumes, a flow control mechanism can be adopted to split a large file into multiple data blocks, and multiple transmission links can be used for parallel transmission to improve the bandwidth utilization rate and optimize the data transmission efficiency. Further, in a specific embodiment, the appropriate transmission protocol can be switched in combination with conditions such as network bandwidth and latency, and the data being transmitted can be compressed and pre-processed to improve the transmission efficiency.

[0056] It can be seen that this application can adjust the storage nodes and data access policies in combination with the real-time load information of each storage node and the prediction results of the corresponding load trends; accurately grasp the load change trends of each storage node, dynamically optimize the storage nodes and access policies, can improve the load balancing ability of the storage nodes, and meet diverse service requirements.

[0057] As Figure 2 shown, an embodiment of this application discloses a storage node load balancing method, including:

[0058] Step S21: Monitor the status of each storage node in real time and collect real-time load information.

[0059] Step S22: Based on the real-time load information and the historical load data of each storage node, predict the load trend of each storage node to obtain corresponding prediction results.

[0060] Step S23: Adjust the number of each storage node and the data replicas on each storage node according to the real-time load information and the prediction results, so as to respond to relevant data requests based on the data replicas in the adjusted storage nodes.

[0061] In this embodiment, in order to cope with the sudden growth and long-term expansion requirements of the load, the elastic resource expansion mechanism can be used to ensure that the storage system can quickly respond to changes in business requirements and dynamically expand computing and storage resources; for the situation where the load is long-term low, the storage resources can be appropriately reduced to avoid resource waste. It can be understood that in the process of adjusting the storage resources according to the real-time load information and the prediction results, it may include adjusting the number of storage nodes and the saved data replicas, so that the number of adjusted storage nodes and the corresponding stored data replicas can cope with the load change trend in the future for a period of time.

[0062] In a specific embodiment, the adjustment of the number of each storage node and the data replicas on each storage node according to the real-time load information and the prediction results may include: if the real-time load information and the prediction results indicate that the load trend of each storage node meets the preset high-load condition, several new nodes are constructed; based on the load trend of each storage node, the relevant data replicas are migrated to the new nodes; if the load trend of a single storage node meets the preset high-load condition, the data replicas on the single storage node are migrated to other nodes. Specifically, if the real-time load information and the prediction results indicate that the load of most storage nodes exceeds the standard or the request volume will increase sharply, in order to distribute the data on more storage nodes and optimize the access efficiency of a single storage node, new nodes can be selected for construction, and the data on the old storage nodes can be allocated to the new storage nodes; if only a few storage nodes have exceeded the load standard or the request volume has increased sharply, on the basis of the original storage node scale, the replica data in each storage node can be directly migrated to ensure that the hot data can be dispersed to relatively idle nodes to weaken the pressure on individual storage nodes in the future for a period of time and avoid the situation of single-node overload.

[0063] Step S24: Adjust the data access policies for each of the storage nodes according to the real-time load information and the prediction result, so as to respond to relevant data requests; the data access policy characterizes the priority of data access requests and the relevant access paths.

[0064] It can be understood that for different high-throughput applications, their data access patterns also vary. For example, real-time video stream processing, mass log data access, large-scale data analysis, etc. To ensure that the storage system can adapt to different application scenarios, the data access policy can be optimized according to the real-time load information and the prediction result. In a specific embodiment, for large-scale data sequential read requests (such as data backup, log analysis, etc.), the disk read policy is optimized, and prefetching and batch read / write technologies are combined to improve the read throughput; for the scenario of frequent read / write of small files, by optimizing the metadata access path and cache policy, the burden of metadata storage is reduced and the access efficiency is improved. Further, in a high-throughput application environment, the concurrent execution of multiple tasks may cause a sharp increase in the load of the storage nodes. To avoid resource competition and performance bottlenecks, scheduling can be performed based on task priorities. Specifically, different priorities can be automatically assigned to each task according to the requirements of the task for latency, throughput, and resource consumption. For example, tasks for real-time processing (such as video transcoding) are given a higher priority, while batch data processing tasks (such as data analysis) are scheduled with a lower priority; tasks are scheduled in combination with task priorities, resource occupancy, and node loads to ensure that high-priority tasks can obtain storage node resources first and meet the business requirements of high throughput and low latency.

[0065] In a specific embodiment, by dynamically adjusting the distribution location of data replicas and optimizing the scheduling path of requests, the balanced distribution of the overall system load and the improvement of access performance are achieved. For hot data with high access frequency and a large number of concurrent requests, the hot data can be distributed to more storage nodes by dynamically increasing replicas, thereby dispersing the request pressure and avoiding overloading of a single node. For example, when the access frequency of a certain data object continuously exceeds a preset threshold, the replica expansion operation can be automatically triggered, and nodes with relatively low current load, excellent network connection, and sufficient storage space are preferentially selected as the carrier nodes for new replicas; when the access heat of hot data drops below the threshold, redundant replicas can be automatically identified, and replicas located on high-load nodes are preferentially deleted to release storage space and avoid the system burden caused by replica redundancy. Further, for different request types (read, write, delete, etc.), the storage system can dynamically select the optimal access path according to multi-dimensional information such as the current load, access latency, and network bandwidth of each replica's storage node. When there are multiple copies of data replicas, read requests are preferentially scheduled to the replica node with the lowest latency and the lightest load to ensure the read request response time; on the premise of ensuring data consistency, write requests are preferentially scheduled to nodes with higher write performance and moderate load, and at the same time, a consistency protocol (the distributed system consensus algorithms Paxos or Raft) is combined to ensure the global consistency of data updates; when a performance bottleneck or anomaly occurs in the request target node, the system automatically triggers request re-scheduling, and other healthy replica nodes are preferentially selected for request forwarding to ensure high availability and fault tolerance of requests.

[0066] In a specific embodiment, the selection of data replicas can be combined with node load and access cost, and the following factors can be considered comprehensively for replica selection: the current CPU, memory, and I / O load conditions of the replica node; the network bandwidth and latency between the replica node and the client; the health status and availability of the replica node; the historical access performance and data consistency latency of the replica node. Based on the above factors, a comprehensive score can be calculated for each replica, and the target node for request scheduling can be dynamically determined according to the score, so as to avoid scheduling imbalance caused by a single load factor. Further, for nodes that have been in a high-load state for a long time, some hot data replicas carried by them can be migrated to other nodes with lower load. The specific process includes: automatically identifying high-load nodes and the hot data replicas they carry; screening suitable migration target nodes according to the global load distribution of the system; completing replica migration online to ensure data consistency and uninterrupted business; automatically updating metadata management after migration to ensure correct routing of subsequent requests. It can be understood that the process of request scheduling and the process of replica adjustment can be carried out in coordination to avoid the invalidation of the request path caused by frequent replica adjustment. After the replica adjustment operation is completed, the system automatically synchronizes and updates the path information of the request scheduling module to ensure the continuity and consistency of data access. At the same time, when the replica adjustment is not completed, the system caches requests through the intermediate scheduling node or forwards them to the sub-optimal replica to avoid affecting the normal response of business requests during the adjustment process.

[0067] It can be seen that this application can adjust the storage nodes and data access policies by combining the real-time load information of each storage node and the prediction results of the corresponding load trends; it can accurately grasp the load change trends of each storage node, optimize the hot data access path through intelligent data replica and request scheduling strategies, reduce the load difference between nodes, improve data access efficiency, improve the load balancing level of the distributed storage system, enhance the high-throughput processing ability and stability of the system, reduce latency, and improve the user experience to meet diverse business needs.

[0068] The following embodiments will introduce a storage node load balancing method disclosed in this application in combination with specific scenarios, specifically including:

[0069] In a specific embodiment, to achieve comprehensive and collaborative optimization of various resource bottlenecks (such as CPU, memory, storage, network, etc.) in complex high-throughput application scenarios, a multi-dimensional joint scheduling decision mechanism can be adopted. By comprehensively considering multiple resource load indicators of storage nodes, combining data replica distribution, access patterns, and predicted load trends, it can dynamically and intelligently decide the request scheduling path and replica adjustment operations, thereby achieving load balancing and optimal performance within the entire system. Specifically, this mechanism is achieved through the following aspects: For the problem that a single load factor is insufficient to comprehensively reflect the node pressure, a joint load evaluation model can be used to conduct a weighted comprehensive evaluation of multiple indicators such as CPU usage rate, memory occupancy rate, storage space usage rate, I / O throughput, I / O latency, and network bandwidth occupancy, forming an overall load score for the node. The weights of the load factors are dynamically adjusted according to the system operation strategy and business scenarios (for example, increasing the weight of I / O indicators in I / O-intensive scenarios and increasing the network load weight when the network bottleneck is obvious); the evaluation model also supports real-time dynamic updates, automatically refreshing the comprehensive score according to the changes in node load to ensure timely and accurate scheduling decisions. According to the resource sensitivity of different types of requests (read, write, metadata access, etc.), a scheduling priority system is established to achieve hierarchical scheduling based on the request type. For example, high-priority requests (such as real-time business read and write requests) are preferentially scheduled to nodes with low load and fast response to ensure business timeliness; low-priority requests (such as background data migration, compression, and sorting tasks) are scheduled to execute after the system load is relatively balanced to avoid affecting the foreground business. At the same time, for emergency scenarios (such as sudden spikes in node load or node failures), the relevant scheduling priorities can be dynamically increased to ensure a quick response from the system.

[0070] Correspondingly, based on the joint scheduling algorithm for multi-objective optimization, the following factors can be comprehensively considered to dynamically determine the request scheduling path and replica layout. Specifically, it involves: the comprehensive load score of nodes: preferentially select nodes with lower load to share requests; the location of replica distribution: combined with data consistency requirements, avoid replicas being concentrated in the same area or high-load nodes; the expected request delay: based on historical access delay data, preferentially select nodes with fast response speed; the network topology and bandwidth status: avoid high-latency paths across regions and data centers, and prefer local or same-rack nodes; future load prediction: based on the prediction results of the dynamic load assessment model, avoid scheduling to nodes that are about to generate load peaks; data access pattern: identify hot data and preferentially schedule it to nodes with hot replicas distributed, reducing additional transmission consumption. It can be understood that the scheduling decision is not only based on the current state but also continuously receives real-time load feedback from each node to achieve dynamic adjustment. During the scheduling execution process, continuously monitor the status of target nodes. When it is found that the load surges or the response slows down, requests can be quickly transferred to other candidate nodes; for long-term high-load nodes, trigger dynamic migration of data replicas and reallocation of requests to form a linkage between scheduling and replica adjustment; the feedback mechanism also supports anomaly identification. For example, when partial resources of a node fail (such as normal storage but abnormal network), only relevant request types are blocked to enhance the system's resilience. It should be noted that the scheduling mechanism in this embodiment supports the coordinated operation of global load balancing and local adaptive scheduling; specifically, global scheduling can be performed by the central control module based on global load assessment and data distribution to make macro scheduling decisions across nodes and regions, ensuring overall system load balance; in local scheduling, each storage node has the ability of local scheduling autonomy. According to the local latest load status, local optimization scheduling is performed on some requests (such as internal data transfer) to improve the response speed and reduce the central load. In a specific embodiment, the storage system can automatically record the effects of each scheduling decision, including request response time, node load changes, failure rates, etc., and continuously optimize the scheduling model through machine learning methods; specifically, according to the scheduling historical record information, dynamically adjust the weights of multi-dimensional load factors to improve scheduling accuracy; mine successful scheduling patterns and failure reasons to provide decision-making basis for subsequent scheduling and enhance the system's adaptive ability. It can be understood that by predicting the load trends of each storage node, potential high-load nodes can be identified in advance, and corresponding load balancing operations can be triggered in advance before the nodes enter overload; hot access areas can be dynamically identified, and the replica layout and access path of hot data can be adjusted in a timely manner; peak business fluctuations such as batch data analysis and scheduled tasks can be predicted, and potential resource bottlenecks of nodes can be estimated; combined with the fault warning mechanism, early warning and migration preparation can be carried out for nodes that may fail due to excessive load.

[0071] It can be seen that through the above multi-dimensional joint scheduling decision-making mechanism, in this embodiment, by comprehensively evaluating various resource loads and combining intelligent scheduling strategies based on data distribution and service characteristics, the system load balancing effect can be significantly improved, ensuring the continuous and stable operation and access efficiency of the distributed storage system in high-throughput and high-concurrency scenarios. At the same time, through means such as dynamic feedback and self-learning optimization, the system has good self-adaptive and self-recovery capabilities, significantly enhancing the system's intelligence level and flexible response capabilities.

[0072] In another specific embodiment, in order to meet the stringent performance requirements of the storage system in high-throughput application scenarios, a high-throughput application adaptation mechanism can be adopted. This mechanism is based on the application characteristics of high concurrency and large data volumes, and combines intelligent storage node load balancing and dynamic scheduling strategies to optimize the resource allocation, request scheduling, and data access patterns of the storage system in a high-throughput environment, thereby ensuring that the storage system can efficiently and stably process access requests for large-scale and high-speed data streams. Specifically, it includes the following aspects: identification and adaptation of data access patterns. For different high-throughput applications, the data access patterns are significantly different. For example, real-time video stream processing, massive log data access, large-scale data analysis, etc. To ensure that the storage system can adapt to different application scenarios, an intelligent data access pattern recognition algorithm can be introduced. By real-time monitoring of data access behaviors, the access paths and strategies of the storage system can be dynamically adjusted. For example, sequential read optimization: for large-scale data sequential read requests (such as data backup, log analysis, etc.), optimize the disk read strategy, and combine prefetching and batch read / write technologies to improve the read throughput; random write optimization: for applications with frequent writes (such as real-time database updates, online transaction processing, etc.), reduce the disk I / O operation latency by adjusting the write cache strategy and log write strategy; small file high-frequency access optimization: for scenarios with frequent read and write of small files (such as data of a large number of IoT (Internet of Things) devices, metadata access in a virtualized environment, etc.), reduce the burden of metadata storage and improve the access efficiency by optimizing the metadata access path and cache strategy. It can be understood that this intelligent recognition and adaptation mechanism can adjust the behavior of the storage system in real time according to the access patterns of specific applications to provide optimal performance.

[0073] Furthermore, in a high-throughput application environment, the concurrent execution of multiple tasks may cause a sharp increase in the load of storage nodes. To avoid resource competition and performance bottlenecks, scheduling can be based on task priorities. Specifically, different priorities can be automatically assigned to each task according to the requirements of the task for latency and throughput, as well as resource consumption. For example, tasks for real-time processing (such as video transcoding) are given higher priorities, while batch data processing tasks (such as data analysis) are scheduled with lower priorities. Furthermore, a dynamic priority scheduling algorithm can be designed by combining task priorities, resource occupancy, and node load to ensure that high-priority tasks can obtain storage node resources first and meet the business requirements of high throughput and low latency. This scheduling algorithm takes into account the resource idle conditions and task types of each node, dynamically allocates processing capabilities, and avoids the phenomenon of task starvation. Correspondingly, to cope with the dynamic demand for storage resources in high-throughput applications, the storage resources can be dynamically expanded. Specifically, when the system detects that the load of a storage node exceeds the standard or the request volume surges, the data can be automatically distributed across more storage nodes to provide more storage space and computing capabilities through horizontal expansion. Moreover, through virtualization technology, the storage resources can be pooled and managed to ensure that the resources can be scheduled on demand and elastically expanded. During peak load periods, new storage nodes can be quickly added through an automated mechanism to reduce the overall system load and improve the response ability. When a storage node faces an overloaded situation, the system automatically triggers data replica migration and load adjustment to ensure that hot data can be dispersed to idle nodes and avoid overloading a single node. In a specific embodiment, high-throughput applications usually need to process the rapid transmission of a large amount of data. Especially during the reading and writing of a large amount of data, the transmission speed and bandwidth become bottlenecks. The data transmission efficiency can be optimized in the following ways: for the writing and reading of large files or a large amount of data, a flow control mechanism can be adopted to split the large file into multiple data blocks and perform parallel transmission, reducing the burden on a single transmission link and improving the bandwidth utilization rate. Moreover, the optimal transmission protocol (such as storage network transmission protocols RDMA, iSCSI, NFS, etc.) can be dynamically selected according to conditions such as network bandwidth and latency, and the data during the transmission process can be compressed and preprocessed to improve the transmission efficiency. Furthermore, for cross-node data transmission, the data replicas of local nodes can be preferentially selected to reduce network bandwidth occupancy and improve the data access speed.

[0074] In a specific embodiment, in high-throughput applications, traditional storage hardware may not be able to meet the performance requirements. Therefore, heterogeneous hardware acceleration can be used to optimize storage performance. Specifically, for applications that require large-scale computing processing (such as video transcoding, machine learning data preprocessing, etc.), hardware such as GPUs (Graphics Processing Unit) or FPGAs (Field Programmable Gate Array) can be used to accelerate the computing and storage processes and improve data processing capabilities. Correspondingly, according to the characteristics of different storage hardware (such as storage devices like mechanical hard disks and solid-state drives), data storage and access policies can be dynamically adjusted to maximize hardware performance. Through the combination of software and hardware, the stable operation of the system in a high-throughput environment is ensured. Further, in order to reduce the load on the storage system and speed up data access, frequently accessed data can be cached to reduce the frequency of disk access and shorten access latency. The cache management policy combines factors such as access popularity and data lifecycle to optimize the cache hit rate; and based on the analysis and prediction of data access patterns, data that may be accessed is pre-loaded into the cache in advance to avoid performance problems caused by cache misses due to sudden access requests. It can be understood that through the above high-throughput application adaptation mechanism, personalized and efficient storage services can be provided for different application scenarios, ensuring that the system can still maintain high performance and low-latency response capabilities in high-throughput environments such as large-scale data processing and concurrent requests, and ensuring the smooth operation of high-throughput applications.

[0075] In a specific embodiment, to address high-throughput, large-scale, and high-concurrency storage requirements, this embodiment provides a system expansion and self-adaptive capability mechanism, aiming to enable the storage system to flexibly expand resources, dynamically adjust, and optimize as the business needs change and the system load fluctuates, thereby enhancing the scalability, elasticity, and self-adaptive capabilities of the system, and ensuring the stable operation of high-throughput applications in any environment. Specifically, it includes the following aspects:

[0076] (1) Elastic resource expansion mechanism

[0077] To address the sudden growth and long-term expansion requirements of system load, the present invention designs an elastic resource expansion mechanism to ensure that the system can quickly respond to changes in business needs and dynamically expand computing and storage resources:

[0078] Horizontal expansion: When the storage node faces resource bottlenecks, the system supports adding more storage nodes through horizontal expansion to enhance the overall storage capacity and computing power of the system. The newly added nodes can be seamlessly connected to the existing nodes and participate in data storage and processing.

[0079] Vertical scaling: For existing nodes, when there are bottlenecks in computing resources or storage resources, it supports vertical scaling by increasing resources such as memory, CPU, and disks of the nodes (such as upgrading hard disks, adding memory modules, etc.) to improve the processing capacity of a single node.

[0080] Automatic scaling up and down: According to the system load conditions, dynamically monitor and evaluate resource requirements, and through the automatic scaling up and down mechanism, increase or decrease computing nodes and storage nodes to optimize the configuration and use of resources, reduce resource waste, and at the same time avoid performance problems caused by overload.

[0081] (2) Dynamic load scheduling and load balancing

[0082] To ensure the stable operation of the system under high concurrency and sudden load conditions, the present invention introduces a dynamic load scheduling and load balancing mechanism:

[0083] Global load awareness: Through the global monitoring system, it can perceive the load conditions, resource usage conditions, and request response times of each storage node in real time, judge the system load conditions in real time, and avoid overload of a certain node.

[0084] Dynamic load migration: Based on global scheduling, when a certain node has a high load, the system automatically triggers the load migration mechanism to migrate some requests to nodes with lower load or newly added nodes, so as to achieve dynamic balance of the system load.

[0085] Adaptive load balancing algorithm: The present invention designs an adaptive load balancing algorithm, which can adjust the scheduling strategy of requests in real time according to the current system state, resource consumption, and response capabilities of each node to achieve optimal load distribution for different tasks.

[0086] (3) Intelligent data distribution and replica management

[0087] As the system expands and the load changes, data distribution and replica management become key factors for system stability. The present invention proposes an intelligent data distribution and replica management mechanism to ensure the efficient distribution of data among nodes and the consistency of replicas:

[0088] Intelligent replica scheduling: According to the data access pattern, load conditions, and node health status, the system intelligently adjusts the location and number of data replicas. Replicas of hot data can be dynamically migrated to nodes with lower load and higher performance, while low-frequency data can be stored on idle nodes or remote nodes.

[0089] Replica consistency management: During the system expansion process, ensure the consistency of data replicas and the reliability of the synchronization mechanism. When a certain node is scaled up or fails, the system automatically adjusts the replica synchronization strategy to ensure data consistency and availability.

[0090] Data Migration and Redundancy Adjustment: When expanding nodes, the system adjusts the data redundancy and replica strategy according to the requirements of replica consistency and resource usage, avoiding resource waste while ensuring high availability.

[0091] (4) Adaptive Algorithm and Intelligent Scheduling

[0092] To ensure that the system can adapt to different load scenarios and automatically adjust configurations, the present invention introduces an adaptive algorithm and an intelligent scheduling mechanism:

[0093] Self-Learning Scheduling Algorithm: By analyzing the system's historical load, resource utilization, and application requirements, the scheduling algorithm can gradually learn and optimize the resource allocation strategy to achieve automated scheduling optimization.

[0094] Intelligent Task Scheduling: According to the system's real-time load and task type, it automatically adjusts the task priority, resource requirements, and execution nodes to ensure that high-throughput tasks can be processed preferentially. The system can adaptively adjust the scheduling strategy to meet the changing business requirements.

[0095] Dynamic Scheduling Parameter Adjustment: The system dynamically adjusts the parameters of the scheduling algorithm according to the current load status, node health, and data transmission requirements to ensure optimal performance under different load conditions.

[0096] (5) Fault Tolerance and Recovery Mechanism

[0097] In a distributed storage system, factors such as hardware failures and network delays may affect the reliability and stability of the system. Therefore, the present invention designs a fault tolerance and recovery mechanism to ensure that the system can quickly recover in case of failures and minimize service interruption time:

[0098] Automatic Fault Detection and Isolation: The system can monitor the health status of each node in real time, automatically identify node failures, and promptly isolate the faulty nodes to prevent the failures from affecting the performance of the overall system.

[0099] Fault-Tolerant Replica Management: Through intelligent replica management, when a node fails, other replica data can be quickly enabled to ensure that data access is not affected.

[0100] Self-Recovery Mechanism: The system has self-recovery capabilities. When a node fails or the load is too high, it can automatically perform data migration, task scheduling, and node reconstruction to ensure the continuous and stable operation of the system.

[0101] (6) Elastic Load Sensing and Dynamic Optimization

[0102] To ensure the stability and high performance of the system under different load conditions, the present invention introduces an elastic load sensing and dynamic optimization mechanism:

[0103] Load awareness model: By monitoring multi-dimensional data such as node load, network bandwidth, and storage capacity in real time, the system can intelligently evaluate the current load level, predict the load change trend, and dynamically adjust the resource allocation and task scheduling strategy according to the prediction results.

[0104] Adaptive optimization strategy: When the system load changes, the system can automatically adjust task allocation and resource configuration to optimize the overall performance. For example, it automatically adds storage nodes under high load and reduces resource waste under low load.

[0105] It can be seen that through the above content, this embodiment can achieve intelligent and dynamic load scheduling of the storage system in a high-concurrency and high-load environment, improving the overall throughput and response speed of the system. Through the multi-dimensional load monitoring mechanism, the load status of storage nodes can be grasped in real time. Combining with the prediction-based dynamic load assessment model, the system load trend can be sensed in advance to achieve proactive optimization. Through the intelligent data replication and request scheduling strategy, the access path of hot data is optimized, the load difference between nodes is reduced, and the data access efficiency is improved. The multi-dimensional joint scheduling decision mechanism realizes the global optimal scheduling based on multiple resource metrics, avoiding the load imbalance problem caused by a single dimension. The high-throughput application adaptation mechanism ensures accurate scheduling for different business types and guarantees the priority processing of core services. The system expansion and self-adaptation mechanism enhance the elasticity and stability of the system in a dynamic business environment. This can significantly improve the load balancing level of the distributed storage system, enhance the high-throughput processing ability and stability of the system, reduce latency, and improve the user experience.

[0106] As Figure 3 shown, an embodiment of the present application discloses a storage node load balancing device, including:

[0107] A monitoring module 11, configured to monitor the status of each storage node in real time and collect real-time load information;

[0108] A prediction module 12, configured to predict the load trend of each storage node based on the real-time load information and the historical load data of each storage node, and obtain corresponding prediction results;

[0109] An adjustment module 13, configured to adjust each storage node and the corresponding data access strategy according to the real-time load information and the prediction results, so as to respond to relevant data requests based on the data replicas in the adjusted storage nodes.

[0110] It can be seen that the present application can adjust the storage nodes and data access policies by combining the real-time load information of each storage node and the prediction results of the corresponding load trends; it can accurately grasp the load change trends of each storage node, dynamically optimize the storage nodes and access policies, improve the load balancing ability of the storage nodes, and meet diverse business requirements.

[0111] In a specific embodiment, the monitoring module 11 may include:

[0112] A monitoring unit for real-time monitoring of the performance usage status, storage space status, network status, data hot zone, and health status of each storage node, and collecting the real-time load information corresponding to each of the storage nodes.

[0113] In a specific embodiment, the prediction module 12 may include:

[0114] A prediction unit for predicting the load trend of the storage node according to a preset multi-model, using the real-time load information and the historical load data of the storage node, so as to obtain a corresponding number of initial prediction results;

[0115] A verification unit for cross-verifying each of the initial prediction results to finally obtain the target prediction result corresponding to the storage node.

[0116] In a specific embodiment, the adjustment module 13 may include:

[0117] A first adjustment sub-module for adjusting the number of each of the storage nodes and the data replicas on each of the storage nodes according to the real-time load information and the prediction result;

[0118] A second adjustment unit for adjusting the data access policy for each of the storage nodes according to the real-time load information and the prediction result; the data access policy represents the priority of the data access request and the related access path;

[0119] Among them, the first adjustment sub-module may include:

[0120] A new node construction unit for constructing a number of new nodes when the real-time load information and the prediction result indicate that the load trends of each of the storage nodes meet the preset high-load condition;

[0121] A first migration unit for migrating the relevant data replicas to the new nodes based on the load trends of each of the storage nodes;

[0122] A second migration unit, configured to migrate the data replicas on the single storage node to other nodes when the load trend of the single storage node meets the preset high-load condition.

[0123] In a specific embodiment, the device may further include:

[0124] A fault identification module, configured to identify whether there is a fault in each of the storage nodes according to the real-time load information;

[0125] An isolation module, configured to isolate the faulty node and enable the storage node corresponding to the faulty node that contains the same data replicas, so as to respond to relevant data requests.

[0126] In another specific embodiment, the device may further include:

[0127] A data caching module, configured to write a number of data replicas that meet the preset popularity condition in each adjusted storage node into the cache based on the access frequency and the data life cycle, so as to respond to relevant data requests by using the data replicas in the cache.

[0128] In yet another specific embodiment, the device may further include:

[0129] A file transfer module, configured to split a file with a data volume greater than a preset data volume threshold, and parallelly transfer each split data block through a number of transmission links, so as to store the file in the corresponding storage node or respond to relevant data requests by using the file.

[0130] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 4 which is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure should not be considered as any limitation to the scope of use of the present application.

[0131] Figure 4 This is a schematic structural diagram of an electronic device 20 provided by an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the storage node load balancing method disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0132] In this embodiment, the power supply 23 is used to provide operating voltages for the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and specific limitations thereof are not provided herein; the input / output interface 25 is used to obtain external input data or output data to the outside, and the specific interface type thereof can be selected according to specific application requirements, and specific limitations are not provided herein.

[0133] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, a random access memory, a magnetic disk, an optical disc, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0134] Among them, the operating system 221 is used to manage and control the various hardware devices and the computer program 222 on the electronic device 20, and it can be Windows Server, Netware, Unix, Linux, etc. The computer program 222 can further include a computer program capable of performing other specific tasks in addition to the computer program capable of implementing the storage node load balancing method executed by the electronic device 20 disclosed in any of the foregoing embodiments.

[0135] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the storage node load balancing method disclosed above is implemented. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details are not repeated herein.

[0136] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, reference can be made to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and reference can be made to the method part for the relevant parts.

[0137] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of the examples have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0138] The steps of the methods or algorithms described in connection with the embodiments disclosed herein may be implemented directly in hardware, in software modules executed by a processor, or in a combination thereof. The software modules may be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0139] Finally, it should also be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the element.

[0140] The technical solutions provided in this application have been introduced in detail above. Specific examples are used in this document to illustrate the principles and implementation manners of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for load balancing of storage nodes, characterized in that, including: monitoring the status of each storage node in real time and collecting real-time load information; predicting the load trend of each storage node based on the real-time load information and the historical load data of each storage node to obtain corresponding prediction results; adjusting each storage node and the corresponding data access policy according to the real-time load information and the prediction results, so as to respond to relevant data requests based on the data replicas in the adjusted storage nodes.

2. The storage node load balancing method according to claim 1, characterized in that The monitoring the status of each storage node in real time and collecting real-time load information includes: monitoring the performance usage status, storage space status, network status, data hot zone and health status of each storage node in real time and collecting the real-time load information corresponding to each storage node.

3. The storage node load balancing method according to claim 1, characterized in that The predicting the load trend of each storage node based on the real-time load information and the historical load data of each storage node to obtain corresponding prediction results includes: predicting the load trend of the storage node according to a preset multi-model by using the real-time load information and the historical load data of the storage node to obtain a corresponding number of initial prediction results; performing cross-validation on each initial prediction result to finally obtain the target prediction result corresponding to the storage node.

4. The storage node load balancing method according to claim 1, characterized in that, The adjusting each storage node and the corresponding data access policy according to the real-time load information and the prediction results includes: adjusting the number of each storage node and the data replicas on each storage node according to the real-time load information and the prediction results; adjusting the data access policy for each storage node according to the real-time load information and the prediction results; the data access policy characterizes the priority of data access requests and related access paths; wherein, the adjusting the number of each storage node and the data replicas on each storage node according to the real-time load information and the prediction results includes: if the real-time load information and the prediction results indicate that the load trend of each storage node meets a preset high-load condition, constructing several new nodes; based on the load trend of each storage node, migrating relevant data replicas to the new nodes; if the load trend of a single storage node meets the preset high-load condition, migrating the data replicas on the single storage node to other nodes.

5. The storage node load balancing method according to any one of claims 1 to 4, characterized in that, After collecting the real-time load information, it further includes: identifying whether there are faults in each storage node according to the real-time load information; isolating the faulty node and enabling the storage node corresponding to the faulty node that contains the same data replicas, so as to respond to relevant data requests.

6. The storage node load balancing method according to any one of claims 1 to 4, characterized in that, Before responding to the relevant data requests, it further includes: writing several data replicas that meet the preset heat condition in each adjusted storage node into the cache based on the access frequency and data life cycle, so as to respond to relevant data requests by using the data replicas in the cache.

7. The storage node load balancing method according to any one of claims 1 to 4, characterized in that It also includes: Split a file with a data volume greater than a preset data volume threshold, and parallelly transmit each split data block through a number of transmission links to store the file in a corresponding storage node or use the file to respond to relevant data requests.

8. A storage node load balancing device, characterized in that It includes: A monitoring module, configured to monitor the status of each storage node in real time and collect real-time load information; A prediction module, configured to predict the load trend of each storage node based on the real-time load information and the historical load data of each storage node to obtain corresponding prediction results; An adjustment module, configured to adjust each storage node and the corresponding data access policy according to the real-time load information and the prediction results, so as to respond to relevant data requests based on the data replicas in the adjusted storage nodes.

9. An electronic device, characterized in that, It includes: A memory, configured to save a computer program; A processor, configured to execute the computer program to implement the storage node load balancing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For saving a computer program, the computer program, when executed by a processor, implements the storage node load balancing method according to any one of claims 1 to 7.

Images