Load balancing based storage optimization control device and system thereof
By leveraging the collaborative application of load awareness, resource management, tiered scheduling, and performance monitoring modules, the problems of performance degradation and low resource utilization in traditional storage frameworks under dynamic loads are solved, achieving efficient storage resource management and optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- WUHU DISASTER RECOVERY TECHNOLOGY CO LTD
- Filing Date
- 2026-04-21
- Publication Date
- 2026-07-21
AI Technical Summary
Traditional storage frameworks cannot grasp the load status in real time, comprehensively and accurately when facing dynamically changing loads, resulting in a lack of basis for optimization decisions. Hot nodes are prone to appear in storage clusters, some nodes are overloaded, performance is degraded, resource utilization is low, and energy consumption is high.
A load awareness module is used to monitor the load status of the storage system in real time, and load features are extracted by combining principal component analysis and wavelet transform techniques. A storage resource management module performs global and local hierarchical scheduling, and a hierarchical reinforcement learning load balancing scheduling algorithm is used to realize the migration of data between different storage levels. A comprehensive performance monitoring module performs multi-dimensional monitoring.
It achieves high resource utilization and low energy consumption in the storage system, avoids performance bottlenecks, and builds an intelligent storage platform that can self-optimize and quickly respond to changes in cluster status.
Smart Images
Figure CN122431604A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data storage management technology, specifically to a storage optimization control device and system based on load balancing. Background Technology
[0002] Load balancing-based storage optimization control is a technology that improves system performance by optimizing the use and allocation of storage resources. It mainly uses load balancing technology to manage and optimize the use of storage resources. Load balancing distributes different data requests evenly across multiple servers or other resources to ensure that no single resource is overloaded, thereby improving system response speed, optimizing resource utilization, and enhancing system reliability and availability. In storage systems, load balancing is not limited to request distribution between servers but also includes data distribution between storage devices. By monitoring the load of storage devices, it ensures that data can be evenly distributed across storage resources to avoid overloading certain storage nodes. It is very suitable for various application scenarios such as large-scale distributed storage systems, cloud computing platforms, and online data warehouses, and is especially important in environments with very large data volumes that require high availability and high performance.
[0003] Traditional storage frameworks often fail to grasp the load status in real time, comprehensively, and accurately when facing dynamically changing loads during storage optimization and control. This leads to a lack of basis for optimization decisions. Furthermore, when implementing load balancing scheduling, hot nodes are prone to appear in the storage cluster, causing some nodes to be overloaded and their performance to degrade, while other nodes remain idle. This prevents the overall performance of the storage system from scaling linearly, resulting in low resource utilization and high energy consumption. Summary of the Invention
[0004] This invention provides a storage optimization control device and system based on load balancing, which can effectively solve the problems mentioned in the background art. Current traditional storage frameworks often cannot grasp the load status in real time, comprehensively and accurately when facing dynamically changing loads, resulting in a lack of basis for optimization decisions. Furthermore, when implementing load balancing scheduling, hot nodes are prone to appear in the storage cluster, causing some nodes to be overloaded and their performance to degrade, while other nodes are idle. This prevents the overall performance of the storage system from scaling linearly, resulting in low resource utilization and high energy consumption.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a storage optimization control system based on load balancing, which improves system storage performance by optimizing the use and allocation of storage resources, including a load awareness module, a storage resource management module, a load balancing scheduling module, a data tiering management module, and a comprehensive performance monitoring module;
[0006] The load awareness module senses the storage load, the storage resource management module manages various resources in the storage system, and the load balancing scheduling module distributes the storage load evenly.
[0007] The data tiered management module enables data transfer between different storage tiers, and the comprehensive performance monitoring module performs performance monitoring by setting monitoring indicators.
[0008] According to the above technical solution, the load sensing module monitors the load status of the sensing storage system in real time, including a status acquisition unit, a protection unit, and an analysis unit;
[0009] The status acquisition unit collects load-related data from various levels of the storage system, ensuring that the unit completes data acquisition, processing, and reporting on a millisecond timescale. The analysis unit extracts load-related features and change patterns from massive amounts of data.
[0010] During the load feature extraction process, principal component analysis is used to identify the main factors affecting storage performance, and wavelet transform is used to capture the periodic and bursty characteristics of the load.
[0011] According to the above technical solution, the load awareness module is responsible for collecting and analyzing indicators including resource usage indicators, performance indicators, queue indicators, application indicators, and hardware-related indicators.
[0012] During the data acquisition process, a combination of active probing and passive collection is used. Active probing measures the system's response time and service quality by periodically sending test requests, while passive collection achieves transparent capture of input and output requests by embedding lightweight probes in the data path.
[0013] According to the above technical solution, the storage resource management module manages and optimizes various resources. In terms of resource management, it adopts a management mode that combines capacity management and performance management.
[0014] In terms of resource scheduling, a hierarchical scheduling design is adopted, which divides resource scheduling into two levels: global scheduling and local scheduling. Global scheduling is decided at the cluster level, while local scheduling is decided within a single node.
[0015] In terms of backup resource scheduling, a load-aware backup resource scheduling mechanism is adopted to dynamically adjust the resource quota of backup tasks according to the real-time load of the business.
[0016] In the process of optimizing storage resource management, optimization is carried out in two aspects: improving resource utilization and ensuring service quality. In terms of improving resource utilization, data reduction technology is used to reduce the actual storage space requirement. In terms of ensuring service quality, service quality control strategies are implemented to set upper and lower limits for the number of input and output operations per second for different services.
[0017] According to the above technical solution, the load balancing scheduling module achieves balanced distribution of storage load among cluster nodes by designing a scheduling algorithm. It adopts a hierarchical reinforcement learning scheduling architecture, which divides the scheduling decision process into a strategic layer and a tactical layer. The strategic layer uses a deep reinforcement learning network to perform end-to-end policy learning, while the tactical layer is responsible for executing short-term scheduling decisions.
[0018] At the decision-making level, the scheduling algorithm adopts a dynamic weighting algorithm with multi-objective optimization. By introducing a dynamic weighting mechanism, the weights are calculated based on the real-time load rate, input and output response time, data locality, and backup status of each node, taking into account load rate, response time, data locality, and available capacity.
[0019] According to the above technical solution, the load balancing scheduling module is designed with a multi-level dynamic adjustment mechanism to continuously adjust according to state changes, including real-time adjustment, periodic adjustment and event-driven adjustment.
[0020] According to the above technical solution, the data layer management module aims to realize the migration and flow of data between different storage layers, so that high-frequency data is stored in the high-performance storage layer and low-frequency data is stored in the large-capacity, low-cost storage layer. The data layer management strategy is to make judgments based on the data access popularity prediction. Access popularity prediction involves maintaining access statistics for each data block and using machine learning models to predict the access popularity of the data in the future.
[0021] The data tiered management module divides data into four popularity levels and maps data of different popularity levels to corresponding storage tiers. The four popularity levels are: Level 1 popularity, Level 2 popularity, Level 3 popularity, and Level 4 popularity.
[0022] Level 1 hot data is stored in a high-performance all-flash storage layer, Level 2 hot data is stored in a hybrid storage array, Level 3 hot data is stored in a high-capacity storage layer, and Level 4 hot data is stored in an archive storage layer.
[0023] According to the above technical solution, the data layer management module adopts a data migration mechanism when realizing data migration and flow, and performs three stages: trigger decision, migration execution, and completion confirmation.
[0024] In the decision-making phase, the main task is to determine whether data migration is necessary. In the migration execution phase, a background replication method combined with incremental replication technology is used to perform the migration. In the completion confirmation phase, the main task is to verify the correctness of the data migration results.
[0025] According to the above technical solution, the comprehensive performance monitoring module achieves performance monitoring by constructing comprehensive monitoring indicators, including performance indicators, capacity indicators, reliability indicators, energy consumption indicators, and data protection indicators.
[0026] In the process of implementing performance monitoring, an alarm mechanism is introduced to ensure that abnormal situations can be detected and handled in a timely manner. Alarm information is divided into four levels: warning, error, critical, and emergency, so that different levels of alarms can trigger different response processes.
[0027] A storage optimization control device based on load balancing includes a processor and a memory, wherein the memory stores programs or instructions that can run on the processor, and the programs or instructions implement storage optimization control strategies when executed by the processor.
[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0029] The load awareness module facilitates the collection of multi-level indicators from devices, nodes, volumes to the business layer, enabling comprehensive status awareness. Furthermore, by employing principal component analysis and wavelet transform techniques, it can easily extract key load features and change patterns from massive amounts of data, thereby achieving accurate load awareness. The storage resource management module combines capacity and performance management, and through global and local hierarchical scheduling, it achieves a unification of cluster-level macro-fairness and node-level micro-optimization. The load balancing scheduling module adopts a hierarchical reinforcement learning architecture that combines strategic and tactical layers, enabling the scheduling strategy to continuously self-optimize and react quickly to various cluster status changes, continuously maintaining cluster load balance and avoiding performance bottlenecks.
[0030] The data tiered management module uses machine learning to predict data popularity and automatically migrates data with different popularity levels to storage tiers with different cost-effectiveness levels to meet business performance requirements at the lowest cost. Meanwhile, the comprehensive performance monitoring module uses a multi-dimensional comprehensive indicator system to reflect the health status of the system in all aspects.
[0031] In summary, by leveraging the collaborative application of the load awareness module, storage resource management module, load balancing scheduling module, data tiering management module, and comprehensive performance monitoring module, the shortcomings of current storage systems, such as static resource allocation, difficulty in guaranteeing storage performance, and complex and inefficient operation and maintenance, are effectively addressed. This results in the construction of a highly resource-utilized intelligent storage platform, thereby reducing storage costs. Attached Figure Description
[0032] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.
[0033] In the attached diagram:
[0034] Figure 1 This is a block diagram of the storage optimization control system of the present invention. Detailed Implementation
[0035] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.
[0036] Example: Figure 1 As shown, the present invention provides a technical solution, a storage optimization control system based on load balancing, which improves system storage performance by optimizing the use and allocation of storage resources, including a load awareness module, a storage resource management module, a load balancing scheduling module, a data tiering management module, and a comprehensive performance monitoring module;
[0037] The load awareness module senses the storage load, the storage resource management module manages various resources in the storage system, and the load balancing scheduling module distributes the storage load evenly.
[0038] The data tiered management module enables data transfer between different storage tiers, while the comprehensive performance monitoring module monitors performance by setting monitoring metrics.
[0039] Based on the above technical solution, the load perception module monitors the load status of the perception storage system in real time, including a status acquisition unit, a protection unit, and an analysis unit.
[0040] The status acquisition unit collects load-related data from various levels of the storage system, including device level, node level, volume level, and business level, ensuring that the unit completes data acquisition, processing, and reporting on a millisecond time scale, and the analysis unit extracts load-related features and change patterns from massive amounts of data.
[0041] During the load feature extraction process, principal component analysis is used to identify the main factors affecting storage performance, and wavelet transform is used to capture the periodic and bursty characteristics of the load.
[0042] Based on the above technical solution, the load awareness module is responsible for collecting and analyzing indicators including resource usage indicators, performance indicators, queue indicators, application indicators, and hardware-related indicators.
[0043] Resource usage metrics include CPU utilization, memory utilization, disk I / O utilization, and network bandwidth utilization of storage nodes; performance metrics include I / O operations per second, throughput, read / write ratio, and response time of storage volumes; queue metrics include queue length, waiting time, and service time for various types of I / O requests; application metrics include request rate, concurrent connections, and operation type distribution at the business level; and hardware-related metrics include the temperature, power consumption, and health status of the collected storage devices.
[0044] During the data acquisition process, a combination of active probing and passive collection is used. Active probing measures the system's response time and service quality by periodically sending test requests, while passive collection achieves transparent capture of input and output requests by embedding lightweight probes in the data path.
[0045] Based on the above technical solution, the storage resource management module manages and optimizes various resources. In terms of resource management, it adopts a management mode that combines capacity management and performance management. Among them, capacity management includes data reduction, space reclamation and capacity prediction, while performance management includes input / output priority scheduling, bandwidth allocation and latency control.
[0046] In terms of resource scheduling, a hierarchical scheduling design is adopted, which divides resource scheduling into two levels: global scheduling and local scheduling. Global scheduling adopts a scheduling algorithm based on capacity fairness allocation, while local scheduling adopts a multi-level feedback queue algorithm. Global scheduling makes decisions at the cluster level and is responsible for resource allocation and load balancing across nodes, while local scheduling makes decisions within a single node and is responsible for the subdivision and allocation of resources within the node. Through standard interface collaboration, the upper layer provides scheduling context to the lower layer, and the lower layer feeds back scheduling results to the upper layer.
[0047] In terms of backup resource scheduling, a load-aware backup resource scheduling mechanism is adopted, which dynamically adjusts the resource quota of backup tasks according to the real-time business load. Specifically, during peak business load periods, the backup resource quota is reduced and backup tasks are paused, while during off-peak business load periods, the backup resource quota is increased to speed up the backup process.
[0048] In the process of optimizing storage resource management, optimization is carried out in two aspects: improving resource utilization and ensuring service quality. In terms of improving resource utilization, data reduction technology is used to reduce the actual storage space requirement. In terms of ensuring service quality, service quality control strategies are implemented to set upper and lower limits for the number of input and output operations per second for different services.
[0049] Based on the above technical solution, the load balancing scheduling module achieves balanced distribution of storage load among cluster nodes by designing scheduling algorithms. It adopts a hierarchical reinforcement learning scheduling architecture, dividing the scheduling decision process into a strategic layer and a tactical layer. The strategic layer is responsible for setting and learning long-term optimization goals and uses a deep reinforcement learning network for end-to-end policy learning. The tactical layer is responsible for executing short-term scheduling decisions.
[0050] At the decision-making level, the scheduling algorithm adopts a dynamic weighting algorithm with multi-objective optimization. By introducing a dynamic weighting mechanism, the weights are calculated based on the real-time load rate, input and output response time, data locality, and backup status of each node, taking into account load rate, response time, data locality, and available capacity.
[0051] Based on the above technical solutions, the load balancing scheduling module designs a multi-level dynamic adjustment mechanism to continuously adjust according to state changes, including real-time adjustment, periodic adjustment and event-driven adjustment.
[0052] Among them, real-time adjustment provides microsecond-level response to scheduling decisions and triggers load rebalancing when a sudden change in the load of a node is detected; periodic adjustment performs comprehensive load assessment and rebalancing at fixed time intervals; and event-driven adjustment is triggered when specific events occur, including node joining, node leaving, capacity alarms, performance anomalies, and the start and end of backup tasks.
[0053] Based on the above technical solution, the data tiered management module aims to realize the migration and flow of data between different storage tiers, so that high-frequency data is stored in the high-performance storage tier, and low-frequency data is stored in the large-capacity, low-cost storage tier. The data tiered management strategy is to make judgments based on the data access popularity prediction. Access popularity prediction involves maintaining access statistics for each data block and using machine learning models to predict the access popularity of the data in the future.
[0054] The data tiered management module divides data into four popularity levels and maps data of different popularity levels to corresponding storage tiers. The four popularity levels are: Level 1 popularity, Level 2 popularity, Level 3 popularity, and Level 4 popularity.
[0055] Level 1 hot data is stored in a high-performance all-flash storage layer, suitable for frequently accessed critical business data; Level 2 hot data is stored in a hybrid storage array, suitable for relatively frequently accessed data; Level 3 hot data is stored in a large-capacity storage layer, suitable for occasionally accessed data; and Level 4 hot data is stored in an archive storage layer, suitable for data that has not been accessed for a long time but needs to be retained.
[0056] In the tiered decision-making process, the importance and protection level of the data also need to be considered. For critical business data, even if the access frequency is not high, it will be stored in the high-performance layer first.
[0057] Based on the above technical solution, the data layer management module adopts a data migration mechanism when realizing data migration and flow, which consists of three stages: trigger decision, migration execution, and completion confirmation.
[0058] During the decision-making phase, the main task is to determine whether data migration is necessary. Data replication and metadata updates are performed according to the plan, using background replication combined with incremental replication technology. During the confirmation phase, the main task is to verify the correctness of the data migration results and handle any abnormal situations.
[0059] Based on the above technical solution, the comprehensive performance monitoring module achieves performance monitoring by constructing comprehensive monitoring indicators, including performance indicators, capacity indicators, reliability indicators, energy consumption indicators, and data protection indicators.
[0060] Among them, performance indicators include input / output operations per second, throughput, response time, and queue depth; capacity indicators reflect the usage status of storage resources, including used capacity, available capacity, and data reduction rate; reliability indicators reflect the health status of the storage system, including system availability, data persistence, and interval between failures; energy consumption indicators reflect the energy consumption of the storage system, including real-time node power consumption, energy efficiency, and data center energy efficiency factor; and data protection indicators are used to monitor data backup and disaster recovery status, including backup task execution status, backup progress percentage, backup data volume, backup window duration, recovery point gap, replica synchronization latency, disaster recovery link status, and failover readiness status.
[0061] In the process of implementing performance monitoring, an alarm mechanism is introduced to ensure that abnormal situations can be detected and handled in a timely manner. Alarm information is divided into four levels: warning, error, critical, and emergency, so that different levels of alarms can trigger different response processes.
[0062] A storage optimization control device based on load balancing includes a processor and a memory. The memory stores a program that can run on the processor, and the program implements a storage optimization control strategy when executed by the processor.
[0063] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A storage optimization control system based on load balancing, characterized in that: Improve system storage performance by optimizing storage resource usage and allocation, including load awareness module, storage resource management module, load balancing scheduling module, data tiering management module, and comprehensive performance monitoring module; The load awareness module senses the storage load, the storage resource management module manages various resources in the storage system, and the load balancing scheduling module distributes the storage load evenly. The data tiered management module enables data transfer between different storage tiers, and the comprehensive performance monitoring module performs performance monitoring by setting monitoring indicators.
2. The storage optimization control system based on load balancing according to claim 1, characterized in that: The load sensing module monitors the load status of the sensing storage system in real time, including a status acquisition unit, a protection unit, and an analysis unit; The status acquisition unit collects load-related data from various levels of the storage system, ensuring that the unit completes data acquisition, processing, and reporting on a millisecond timescale. The analysis unit extracts load-related features and change patterns from massive amounts of data. During the load feature extraction process, principal component analysis is used to identify the main factors affecting storage performance, and wavelet transform is used to capture the periodic and bursty characteristics of the load.
3. A storage optimization control system based on load balancing according to claim 2, characterized in that: The load awareness module is responsible for collecting and analyzing metrics including resource usage metrics, performance metrics, queue metrics, application metrics, and hardware-related metrics. During the data acquisition process, a combination of active probing and passive collection is used. Active probing measures the system's response time and service quality by periodically sending test requests, while passive collection achieves transparent capture of input and output requests by embedding lightweight probes in the data path.
4. A storage optimization control system based on load balancing according to claim 1, characterized in that: The storage resource management module manages and optimizes various resources, and in terms of resource management, it adopts a management mode that combines capacity management and performance management. In terms of resource scheduling, a hierarchical scheduling design is adopted, which divides resource scheduling into two levels: global scheduling and local scheduling. Global scheduling is decided at the cluster level, while local scheduling is decided within a single node. In terms of backup resource scheduling, a load-aware backup resource scheduling mechanism is adopted to dynamically adjust the resource quota of backup tasks according to the real-time load of the business. In the process of optimizing storage resource management, optimization is carried out in two aspects: improving resource utilization and ensuring service quality. In terms of improving resource utilization, data reduction technology is used to reduce the actual storage space requirement. In terms of ensuring service quality, service quality control strategies are implemented to set upper and lower limits for the number of input and output operations per second for different services.
5. A storage optimization control system based on load balancing according to claim 1, characterized in that: The load balancing scheduling module achieves balanced distribution of storage load among cluster nodes by designing scheduling algorithms. It adopts a hierarchical reinforcement learning scheduling architecture, dividing the scheduling decision process into a strategic layer and a tactical layer. The strategic layer uses a deep reinforcement learning network for end-to-end policy learning, while the tactical layer is responsible for executing short-term scheduling decisions. At the decision-making level, the scheduling algorithm adopts a dynamic weighting algorithm with multi-objective optimization. By introducing a dynamic weighting mechanism, the weights are calculated based on the real-time load rate, input and output response time, data locality, and backup status of each node, taking into account load rate, response time, data locality, and available capacity.
6. A storage optimization control system based on load balancing according to claim 5, characterized in that: The load balancing scheduling module is designed with a multi-layered dynamic adjustment mechanism to continuously adjust according to state changes, including real-time adjustment, periodic adjustment and event-driven adjustment.
7. A storage optimization control system based on load balancing according to claim 1, characterized in that: The data tiered management module aims to enable data migration and flow between different storage tiers, storing high-frequency data in high-performance storage tiers and low-frequency data in high-capacity, low-cost storage tiers. The data tiered management strategy is based on data access popularity prediction. Access popularity prediction involves maintaining access statistics for each data block and using machine learning models to predict the access popularity of the data in the future. The data tiered management module divides data into four popularity levels and maps data of different popularity levels to corresponding storage tiers. The four popularity levels are: Level 1 popularity, Level 2 popularity, Level 3 popularity, and Level 4 popularity. Level 1 hot data is stored in a high-performance all-flash storage layer, Level 2 hot data is stored in a hybrid storage array, Level 3 hot data is stored in a high-capacity storage layer, and Level 4 hot data is stored in an archive storage layer.
8. A storage optimization control system based on load balancing according to claim 7, characterized in that: When implementing data migration and flow, the data layer management module adopts a data migration mechanism, which executes three stages: trigger decision, migration execution, and completion confirmation. In the decision-making phase, the main task is to determine whether data migration is necessary. In the migration execution phase, a background replication method combined with incremental replication technology is used to perform the migration. In the completion confirmation phase, the main task is to verify the correctness of the data migration results.
9. A storage optimization control system based on load balancing according to claim 1, characterized in that: The comprehensive performance monitoring module achieves performance monitoring by constructing comprehensive monitoring indicators, including performance indicators, capacity indicators, reliability indicators, energy consumption indicators, and data protection indicators. In the process of implementing performance monitoring, an alarm mechanism is introduced to ensure that abnormal situations can be detected and handled in a timely manner. Alarm information is divided into four levels: warning, error, critical, and emergency, so that different levels of alarms can trigger different response processes.
10. A storage optimization control device based on load balancing, characterized in that: It includes a processor and a memory, the memory storing programs or instructions that can run on the processor, the programs or instructions implementing the storage optimization control strategy as described in any one of claims 1 to 9 when executed by the processor.