Method and system for dynamically recycling background resources of storage system

Abstract

The invention provides a method and a system for dynamically recovering background resources of a storage system, which belong to the technical field of cloud storage, and adopt a dynamic priority scheduling mechanism to intelligently optimize the execution sequence of cleaning tasks and ensure that key services are not influenced according to the use condition of system resources and service requirements; by setting a configurable time interval, the execution opportunity is intelligently selected, and the situation that a cleaning task is executed in the high-load time period of the system is avoided; the number of threads is controlled, system resources are reasonably distributed, and excessive consumption of the resources is avoided; and intelligent resource scheduling is introduced, and according to the conditions of CPU, memory, network load and the like of the storage nodes, tasks are dynamically allocated and cleaned, the system load is balanced, and the overall performance is improved. The resource recovery efficiency is improved, the influence of the cleaning process on the system performance is effectively reduced, stable operation of key services and optimization of user experience are ensured, and a more stable, efficient and intelligent background resource cleaning management solution is provided for a large-scale storage system.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud storage, and in particular, to a method and system for dynamically reclaiming background resources of a storage system. Background Art

[0002] In large-scale storage systems, with frequent user operations such as creating, modifying, and deleting files, directories, block device data, and snapshot data, a large number of resources marked as "deleted" but not actually cleared accumulate inside the storage system. These resources are not cleared temporarily to be able to respond to user operations in a timely manner. However, these residual resources not only occupy valuable storage space but may also become an invisible burden on system performance. Especially when performing operations such as disk defragmentation and metadata updates, it will significantly increase disk I / O overhead, thereby affecting the system response speed and overall performance. Traditional methods for cleaning background resources often adopt a full-disk scanning strategy with a fixed frequency. Although simple and easy to implement, they lack flexibility and intelligence, and are likely to exacerbate system load during peak hours, squeezing CPU, memory, network, and other resources used for user data reading and writing, resulting in a poor user experience. Summary of the Invention

[0003] To solve the above technical problems, the present invention provides a method for dynamically reclaiming background resources of a storage system. Through dynamic resource scheduling, priority adjustment, time interval configuration, thread pool management, and efficient algorithm optimization, intelligent management and optimization of storage system resources are achieved, and the overall system performance and resource utilization efficiency are improved.

[0004] The technical solution of the present invention is as follows:

[0005] A method for dynamically reclaiming background resources of a storage system, comprising the following steps:

[0006] 1) Monitor and collect the resource usage of storage nodes in real time, including but not limited to CPU load, memory usage, network load, hard disk load, and hard disk health status;

[0007] 2) According to the collected resource usage, calculate the suitability score SuitabilityScore of each node using a formula, and the formula is:

[0008] [SuitabilityScore = w1×CPU_Load+w2×Memory_Usage+w3×Network_Load+w4×Disk_Health+w5×Disk_Load], where w1,w2,w3,w4,w5w1,w2,w3,w4,w5 are the weight coefficients of each indicator, CPULoad, MemoryUsage, NetworkLoad, DiskHealth and DiskL oad are the CPU load, memory usage, network load, hard disk health status and hard disk load of the node respectively;

[0009] 3) Normalize the above indicators and unify their value ranges to the interval [0,1];

[0010] 4) Set a threshold θ. When SuitabilityScore<θ, the node is determined to be suitable for performing resource cleanup tasks.

[0011] 5) Based on the calculated SuitabilityScore, dynamically adjust the priority and execution timing of resource cleanup tasks to ensure that key businesses are not affected;

[0012] 6) Allow administrators to configure the execution time interval of cleanup tasks according to system load and business periodicity to avoid executing cleanup tasks during high-load periods;

[0013] 7) Control the thread pool size, allocate system resources reasonably, avoid excessive resource consumption, and improve cleaning efficiency;

[0014] 8) Use efficient algorithms to optimize the cleaning process, including but not limited to batch processing, data compression, and intelligent indexing technology to reduce disk I / O operations and improve cleaning efficiency.

[0015] Furthermore,

[0016] The settings of the weight coefficients w1, w2, w3, w4, w5w1, w2, w3, w4, w5 are optimized and adjusted according to specific application scenarios and business requirements to achieve the optimal state of resource allocation.

[0017] The specific implementation method of dynamically adjusting the priority of resource cleaning tasks is: when the SuitabilityScore of a node falls within a higher range, it means that the node load is high, and the priority of the cleaning task is reduced; conversely, if the SuitabilityScore is lower, it means that the node load is low, and the priority of the cleaning task is increased to ensure the flexibility and intelligence of resource allocation.

[0018] The implementation mechanism for configuring the cleaning task execution time interval further includes: when the remaining space in the cluster is greater than a predetermined threshold, arranging the cleaning task to be executed during a period when the user's business is relatively less busy, so as to avoid competing with the user's business for resources.

[0019] The mechanism for controlling the thread pool size includes: moderately increasing the number of threads when system resources are sufficient to accelerate the cleaning speed, reducing the number of threads when resources are tense to avoid excessive resource consumption, and ensuring balanced task distribution among threads through a load balancing algorithm.

[0020] The implementation method for optimizing the efficient algorithm further includes: using a Bloom filter to pre-screen the resources to be cleaned, intelligent indexing technology to accelerate resource location, and data compression technology to reduce disk space occupancy, jointly improving the storage efficiency and cleaning speed.

[0021] In addition, the present invention also provides an intelligent optimization device for cleaning background resources in a storage system, including:

[0022] A resource monitoring module for real-time monitoring of the resource usage of storage nodes;

[0023] An applicability scoring module for calculating the applicability score of each node;

[0024] A task scheduling module for dynamically adjusting the priority and execution timing of resource cleaning tasks according to the applicability score;

[0025] A time interval configuration module for allowing an administrator to configure the execution time interval of the cleaning task;

[0026] A thread pool management module for controlling the thread pool size and reasonably allocating system resources;

[0027] An efficient algorithm optimization module for optimizing the cleaning process by using technologies such as batch processing, data compression, and intelligent indexing.

[0028] In addition, the present invention also provides a computer device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the method described in any one of the above.

[0029] The present invention also provides a computer-readable storage medium, on which a computer program is stored, and the program is executed by a processor to be used for implementing the method described in any one of the above.

[0030] The beneficial effects of the present invention are

[0031] The present invention deeply optimizes each link of the background resource cleaning in the storage system, significantly improving the efficiency of resource cleaning and the overall performance of the system, specifically reflected in the following aspects:

[0032] 1. Efficiency and balance of intelligent resource scheduling: By building a resource monitoring and scheduling system, evaluating the resource status of storage nodes in real time, and combining intelligent algorithms to dynamically allocate cleaning tasks, we can effectively avoid resource contention and local overload, and significantly improve the utilization efficiency and overall performance of system resources. The intelligent node suitability scoring mechanism ensures that cleaning tasks are always executed on the most suitable node, avoiding potential threats to system stability.

[0033] 2. Flexibility and intelligence of dynamic priority scheduling: Dynamically adjust the priority of cleanup tasks to ensure that high-priority tasks are executed first, effectively guaranteeing the stable operation of key businesses. At the same time, it optimizes the task execution order in real time according to the node load conditions, improves the intelligent decision-making ability of resource cleanup, and ensures the optimal state of resource allocation.

[0034] 3. Flexibility and intelligent selection of configurable time intervals: Based on system load and business periodicity, the best execution time for background cleanup tasks is intelligently selected, avoiding the execution of cleanup tasks during high-load periods, significantly reducing the impact on system performance and improving user experience.

[0035] 4. Reasonable control of the number of threads and resource conservation: By dynamically adjusting the thread pool size and reasonably allocating system resources, excessive resource consumption is avoided, ensuring that the system speeds up the cleanup when resources are sufficient and reduces the number of threads when resources are tight, thus maintaining the stability and efficiency of the system.

[0036] 5. Efficient algorithm optimization for simplicity and speed: The use of advanced data structures and algorithms, such as Bloom filter pre-screening, batch processing, intelligent indexing and data compression technology, significantly reduces disk I / O operations, greatly improves the efficiency of resource cleanup, reduces the impact on the storage system, and improves the utilization efficiency of storage space.

[0037] In summary, the present invention realizes the intelligent, efficient and low-impact background resource cleaning of the storage system through key technologies such as intelligent resource scheduling, dynamic priority scheduling, configurable time intervals, thread quantity control and efficient algorithm optimization. It not only effectively solves the problems of low efficiency and unreasonable resource allocation in traditional cleaning methods, but also significantly improves the overall performance and stability of the storage system through refined management and intelligent decision-making, providing a more advanced, efficient and intelligent resource management solution for large-scale storage systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the interaction design between storage management nodes and storage nodes;

[0039] Figure 2 Schematic diagram of the timing of storage node performing resource cleanup. Detailed Implementation Manner

[0040] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0041] The present invention includes the following aspects:

[0042] 1. Intelligent resource scheduling: balance the load and improve the overall performance of the system

[0043] Function description: According to the CPU, memory, network load, disk load, etc. of the storage nodes, intelligently schedule the background cleaning resource tasks to the most suitable storage node for execution, balance the system load, and improve the overall performance.

[0044] Implementation mechanism: Build a resource monitoring and scheduling system, collect the resource usage of each storage node in real time (CPU load and utilization rate, remaining memory space, network load, disk load, and smart information), and combine intelligent algorithms to dynamically allocate cleaning tasks. The system preferentially selects nodes with abundant resources and low load to execute cleaning tasks to avoid node overload. The specific calculation formula is as follows:

[0045] Suitability Score = w1 × CPU_Load + w2 × Memory_Usage + w3

[0046] × Network_Load + w4 × Disk_Health + w5 × Disk_Load

[0047] Among them,

[0048] · Suitability Score represents the suitability score of the node, and the lower the value, the more suitable the node is for executing resource cleaning.

[0049] · w1, w2, w3, w4, w5 are weight coefficients used to optimize the importance of each index. These coefficients should be optimized according to the specific application scenario and priority.

[0050] · CPU_Load represents the CPU load, which can be the average load or the highest load within a specific time period. The larger the value, the higher the load.

[0051] · Memory_Usage represents the memory utilization rate, and the larger the value, the less available memory.

[0052] ·Network_Load represents the network load, which can be the network bandwidth utilization rate. The larger the value, the busier the network.

[0053] ·Disk_Health represents the hard disk health status, which can be scored based on SMART information. The smaller the value, the better the hard disk status.

[0054] ·Disk_Load represents the hard disk load, which can be the I / O operation frequency or queue depth of the hard disk. The larger the value, the busier the hard disk.

[0055] ·To unify these metrics to the same scale, we can normalize each metric so that its value range falls between [0, 1], where 0 represents the best state and 1 represents the worst state. For example, for CPU_Load, the following formula can be used for normalization:

[0056]

[0057] Similarly, perform similar normalization on other metrics, and then substitute the normalized metric values into the above algorithm formula to calculate the Suitability Score.

[0058] Finally, set a threshold θ. When the Suitability Score is lower than θ, it is considered that the node is suitable for performing resource cleaning. The selection of the threshold θ should be based on a comprehensive consideration of the impact on system stability and performance, and can be determined through experiments and experience.

[0059] 2. Dynamic Priority Scheduling: Intelligent Decision-making, Optimizing Task Execution Order

[0060] Function Description: Supports setting the priority of background resource cleaning tasks. The system dynamically optimizes the execution order of cleaning tasks according to the current resource usage and business requirements, and gives priority to processing high-priority cleaning tasks to ensure that key services are not affected.

[0061] Implementation mechanism: Introduce a priority queue management mechanism. Combine business analysis and system monitoring data. According to the calculation formula provided above, intelligently evaluate the importance and urgency of tasks. Multiply the Suitability Score by N coefficients to obtain N ranges. When the score falls within a higher range, it indicates that the load of the storage node is relatively high. At this time, the priority of the resource cleaning task is reduced, and the storage system preferentially processes user tasks with higher priorities. If the score is low, it means that the load of the storage node is low, and resource cleaning can be performed. At this time, the priority of resource cleaning can be increased. In addition, considering the dynamic changes in business fluctuations and system loads, the system also has the ability to optimize task priorities in real time to ensure that resource allocation is always in an optimal state.

[0062] 3. Configurable time interval: Flexibly adapt and intelligently select the execution timing

[0063] Function description: Allow administrators to set the execution time interval of background cleaning tasks according to system loads and business periodicity, avoiding frequent execution of background resource cleaning during high-load periods and reducing the impact on system performance.

[0064] Implementation mechanism: According to the Suitability Score calculated in 1, adjust the interval time for cleaning background resources in a timely manner. The timed task of the thread determines that the resource cleaning task in the queue is not executed within the time interval, and the resource cleaning task continues to be executed after the time interval is exceeded. In addition, when the remaining space of the cluster collected by the resource monitoring system is greater than a certain threshold, the cleaning task can be set to be executed during a period when the user business is relatively less busy to avoid competing with user business for resources.

[0065] 4. Control the number of threads: Reasonably allocate to avoid excessive resource consumption

[0066] Function description: Support setting the number of background cleaning tasks that each storage process can execute simultaneously. By controlling the size of the thread pool, reasonably allocate system resources to avoid excessive competition among multiple threads for system resources, which affects system stability.

[0067] Implementation mechanism: Design a thread pool management module to dynamically optimize the size of the thread pool according to the usage of system resources. When system resources are sufficient, moderately increase the number of threads to accelerate the cleaning speed. When resources are scarce, the thread pool reclaims redundant threads, reduces the number of threads, and avoids excessive consumption. At the same time, introduce a load balancing algorithm to ensure balanced task distribution among threads and improve the overall cleaning efficiency.

[0068] 5. Efficient algorithm optimization: Streamline operations and significantly improve cleaning efficiency

[0069] Functional description: Optimize the algorithm to efficiently process a large number of files, reduce disk I / O operations, and improve the cleaning efficiency and reduce the impact on the storage system through techniques such as batch processing, data compression, and intelligent indexing.

[0070] Implementation mechanism: Adopt advanced data structures and algorithms, such as using Bloom filters to pre-screen resources to be cleaned, effectively reducing invalid disk accesses. The batch processing mechanism combines multiple small-scale operations into one large-scale operation, significantly reducing the number of disk I / Os. In addition, the intelligent indexing technology accelerates resource location, reduces search time, and further improves the cleaning speed. The data compression technology reduces disk space occupancy without affecting the cleaning effect, improving storage efficiency.

[0071] Improve cleaning efficiency: By optimizing the algorithm to reduce disk I / O operations and adopting techniques such as batch processing, data compression, and intelligent indexing, significantly improve the speed and efficiency of resource recovery.

[0072] Reduce performance impact: Dynamically optimize the priority of cleaning tasks, reasonably control the number of threads, and intelligently select the execution time to avoid executing cleaning tasks during high system load periods, ensure the stable operation of critical services, and reduce the impact on system performance.

[0073] Intelligent resource allocation: According to the actual load conditions of storage nodes, intelligently schedule cleaning tasks to the most suitable nodes for execution, balance system resources, and improve overall performance.

[0074] Flexibility and customizability: Support users to flexibly configure the cleaning time interval and the number of threads according to system load and business requirements, ensuring that the resource cleaning strategy highly matches business needs.

[0075] By achieving the above goals, the present invention is committed to providing a more robust, efficient, and intelligent background resource management solution for modern large-scale storage systems, helping enterprises maximize the utilization of data resources and improve cost-effectiveness, while providing users with a more stable, efficient, and responsive storage service experience.

[0076] The above are only the preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention and are not used to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for dynamically reclaiming background resources of a storage system, characterized in that: The following steps are involved: 1) Real-time monitoring and collection of storage node resource usage, including CPU load, memory usage, network load, hard disk load, and hard disk health status; 2) Calculate the suitability score of each node based on the collected resource usage; 3) Normalize the above indicators and unify their value ranges to the interval [0,1]; 4) Set a threshold θ. When SuitabilityScore<θ, the node is determined to be suitable for performing resource cleanup tasks. 5) Based on the calculated SuitabilityScore, dynamically adjust the priority and execution timing of resource cleanup tasks to ensure that key businesses are not affected; 6) Allow administrators to configure the execution time interval of cleanup tasks according to system load and business periodicity to avoid executing cleanup tasks during high-load periods; 7) Control the thread pool size and allocate system resources reasonably; 8) Use efficient algorithms to optimize the cleaning process, including batch processing, data compression, and intelligent indexing technology to reduce disk I / O operations and improve cleaning efficiency.

2. The method according to claim 1, characterized in that The suitability score SuitabilityScore of each node is calculated using the formula: [SuitabilityScore=w1×CPU_Load+w2×Memory_Usage+w3×Network_Load+w4×Disk_Health+w5×Disk_Load], where w1,w2,w3,w4,w5w1,w2,w3,w4,w5 are the weight coefficients of each indicator, CPULoad, MemoryUsage, NetworkLoad, DiskHealth and DiskL oad are the CPU load, memory usage, network load, hard disk health status and hard disk load of the node respectively.

3. The method according to claim 2, characterized in that The settings of the weight coefficients w1, w2, w3, w4, w5w1, w2, w3, w4, w5 are optimized and adjusted according to specific application scenarios and business requirements to achieve the optimal state of resource allocation.

4. The method according to claim 1, characterized in that The specific implementation method of dynamically adjusting the priority of resource cleaning tasks is: when the SuitabilityScore of a node reaches above 70%, it means that the node load is high, and the priority of the cleaning task is reduced; conversely, if the SuitabilityScore is below 50%, it means that the node load is low.

5. The method according to claim 1, characterized in that The configuration of the cleaning task execution time interval also includes: when the remaining space of the cluster is greater than a predetermined threshold, arranging the cleaning task to be executed in a time period when the CPU usage rate of the user business is less than 30% to avoid competing for resources with the user business.

6. The method according to claim 1, characterized in that The control of the thread pool size includes: increasing the number of threads to speed up the cleaning process when system resources are sufficient, that is, when the CPU utilization rate is below 50%, and reducing the number of threads when the CPU resource utilization rate reaches 70%. At the same time, a load balancing algorithm is used to ensure balanced task distribution among threads.

7. The method according to claim 1, characterized in that The implementation method of the efficient algorithm optimization also includes: using Bloom filters to pre-screen resources to be cleaned, intelligent indexing technology to accelerate resource positioning, and data compression technology to reduce disk space usage, thereby jointly improving storage efficiency and cleaning speed.

8. A system for dynamically reclaiming background resources of a storage system, characterized in that: include: Resource monitoring module, used to monitor the resource usage of storage nodes in real time; The suitability scoring module is used to calculate the suitability score of each node; Task scheduling module, used to dynamically adjust the priority and execution timing of resource cleanup tasks according to the applicability score; The time interval configuration module allows the administrator to configure the execution time interval of the cleanup task; Thread pool management module, used to control thread pool size and reasonably allocate system resources; An efficient algorithm optimization module is used to optimize the cleaning process using technologies such as batch processing, data compression, and intelligent indexing.

9. The system according to claim 8, characterized in that The suitability score SuitabilityScore of each node is calculated using the formula: [SuitabilityScore=w1×CPU_Load+w2×Memory_Usage+w3×Network_Load+w4×Disk_Health+w5×Disk_Load], where w1,w2,w3,w4,w5w1,w2,w3,w4,w5 are the weight coefficients of each indicator, CPULoad, MemoryUsage, NetworkLoad, DiskHealth and DiskL oad are the CPU load, memory usage, network load, hard disk health status and hard disk load of the node respectively.

10. The system according to claim 8, characterized in that The specific implementation method of dynamically adjusting the priority of resource cleaning tasks is: when the SuitabilityScore of a node reaches above 70%, it means that the node load is high, and the priority of the cleaning task is reduced; conversely, if the SuitabilityScore is below 50%, it means that the node load is low.

Images