Cloud storage dynamic optimization method and system

Abstract

The invention relates to the technical field of cloud computing storage, in particular to a cloud storage dynamic optimization method and system, and the method comprises the following steps: based on a cloud storage data access record, recognizing an access time interval sequence, analyzing the variation amplitude of adjacent time intervals, and calculating the standard deviation of the access time intervals; and determining the access period stability according to the standard deviation and the change amplitude, and classifying the data objects to obtain a data access period stability classification value. According to the method, task scheduling is more accurate by identifying a data access time interval sequence and changes thereof and refining data classification and storage resource allocation, a high-throughput and low-delay storage path is matched, the health condition of a storage node is evaluated, a fault node is identified in advance, data stability is guaranteed, and data copy distribution is adjusted; the storage reliability and stability are improved, the storage path for accessing the intensive data is optimized, the dynamic load balance of the storage resources is realized, and the storage efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud computing storage, and in particular, to a cloud storage dynamic optimization method and system. Background Art

[0002] The technical field of cloud computing storage includes core contents such as data storage, distributed storage architecture, data management, resource scheduling, access control, etc. The basis of this technical field is to integrate computing resources and storage resources through a network to provide an efficient and scalable data storage and management solution. Cloud computing storage relies on various storage methods such as virtualization, distributed file, object storage, and block storage, enabling users to access data on demand while optimizing the utilization rate of storage space. The reliability of data storage is ensured through methods such as multi-copy storage, erasure coding technology, and data snapshots, while the efficiency of data access depends on cache optimization, data sharding, and load balancing strategies. Cloud computing storage uses a distributed consistency protocol to ensure data synchronization and combines access control strategies and encryption technology to enhance data security. In a large-scale data environment, cloud computing storage is widely used in scenarios such as cloud databases, big data processing, streaming media storage, and enterprise-level file sharing, improving storage performance and availability through automated management and dynamic resource allocation.

[0003] Among them, the cloud storage dynamic optimization method refers to a technical solution that addresses problems such as data access performance, storage resource utilization rate, and data consistency in a cloud computing storage environment by dynamically adjusting data distribution, storage strategies, and access paths to improve the storage operation efficiency. The patent theme covers key technical matters such as dynamic adjustment of data distribution, storage load balancing, access path optimization, and data replica management. The dynamic adjustment of data distribution optimizes the data storage location in real time based on the load status of storage nodes, data access frequency, and network topology information. Storage load balancing adopts a resource allocation adjustment strategy between nodes and combines a data migration mechanism to avoid overloading of storage nodes or waste of resources. Access path optimization dynamically adjusts the interaction path between the client and storage nodes by analyzing the data access pattern and combining a multi-path selection strategy to improve the response speed of data reading and writing. Data replica management is based on the reliability evaluation of storage nodes and combines replica creation, synchronization, and recycling strategies to ensure high data availability and optimize the utilization of storage resources.

[0004] In the prior art in a cloud storage environment, although high availability of data can be achieved through a distributed storage architecture, there are limitations in the accurate identification of data access patterns, making it difficult to refine the stability classification of access cycles. As a result, it is difficult for storage resources to match data access requirements, affecting the optimization of storage performance. During the task scheduling process, the access frequency, the amount of stored access data, and the parallel execution requirements are not fully considered, resulting in inefficient allocation of storage resources. It is difficult to make full use of high-throughput and low-latency storage devices, affecting the overall performance of data processing. The health assessment methods for storage nodes are relatively single, unable to accurately predict the lifespan and health trends of storage media, causing some nodes about to fail to still undertake key data storage tasks, increasing the risk of data loss. The management strategy for data replicas does not fully consider the health changes of storage nodes, easily causing replica redundancy or waste of storage resources. At the same time, in terms of storage resource scheduling, path optimization is not carried out for access-intensive data, causing some storage nodes to be in a high-load state for a long time, affecting the balance and operation efficiency of the overall storage. Summary of the Invention

[0005] The object of the present invention is to solve the deficiencies existing in the prior art and to propose a cloud storage dynamic optimization method and system.

[0006] To achieve the above object, the present invention adopts the following technical solutions: A cloud storage dynamic optimization method includes the following steps: S1: Based on the cloud storage data access records, identify the access time interval sequence, analyze the change amplitude of adjacent time intervals, calculate the standard deviation of the access time intervals, determine the access cycle stability according to the standard deviation and the change amplitude, and classify data objects to obtain the data access cycle stability classification value; S2: Based on the data access cycle stability classification value, extract the storage access frequency, the amount of stored access data, and the parallel execution requirements, analyze the resource occupancy of task scheduling, identify the storage task priorities, match high-throughput and low-access-latency storage devices, adjust the storage paths of tasks, and generate the storage task priority allocation result; S3: Based on the storage task priority allocation result, extract the hardware state data of storage nodes, evaluate the write amplification factor, the bad block growth rate, and the read / write load change of the storage media, analyze the health decline trend of storage nodes, and infer the remaining available lifespan to obtain the storage node health trend value; S4: Based on the storage node health trend value, screen the storage nodes with declining lifespan, reduce the number of data replicas, create new replicas through healthy state stable nodes, adjust the migration paths of stored data, and implement data replica distribution adjustment.

[0007] As a further solution of the present invention, the classification value of the data access cycle stability includes a high stability category, a medium stability category, and a low stability category. The storage task priority allocation result includes a task priority level, a storage path allocation, and a storage device matching. The storage node health trend value includes a health level assessment, a remaining available life prediction, and a failure risk assessment. The data replica distribution adjustment includes replica quantity optimization, replica storage location adjustment, and data migration path optimization.

[0008] As a further solution of the present invention, the specific steps for obtaining the classification value of the data access cycle stability are as follows: S111: Based on the cloud storage data access records, extract the access time interval sequence, analyze the change amplitude of adjacent time intervals, parse the time intervals of adjacent time points, detect the change of time intervals between adjacent time points, and obtain the time interval change amplitude value; S112: Invoke the time interval change amplitude value, identify the standard deviation of the access time interval, measure the deviation degree using the difference between each time interval value and the mean value, and obtain the access time interval standard deviation value; S113: Invoke the access time interval standard deviation value, calculate the access cycle stability value, classify the data objects according to the stability value, and use the formula: ; Obtain the classification value of the data access cycle stability; Among them, represents the access cycle stability value, represents the number of accesses in the access cycle, represents the th access time, represents the th access time, represents the th time interval of the access time.

[0009] As a further solution of the present invention, the specific steps for obtaining the storage task priority allocation result are as follows: S211: Based on the classification value of the data access cycle stability, extract the storage access frequency, the storage access data volume, and the parallel execution requirement, identify the resource occupancy of the storage task, compare the resource occupancy of each task, and obtain the storage task resource occupancy coefficient; S212: Invoke the storage task resource occupancy coefficient, analyze the priority factor of the storage task, perform the priority sorting between tasks, and use the formula: ; Obtain the storage task priority coefficient; Among them, represents the storage task priority coefficient, represents the storage access frequency, represents the storage access data volume, represents the storage task resource occupancy coefficient, represents the storage benchmark occupancy value, represents the task parallel execution requirement; S213: Invoke the storage task priority coefficient, match a high-throughput and low-access-latency storage device, adjust the storage path of the storage task, and generate a storage task priority allocation result.

[0010] As a further solution of the present invention, the step of obtaining the storage node health trend value is specifically as follows: S311: Based on the storage task priority allocation result, extract the storage node hardware status data, judge the abnormal growth trend of the storage node, and calculate the bad block growth rate value of the node; S312: Invoke the bad block growth rate value of the node, combine with the change of storage load, extract the difference between the current write amplification factor of the storage medium and the threshold, and correct it in combination with the change trend of the write rate to obtain the storage medium write amplification coefficient; S313: Invoke the storage medium write amplification coefficient, combine with the bad block growth rate value of the node and the change of read and write load, analyze the health decline trend of the storage node, and use the formula: ; Speculate the remaining available life of the storage medium to obtain the storage node health trend value; Wherein, represents the storage node health trend value, represents the storage medium write amplification coefficient, represents the bad block growth rate value of the node, represents the read rate of the storage node, represents the rate of the storage node, represents the current available storage unit number of the storage medium, represents the change of the storage load of the storage node.

[0011] As a further solution of the present invention, the step of obtaining the data replica distribution adjustment is specifically as follows: S411: Based on the storage node health trend value, detect the original operation data of the storage node, extract the health parameters, compare with the set health decline threshold, and screen the storage nodes with declining storage life to obtain the set of storage nodes with declining storage life; S412: Invoke the set of nodes with decreased storage lifespan, compare the data replica distribution, identify the data replica storage amount on the nodes, analyze the migration priority, filter out the set of nodes whose data replica quantity needs to be adjusted, and use the formula: ; Calculate the node data replica adjustment value, filter out the target nodes that need to reduce data replicas, and obtain the set of replica adjustment target nodes; Among them, represents the node data replica adjustment value, represents the th node's current data replica storage amount, represents the th node's optimal storage capacity, represents the storage balance threshold, represents the total storage upper limit value, represents the th storage node's available storage amount, represents the total number of target nodes, represents the total number of the storage node set; S413: Based on the set of replica adjustment target nodes, select storage nodes with stable health status, allocate the storage paths of data replicas, and perform storage migration to obtain the optimized data replica distribution state and implement data replica distribution adjustment.

[0012] As a further solution of the present invention, the method further includes step S5: S5: Based on the data replica distribution adjustment, analyze the storage load status of cloud storage resource scheduling, adjust the storage paths of access-intensive data, optimize the storage distribution of task-occupied storage resources, perform load distribution of storage resources, and execute storage resource distribution adjustment; The storage resource distribution adjustment includes storage load balancing, access-intensive data optimization, and task storage resource allocation optimization.

[0013] As a further solution of the present invention, the obtaining steps of the storage resource distribution adjustment are specifically: S511: Based on the data replica distribution adjustment, extract the storage occupancy, access request volume, and storage load status of storage nodes, calculate the storage load rate, and obtain the storage load status index; S512: Invoke the storage load status index, extract the access frequency and storage load rate of access-intensive data, adjust the storage path, and obtain the storage balance coefficient; S513: Invoke the storage balance coefficient, optimize the task storage distribution, analyze the optimal allocation of storage resources, and use the formula: ; Calculate the optimal allocation metrics of storage resources, obtain the storage resource allocation plan, and perform storage resource distribution adjustment; Among them, represents the optimal allocation metrics of storage resources, represents the storage capacity of storage node ; represents the storage utilization rate of storage node ; represents the storage requirement of task ; represents the remaining storage space of storage node ; represents the total number of storage nodes, represents the redundancy of data replica distribution, represents the total number of storage nodes.

[0014] The cloud storage dynamic optimization system is used to execute the above cloud storage dynamic optimization method. The system includes: The access cycle analysis module calculates the access time interval and standard deviation based on the cloud storage data access record, analyzes the cycle stability, classifies data objects, and obtains the data access cycle stability classification value; The task priority allocation module extracts the access frequency, data volume, and parallel requirements based on the data access cycle stability classification value, calculates the resource occupancy ratio, allocates task priorities, filters high-throughput and low-latency storage devices based on the task scheduling scenario of cloud storage dynamic optimization, matches priority tasks to corresponding devices, and adjusts the task storage path to obtain the storage task priority allocation result; The storage health assessment module analyzes the write amplification factor, bad block growth rate, and read / write load of the storage medium based on the storage task priority allocation result, estimates the remaining life, and obtains the storage node health trend value; The replica distribution adjustment module filters the nodes with decreasing life based on the storage node health trend value, adjusts the replica distribution, reduces the number of replicas, creates new replicas based on healthy nodes, and adjusts the migration path to obtain the data replica distribution adjustment result; The storage resource optimization module analyzes the storage load based on the data replica distribution adjustment result, identifies the storage paths of access-intensive data, adjusts the storage distribution, optimizes the resource occupancy, and performs storage resource distribution adjustment.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In the present invention, by identifying the data access time interval sequence and calculating the standard deviation and variation range of the access time interval, the cycle stability of data access can be accurately measured, making data classification more refined, which helps to further optimize the storage resource allocation. Based on the classification of access cycle stability, combined with the storage access frequency, the amount of accessed data, and the parallel execution requirements, the refined management of task scheduling is realized, enabling the storage task to match the optimal storage path and ensuring that the high-throughput and low-latency storage requirements are met. The health status of the storage node is evaluated through the write amplification factor, the bad block growth rate, and the changes in read and write loads, which can effectively predict the health trend and remaining life of the storage node, enabling the storage to identify potential faulty nodes in advance, ensuring the stability of data storage, adjusting the data replica distribution for nodes with a declining storage life, reducing the number of replicas of unstable nodes, and creating new data replicas through nodes with a more stable health status, making the data redundancy distribution more reasonable, enhancing the reliability and access stability of data storage. Based on the adjustment of data replicas, the distribution of storage resources is further optimized. The adjustment of the storage path for access-intensive data makes the access hotspots more evenly distributed, and the load of storage resources is dynamically allocated in the cloud environment to ensure the overall load balance of storage nodes, reduce the performance bottleneck of high-load nodes, and improve the operating efficiency of storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of the working process of the present invention; Figure 2 is a flowchart for obtaining the classification value of the data access cycle stability in the present invention; Figure 3 is a flowchart for obtaining the allocation result of the storage task priority in the present invention; Figure 4 is a flowchart for obtaining the health trend value of the storage node in the present invention; Figure 5 is a flowchart for obtaining the adjustment of the data replica distribution in the present invention; Figure 6 is a flowchart for obtaining the adjustment of the storage resource distribution in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0017] In order to make the objectives, technical solutions, and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] In the description of the present invention, it should be understood that the orientation or positional relationships indicated by the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. are based on the orientation or positional relationships shown in the drawings. These are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be construed as a limitation to the present invention. In addition, in the description of the present invention, the meaning of "a plurality of" is two or more, unless otherwise specifically defined.

[0019] Embodiment 1: Please refer to Figure 1 , the present invention provides a technical solution: a cloud storage dynamic optimization method, including the following steps: S1: Based on the cloud storage data access records, identify the access time interval sequence, analyze the change amplitude of adjacent time intervals, calculate the standard deviation of the access time intervals, determine the access cycle stability according to the standard deviation and the change amplitude, and classify the data objects to obtain the data access cycle stability classification value; "Classifying data objects" refers to different data groups or data categories divided according to the access cycle stability. The "data access cycle stability classification value" is used to measure the stability and regularity of data access behavior in cloud storage, especially the change of data access time intervals; S2: Based on the data access cycle stability classification value, extract the storage access frequency, the amount of storage access data, and the parallel execution requirements, analyze the resource occupancy of task scheduling, identify the storage task priorities, match high-throughput and low-access-latency storage devices, adjust the storage paths of the tasks, and generate the storage task priority allocation result; S3: Based on the storage task priority allocation result, extract the hardware status data of the storage nodes, evaluate the write amplification factor, the bad block growth rate, and the read / write load change of the storage medium, analyze the trend of the health decline of the storage nodes, and estimate the remaining available life to obtain the storage node health trend value; "Bad block growth rate" and "read / write load change", these are indicators of the health status of the storage nodes, indicating the possible performance degradation problems faced by the storage medium; S4: Based on the storage node health trend value, screen the storage nodes with declining storage life, reduce the number of data copies, create new copies through the nodes with stable health status, adjust the migration paths of the stored data, and implement the adjustment of the data copy distribution; S5: Based on the adjustment of the data copy distribution, analyze the storage load status of the cloud storage resource scheduling, adjust the storage paths of the access-intensive data, optimize the storage distribution of the tasks occupying the storage resources, perform the load distribution of the storage resources, and execute the adjustment of the storage resource distribution.

[0020] The classification values of the stability of the data access cycle include high - stability category, medium - stability category, and low - stability category. The allocation results of the storage task priorities include task priority levels, storage path allocation, and storage device matching. The health trend values of the storage nodes include health level assessment, remaining available life prediction, and failure risk assessment. The adjustment of data replica distribution includes replica quantity optimization, replica storage location adjustment, and data migration path optimization. The adjustment of storage resource distribution includes storage load balancing, access - intensive data optimization, and task - storage resource allocation optimization.

[0021] Please refer to Figure 2 , and the specific steps for obtaining the classification values of the stability of the data access cycle are as follows: S111: Based on the cloud - storage data access records, extract the access time - interval sequence, analyze the change amplitude of adjacent time intervals, parse the time intervals of adjacent time points, detect the change of the time intervals of adjacent time points, and obtain the change - amplitude value of the time intervals; Extract the access time - interval sequence through the data access records. The sequence can reflect the time differences of users' access to the cloud - storage service. Calculate and analyze the time differences. First, calculate the difference between adjacent time intervals and represent the difference in absolute - value form to effectively measure the stability or volatility of the time intervals. For example, in a typical data - center operation, an administrator can observe significant fluctuations in time intervals during peak working hours, which helps further analyze load balancing and performance bottlenecks. Integrate the differences into a change - amplitude sequence of time intervals to obtain a more comprehensive view. This method is particularly suitable for scenarios that require real - time monitoring and optimization of data - access patterns, such as real - time data analysis and online transaction processing. Finally, obtain the change - amplitude value of the time intervals. The result can help the administrator evaluate and optimize the data - access strategy of the cloud storage to ensure the efficient operation of the service.

[0022] S112: Invoke the change - amplitude value of the time intervals, identify the standard deviation of the access time intervals, measure the deviation degree by using the difference between each time - interval value and the mean value, and obtain the standard - deviation value of the access time intervals; Calculate the average value of each time interval, then calculate the deviation of each time interval from the average value, and square the deviation to eliminate the influence of negative deviations. The sum of the squared values is then normalized, that is, divided by the total number of time intervals. Finally, extract the square root of the sum of squares to obtain the standard deviation. The mathematical processing process is very useful in quality control and performance monitoring. For example, in cloud - service monitoring, this value can be used to determine the consistency and predictability of data access, thereby helping the administrator adjust resource allocation and caching strategies. Finally, obtain the standard - deviation value of the access time intervals. This result is crucial for ensuring the balance and effectiveness of data access in the cloud - storage service.

[0023] S113: Call the standard deviation of the access time interval, calculate the access cycle stability value, and classify the data objects according to the stability value. The formula is: ; Obtain the data access cycle stability classification value; Among them, represents the access cycle stability value, represents the number of accesses in the access cycle, represents the th access time, represents the th access time, represents the th time interval of the access time; An improved formula is used to calculate the stability of the access cycle. Among them, represents the access cycle stability value, is the number of events in the time interval sequence (here are 5 time points), is the th access time, and the specific time points are 20, 30, 35, 40, 45, 50 minutes, is the corresponding th access time interval, which are set to 5, 6, 4, 7, 6 minutes respectively here; And substitute specific values for calculation: ; Through calculation, the values of each item are respectively: , , , , ; After summing up: ; Divide this value by N (5) and take the square root to get: ; This value 1.2215 represents the access cycle stability. A lower value indicates higher stability. This quantization method can be applied to the cloud computing environment to evaluate and optimize the performance of the data center. Especially in high-load situations, it can help administrators understand and improve the response time and processing ability of the service. Classify the data objects according to the access cycle stability value obtained by this calculation. According to the classification results, the resource allocation can be automatically adjusted to improve the service quality. Finally, the data access cycle stability classification value will directly affect the operation efficiency and customer satisfaction of the data center.

[0024] Please refer to Figure 3 , and the specific steps for obtaining the storage task priority allocation result are as follows: S211: Based on the classification value of the data access cycle stability, extract the storage access frequency, the storage access data volume, and the parallel execution requirement, identify the resource occupancy of the storage task, compare the resource occupancy of each task, and obtain the resource occupancy coefficient of the storage task; First, collect the real-time data of the storage access frequency and the storage access data volume. For example, the weekly access volume and data throughput of an enterprise database. This data can be obtained through database monitoring tools. Next, analyze the parallel execution requirement through the integrated resource management system. The requirement is determined based on the number of currently running tasks and the task type. The evaluation of the parallelism requirement can be obtained through the log analysis of the task manager. For example, the concurrent task request volume when processing data report generation. Continue to compare the resource occupancy of each task through the resource monitoring tool, such as the CPU occupancy rate and the memory usage. Through the metrics, the resource occupancy coefficient of each task can be calculated. This coefficient is a numerical representation of the comparison of resource usage efficiency, helping the administrator identify resource-intensive tasks and optimize them. For example, if a certain data analysis task occupies a large amount of CPU but the processing speed does not increase but decreases, the administrator will optimize the query algorithm or increase the processor resources. Finally, obtain the resource occupancy coefficient of the storage task.

[0025] S212: Invoke the resource occupancy coefficient of the storage task, analyze the priority factor of the storage task, perform the priority sorting between tasks, and use the formula: ; Obtain the storage task priority coefficient; Among them, represents the storage task priority coefficient, represents the storage access frequency, represents the storage access data volume, represents the resource occupancy coefficient of the storage task, represents the storage benchmark occupancy value, represents the task parallel execution requirement; Invoke the resource occupancy coefficient of the storage task to calculate the task priority factor. Assume that in a video processing, tasks with high-frequency access such as video upload (assuming the storage access frequency is 120 times / hour), and tasks with large data volume such as video encoding (assuming the storage access data volume is 500GB / task). The resource occupancy coefficient of the task will be used to calculate its priority; Perform the priority calculation. Assume that the resource occupancy coefficient is 0.8, the set storage benchmark occupancy value is 0.5, and the task parallel execution requirement is 3. After inserting the values, the formula becomes: ; First, solve the part inside the square root. The difference , whose absolute value is 0.3, and after taking the square root is , and the denominator is ; Furthermore, calculate the result of the entire formula: ; This calculation result indicates that the priority of the video upload task has been significantly improved compared to the task, enabling this task to obtain more storage and processing resources in an environment with fierce resource competition, thereby optimizing the overall response time and processing capacity, and then obtaining the priority sorting result. This result helps the administrator to reasonably schedule resources in a multi-task environment.

[0026] S213: Invoke the storage task priority coefficient, match a high-throughput and low-access-latency storage device, adjust the storage path of the storage task, and generate a storage task priority allocation result; Select the most suitable storage device according to the performance characteristics of the storage device such as throughput and access latency. For example, in a big data processing scenario, for a high-priority big data analysis task, select an SSD storage with high throughput, while for a data backup task with a lower access frequency, an HDD storage with high capacity but higher access latency can be used. In this way, not only the storage performance is optimized but also the overall efficiency is improved. The administrator will adjust the storage path according to the task priority to ensure the optimization of the data flow. For example, store frequently accessed data on a faster storage device and migrate infrequently accessed data to a lower-cost storage. Finally, generate a storage task priority allocation result, which provides an automated resource optimization solution for the system and realizes the maximization of resource utilization.

[0027] Please refer to Figure 4 , and the specific steps for obtaining the storage node health trend value are as follows: S311: Based on the storage task priority allocation result, extract the storage node hardware status data, judge the abnormal growth trend of the storage node, and calculate the bad block growth rate value of the node; Monitor the performance metrics of each storage node, such as the read and write speeds and available space of the hard disk, including invoking the system's monitoring tools to record the real-time data of each node. For example, in a data center, monitoring a node shows that the write speed is 500MB / s and the available space is 2TB. The data will be recorded and used for subsequent analysis. By applying a preset algorithm to the monitored data, determine whether the node reaches the warning bad block growth rate. The judgment is based on the historical data of the node and industry standards. If the bad block growth rate of a storage node exceeds the industry safety standard of 0.1% per month, mark the node as a risk node. The refined execution process includes obtaining the historical bad block records of each node and calculating the monthly bad block growth rate. If the calculation result shows that the bad block growth rate is 0.15%, further analyze the specific hardware situation of the node and make a judgment against the industry standard to finally obtain the bad block growth rate value of the node.

[0028] S312: Invoke the bad block growth rate value of the node, combine it with the change in storage load, extract the difference between the current write amplification factor of the storage medium and the threshold, and correct it in combination with the change trend of the write speed to obtain the write amplification coefficient of the storage medium; Combine the write amplification factor of the storage node and the change in load to further calculate the write amplification coefficient of the storage medium, including comparing the current write amplification factor with the preset threshold. For example, in a specific application scenario, a certain cloud service provider sets the optimal threshold of the write amplification factor for its database storage to 1.5. If the current write amplification factor of a monitored node is 2.0, then it is necessary to calculate the difference from the threshold and make adjustments. This adjustment is achieved by modifying the write policy or performing hardware upgrades. The specific calculation process is to first obtain the current write amplification amount and the total number of writes, and then calculate the write amplification factor based on these two parameters. If the amplification factor exceeds the preset threshold, issue a warning and put forward improvement suggestions to finally obtain the write amplification coefficient of the storage medium.

[0029] S313: Invoke the write amplification coefficient of the storage medium, combine it with the bad block growth rate value of the node and the change in read and write load, analyze the downward trend of the storage node's health, and use the formula: ; Estimate the remaining available life of the storage medium to obtain the storage node health trend value; Among them, represents the storage node health trend value, represents the write amplification coefficient of the storage medium, represents the bad block growth rate value of the node, represents the read rate of the storage node, represents the rate of the storage node, represents the current available storage unit number of the storage medium, Represents the change in the storage load of the storage node.

[0030] To calculate the decreasing trend of the storage node health for predicting the remaining available life of the storage medium and ensuring the long-term stable operation of the storage node, it is first necessary to obtain multiple key parameters of the storage node, including the write amplification factor, the bad block growth rate, the current read and write rates, the available amount of storage units, and the change in the storage load. For example, in a certain storage server, the recorded write amplification factor is 1.5, and the bad block growth rate is 0.12% (i.e., 0.0012), the read rate of the storage node is 450 MB / s, the write rate is 500 MB / s, the current number of available storage units is 1800 TB, and the measurement result of the change in the storage load is 0.2; Perform specific calculations: ; The calculated health trend value of the storage node , this value represents the health status of the storage node. During the long-term monitoring process, if this value gradually rises, it means that the health status of the storage node is declining. At this time, measures need to be taken, such as adjusting the storage strategy, optimizing the load balance, or replacing the storage medium, and finally obtaining the health trend value of the storage node.

[0031] Please refer to Figure 5 , and the specific steps for obtaining the adjustment of the data replica distribution are as follows: S411: Based on the health trend value of the storage node, detect the original operation data of the storage node, extract the health parameters, compare with the set health decline threshold, and screen the storage nodes with a decreasing storage life to obtain the set of storage nodes with a decreasing storage life; First, collect and organize the historical operation data of all storage nodes, which includes CPU usage rate, memory usage, disk I / O performance, etc. A set of health parameter sets is generated for each node, and this set is determined by the deviation between the historical data average and the real-time data. For example, a specific instance is that the CPU usage rate of a certain node in the data center exceeds 85% for three consecutive days, then it is considered that there is a risk of a decline in the health trend of this node. Then, use a series of preset thresholds, such as the health decline threshold is set to an average CPU usage rate of 85%, a memory usage rate of 90%, and a 50% increase in disk I / O response time, to determine whether the node should be classified as a node with a declining storage life. According to the judgment criteria, calculate the health stability coefficient, which is a value obtained by weighting the above-mentioned various health parameters. Finally, obtain the set of nodes with a declining storage life, which is the result of comparing the health stability coefficient of each node with the health decline threshold. For example, the health stability coefficient of node A in a certain data center is 0.75, which is lower than the threshold of 0.8, so it is classified into the set of nodes with a declining storage life.

[0032] S412: Call the set of nodes with a declining storage life, compare the data replica distribution, identify the data replica storage amount on the nodes, and analyze the migration priority to screen the set of nodes that need to adjust the data replica quantity, using the formula: ; Calculate the node data replica adjustment value, screen the target nodes that need to reduce the data replicas, and obtain the set of replica adjustment target nodes; Among them, represents the node data replica adjustment value, represents the th node's current data replica storage amount, represents the th node's optimal storage capacity, represents the storage balance threshold, represents the total storage upper limit value, represents the th storage node's available storage amount, represents the total number of target nodes, represents the total number of the storage node set; First, calculate the data replica storage amount on each node, which is obtained by reading the data storage logs and real-time monitoring on each node. For example, the current data replica storage amount of node B is 120TB ( TB), then analyze the migration priority of the data. The process involves calculating factors such as data access frequency and data importance score. The scoring criteria are set according to historical access records and user-defined priority parameters. For example, if the access frequency of data file X exceeds 100 times in the past week, its importance score is high. Based on the calculation and scoring results; Among them, is the current data replica storage volume of the node (120TB), is the optimal storage capacity assumption of the node, which is 100TB ( TB), is the storage balance threshold calculated based on the historical mean time between failures (IFMT), which is set to 10 ( ), is the designed total storage upper limit, which is set to 1000TB ( TB), is the available storage volume of the current storage node. Assuming that each node has an average available storage of 80TB and there are 10 nodes in total ( TB, ), substitute into the formula: ; Get the set of nodes that need to adjust the number of data replicas. For example, if the adjustment value of node C is calculated to be 27.61, which is lower than the set adjustment threshold of 50, it is considered that this node needs to reduce the number of data replicas, thus forming a set of target nodes for replica adjustment.

[0033] S413: Based on the set of target nodes for replica adjustment, select storage nodes with stable health status, allocate the storage paths of data replicas, and perform storage migration to obtain the optimized data replica distribution state and implement data replica distribution adjustment; Select nodes with stable health status and strong data storage capabilities as the new storage targets for data replicas, and perform migration and reallocation of data replicas. The main operations involved in the process include data replication, data verification, and path optimization. Data replication is carried out through the internal network to ensure the integrity and consistency of data during transmission. Data verification is achieved through checksum technology to ensure that the data remains consistent before and after storage. Path optimization is determined by analyzing network traffic and communication latency between nodes to ensure the highest access efficiency after data migration. After all steps are completed, the optimized data replica distribution state is finally obtained, and data replica distribution adjustment is implemented. For example, the data replica of node D is successfully migrated to node E, and the access efficiency of node E is expected to increase by 20%, thus significantly improving the data access performance of the entire data center.

[0034] Please refer to Figure 6, the acquisition steps for storage resource distribution adjustment are specifically as follows: S511: Based on the data replica distribution adjustment, extract the storage occupancy, access request volume, and storage load status of storage nodes, calculate the storage load rate, and obtain the storage load status indicator; Collect data on the actual storage capacity and current storage status of each node. For example, if a data center consists of 10 storage nodes, obtain the occupancy and request frequency of each node through real-time monitoring. The data is transmitted to the central database in real time through a network monitoring tool. Storage node 1 currently has a storage occupancy of 75TB and an average of 300 requests per second, while storage node 2 has an occupancy of 50TB and 150 requests. Based on the real-time data, the storage load rate of each node can be further calculated, that is, the storage occupancy of each node divided by its total storage capacity. For example, if the total capacity of storage node 1 is 100TB, its storage load rate is 75%. Without calling external methods or technologies, simply performing simple numerical processing on the collected data can obtain the storage load status indicator, which reflects the usage status and operation efficiency of each storage node.

[0035] S512: Invoke the storage load status indicator, extract the access frequency and storage load rate of access-intensive data, adjust the storage path, and obtain the storage balance coefficient; First, based on the storage load status indicator, considering the load balance between nodes. For example, if the load rate of storage node 1 is as high as 90% while that of storage node 2 is only 40%, then some high-frequency access data needs to be adjusted to be transferred from node 1 to node 2. The decision is made by comparing the storage load rates and access frequencies of each node. After adjustment, it is necessary to recalculate the adjusted storage balance, that is, the storage load difference between nodes is small, which can be calculated through a formula. For example, take the square root of the sum of the squares of the differences between the current storage volume of each node and the ideal storage volume to obtain the storage balance coefficient. The smaller the coefficient, the more balanced the storage distribution. The calculation of the coefficient is a key parameter directly reflecting the effect of the adjustment strategy.

[0036] S513: Invoke the storage balance coefficient, optimize the task storage distribution, analyze the optimal allocation of storage resources, and use the formula: ; Calculate the optimal allocation indicator of storage resources, obtain the storage resource allocation plan, and execute the storage resource distribution adjustment; Among them, represents the optimal allocation indicator of storage resources, represents the storage node 's storage capacity, represents the storage node 's storage utilization rate, represents the task storage requirements represents the remaining storage space of the storage node represents the total number of storage nodes represents the redundancy of data replica distribution indicates the total number of storage nodes; First, conduct a detailed analysis of the storage balance coefficient and the specific data of each storage node. For example, assume there are three storage nodes with storage capacities of 100TB, 150TB, and 120TB respectively, and current utilization rates of 0.75, 0.60, and 0.50 respectively. The actual storage requirements of each node are 80TB, 90TB, and 65TB respectively, and the remaining storage space of each node is 20TB, 60TB, and 60TB respectively. The total number of storage nodes is 3, and the redundancy of data replicas is set to 2, which directly affects the resource allocation strategy and the final storage efficiency; to calculate the optimal allocation metric The formula aims to balance the storage load of each node and minimize the overall storage cost; First, calculate : ; ; ; Therefore, ; Then, calculate : ; ; ; Therefore, and ; Finally, substitute into the formula: ; The optimal allocation metric obtained through this calculation ​It is 46.95. This indicator is used to evaluate and adjust the storage resource allocation scheme, perform storage resource distribution adjustment to ensure the optimization of storage efficiency and access speed. The numerical result shows that the current storage resource allocation is relatively reasonable, but there is still room for improvement, especially in reducing the load of high-load nodes and utilizing the resources of low-load nodes. This calculation method not only provides a quantitative method for storage resource management but also helps data center operators better understand the load balancing state among nodes.

[0037] The cloud storage dynamic optimization system is used to execute the above cloud storage dynamic optimization method. The system includes: The access cycle analysis module calculates the access time interval and standard deviation based on the cloud storage data access records, analyzes the cycle stability, classifies data objects, and obtains the data access cycle stability classification value. The task priority allocation module extracts the access frequency, data volume, and parallel requirements based on the data access cycle stability classification value, calculates the resource occupancy ratio, allocates task priorities, filters high-throughput and low-latency storage devices based on the task scheduling scenario of cloud storage dynamic optimization, matches priority tasks to corresponding devices, and adjusts the task storage path to obtain the storage task priority allocation result. The storage health assessment module analyzes the write amplification factor, bad block growth rate, and read-write load of the storage medium based on the storage task priority allocation result, estimates the remaining life, and obtains the storage node health trend value. The replica distribution adjustment module filters nodes with decreasing life based on the storage node health trend value, adjusts the replica distribution, reduces the number of replicas, creates new replicas based on healthy nodes, and adjusts the migration path to obtain the data replica distribution adjustment result. The storage resource optimization module analyzes the storage load based on the data replica distribution adjustment result, identifies the storage paths of access-intensive data, adjusts the storage distribution, optimizes the resource occupancy, and performs storage resource distribution adjustment.

[0038] The above is only the preferred embodiment of the present invention and does not limit the present invention in other forms. Any person skilled in the art may use the disclosed technical content to make changes or modifications into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change, and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution content of the present invention still fall within the protection scope of the technical solution of the present invention.

Claims

1. A cloud storage dynamic optimization method, characterized in that: The following steps are involved: S1: Based on the cloud storage data access records, identify the access time interval sequence, analyze the variation range of adjacent time intervals, calculate the standard deviation of the access time interval, determine the access cycle stability according to the standard deviation and variation range, and classify the data objects to obtain the data access cycle stability classification value; S2: Based on the data access cycle stability classification value, extract the storage access frequency, storage access data volume and parallel execution requirements, analyze the resource usage of task scheduling, identify the storage task priority, match the high-throughput and low-access latency storage device, adjust the storage path of the task, and generate the storage task priority allocation result; S3: Based on the storage task priority allocation result, extract the storage node hardware status data, evaluate the write amplification factor of the storage medium, the bad block growth rate, and the read and write load changes, analyze the storage node health decline trend, infer the remaining available life, and obtain the storage node health trend value; S4: Based on the health trend value of the storage node, select nodes with declining storage lifespan, reduce the number of data copies, create new copies through healthy and stable nodes, adjust the migration path of storage data, and implement data copy distribution adjustment.

2. The cloud storage dynamic optimization method according to claim 1, characterized in that: The data access cycle stability classification values ​​include high stability category, medium stability category, and low stability category. The storage task priority allocation results include task priority level, storage path allocation, and storage device matching. The storage node health trend value includes health level assessment, remaining available life prediction, and failure risk assessment. The data copy distribution adjustment includes copy quantity optimization, copy storage location adjustment, and data migration path optimization.

3. The cloud storage dynamic optimization method according to claim 1, characterized in that: The steps for obtaining the data access cycle stability classification value are specifically as follows: S111: extracting access time interval sequences based on cloud storage data access records, analyzing the variation range of adjacent time intervals, resolving time intervals at adjacent time points, detecting the variation of time intervals at adjacent time points, and obtaining the variation range value of time intervals; S112: calling the time interval variation amplitude value, identifying the standard deviation of the access time interval, using the difference between each time interval value and the mean value, measuring the degree of deviation, and obtaining the access time interval standard deviation value; S113: Call the access time interval standard deviation value, calculate the access cycle stability value, and classify the data objects according to the stability value, using the formula: ； Obtaining a data access cycle stability classification value; in, Represents the access cycle stability value, Represents the number of visits in the visit cycle, Representative Visit time, Indicates Visit time, Representative The time interval between visits.

4. The cloud storage dynamic optimization method according to claim 3, characterized in that: The steps for obtaining the storage task priority allocation result are specifically as follows: S211: extracting storage access frequency, storage access data volume and parallel execution requirements based on the data access cycle stability classification value, identifying resource occupancy of storage tasks, comparing resource occupancy of each task, and obtaining storage task resource occupancy coefficient; S212: calling the storage task resource occupancy coefficient, analyzing the priority factor of the storage task, and performing priority sorting among the tasks, using the formula: ； Get the storage task priority coefficient; in, Represents the storage task priority coefficient, represents the storage access frequency, Represents the amount of storage access data, Represents the storage task resource occupancy coefficient, Represents the storage base occupancy value, Represents the need for parallel execution of tasks; S213: calling the storage task priority coefficient, matching a high-throughput and low-access-latency storage device, adjusting the storage path of the storage task, and generating a storage task priority allocation result.

5. The cloud storage dynamic optimization method according to claim 4, characterized in that: The steps for obtaining the health trend value of the storage node are specifically as follows: S311: Based on the storage task priority allocation result, extract the storage node hardware status data, determine the abnormal growth trend of the storage node, and calculate the bad block growth rate value of the node; S312: calling the bad block growth rate value of the node, extracting the difference between the current write amplification factor of the storage medium and the threshold value in combination with the storage load change, and making corrections in combination with the write rate change trend to obtain the storage medium write amplification factor; S313: Call the storage medium write amplification factor, combine the bad block growth rate value of the node and the change of the read and write load, analyze the health decline trend of the storage node, and use the formula: ； Estimate the remaining useful life of the storage medium and obtain the health trend value of the storage node; in, Represents the health trend value of the storage node. Represents the storage medium write amplification factor, Represents the bad block growth rate value of the node, Represents the read rate of the storage node, Represents the rate of the storage node, Represents the current number of available storage units of the storage medium. Indicates the storage load changes of the storage node.

6. The cloud storage dynamic optimization method according to claim 5, characterized in that: The acquisition steps of the data replica distribution adjustment are specifically as follows: S411: Based on the health trend value of the storage node, detect the original operation data of the storage node, extract the health parameter, compare it with the set health decline threshold, filter the storage life decline node, and obtain the storage life decline node set; S412: calling the node set with reduced storage lifespan, comparing the data copy distribution, identifying the data copy storage amount on the node, analyzing the migration priority, screening the node set for which the number of data copies needs to be adjusted, and using the formula: ； Calculate the node data replica adjustment value, filter the target nodes that need to reduce data replicas, and obtain the replica adjustment target node set; in, Represents the node data replica adjustment value, Representative The current data replica storage capacity of each node, Representative The optimal storage capacity of each node, Represents the storage balancing threshold, Represents the total storage upper limit, Representative The available storage capacity of storage nodes, Represents the total number of target nodes, Represents the total number of storage nodes; S413: Based on the replica adjustment target node set, select a storage node with a stable and healthy state, allocate a storage path for the data replica, and perform storage migration to obtain an optimized data replica distribution state, and implement data replica distribution adjustment.

7. The cloud storage dynamic optimization method according to claim 1, characterized in that: The method further comprises step S5: S5: Based on the data replica distribution adjustment, analyze the storage load status of cloud storage resource scheduling, adjust the access-intensive data storage path, optimize the storage distribution of storage resources occupied by tasks, perform load distribution of storage resources, and perform storage resource distribution adjustment; The storage resource distribution adjustment includes storage load balancing, access-intensive data optimization, and task storage resource allocation optimization.

8. The cloud storage dynamic optimization method according to claim 7, characterized in that: The steps for obtaining the storage resource distribution adjustment are specifically as follows: S511: Based on the data replica distribution adjustment, extract the storage occupancy, access request volume and storage load status of the storage node, calculate the storage load rate, and obtain the storage load status indicator; S512: calling the storage load status indicator, extracting the access frequency and storage load rate of access-intensive data, adjusting the storage path, and obtaining a storage balance coefficient; S513: Call the storage balance coefficient, optimize the task storage distribution, analyze the optimal allocation of storage resources, and use the formula: ； Calculate the optimal allocation index of storage resources, obtain the storage resource allocation plan, and perform storage resource distribution adjustment; in, represents the optimal allocation index of storage resources, Represents a storage node storage capacity, Represents a storage node Storage utilization, Representative tasks storage requirements, Represents a storage node of remaining storage space, Represents the total number of storage nodes, Represents the redundancy of data copy distribution, Indicates the total number of storage nodes.

9. A cloud storage dynamic optimization system, characterized in that: According to the cloud storage dynamic optimization method according to any one of claims 1 to 8, the system comprises: The access cycle analysis module calculates the access time interval and standard deviation based on the cloud storage data access records, analyzes the cycle stability, classifies the data objects, and obtains the data access cycle stability classification value; The task priority allocation module extracts the access frequency, data volume, parallel requirements, calculates the resource occupancy ratio, allocates task priorities based on the data access cycle stability classification value, and screens high-throughput and low-latency storage devices based on the task scheduling scenario dynamically optimized by cloud storage, matches priority tasks to corresponding devices, adjusts the task storage path, and obtains the storage task priority allocation result; The storage health assessment module analyzes the storage medium write amplification factor, bad block growth rate, and read / write load based on the storage task priority allocation result, estimates the remaining life, and obtains the storage node health trend value; The replica distribution adjustment module screens nodes with declining lifespan based on the health trend value of the storage node, adjusts the replica distribution, reduces the number of replicas, creates new replicas based on healthy nodes, adjusts the migration path, and obtains the data replica distribution adjustment result; The storage resource optimization module analyzes the storage load based on the data copy distribution adjustment result, identifies the storage path of the access-intensive data, adjusts the storage distribution, optimizes the resource occupancy, and performs the storage resource distribution adjustment.

Images