A method for optimizing configuration of a storage resource pool across data centers

Abstract

The application provides a cross-data-center storage resource pool optimization configuration method, and belongs to the technical field of resource pool optimization, and comprises the following steps: step 1, a centralized management platform is constructed, and the storage resources of the data center are scheduled in combination with a distributed storage architecture; step 2, resources are allocated according to a load balancing algorithm, and the allocation result is dynamically adjusted; step 3, the performance and use of the storage resource pool are acquired, and potential storage abnormal conditions are determined; step 4, the optimization strategy of the storage resource pool is acquired according to the potential storage abnormal conditions, and the cross-data-center storage resource pool is optimized and configured; and step 5, the optimization configuration is audited according to a performance test tool, and the optimization configuration of the storage resource pool is adjusted in real time. The cross-data-center storage resource pool optimization configuration method can realize cross-data-center resource acquisition, meanwhile, the allocation of resources is reasonable, the utilization of the resource pool is improved, and the high availability of the resource pool is improved.

Description

Technical Field

[0001] This invention relates to the field of resource pool optimization technology, and in particular to a method for optimizing the configuration of storage resource pools across data centers. Background Technology

[0002] In today's digital age, data centers have become a core component of enterprise IT infrastructure. As data centers continue to expand and increase, managing and utilizing these resources has become increasingly important.

[0003] However, traditional cross-datacenter storage resource pool optimization configuration methods cannot achieve resource acquisition across data centers. At the same time, resources in each data center are allocated to different applications or workloads, but the allocation of these resources is not always reasonable. Some workloads may require more computing resources, but they are allocated to a data center that does not match their workload requirements, which will reduce the utilization of the resource pool and reduce the high availability of the resource pool.

[0004] Therefore, this invention proposes a method for optimizing the configuration of storage resource pools across data centers. Summary of the Invention

[0005] This invention provides a method for optimizing the configuration of storage resource pools across data centers, which solves the problem that existing technologies cannot achieve resource acquisition across data centers. In addition, the resources of each data center are allocated to different applications or workloads, but the allocation of these resources is not always reasonable. Some workloads may require more computing resources, but they are allocated to a data center that does not match their workload requirements, which reduces the utilization rate of the resource pool and reduces the high availability of the resource pool.

[0006] On one hand, the present invention provides a method for optimizing the configuration of storage resource pools across data centers, including:

[0007] Step 1: Build a centralized management platform, and monitor and schedule storage resources in different data centers based on the management platform and in conjunction with a distributed storage architecture;

[0008] Step 2: Allocate the storage resources according to the load balancing algorithm, and dynamically adjust the resource allocation results using artificial intelligence technology;

[0009] Step 3: Based on the adjustment results, obtain the performance and usage of the storage resource pool, and determine potential storage anomalies and their causes based on the performance and usage.

[0010] Step 4: Obtain optimization strategies for storage resource pools based on the potential storage anomalies and their causes, and optimize the configuration of storage resource pools across data centers according to the optimization strategies;

[0011] Step 5: Regularly audit the optimized configuration of the storage resource pool using performance testing tools and techniques, and adjust the optimized configuration of the storage resource pool in real time based on the audit results.

[0012] According to the present invention, a method for optimizing the configuration of storage resource pools across data centers is provided, which constructs a centralized management platform and monitors and schedules storage resources in different data centers based on the management platform and in conjunction with a distributed storage architecture, including:

[0013] Obtain the requirements and objectives of the management platform, and determine the overall architecture of the platform based on the requirements and objectives;

[0014] Based on the overall architecture of the platform and the data acquisition process, the corresponding technology stack is obtained, and a centralized management platform is built based on the technology stack.

[0015] Construct a distributed storage architecture based on distributed file systems and distributed object storage technologies;

[0016] Acquire monitoring tools and combine them with a distributed storage architecture to monitor storage resources in different data centers;

[0017] A data acquisition and processing system is built based on ETL tools, and storage resources of different data centers are scheduled based on the data acquisition and processing system.

[0018] According to the present invention, a method for optimizing the configuration of storage resource pools across data centers allocates storage resources based on a load balancing algorithm and dynamically adjusts the resource allocation results using artificial intelligence technology, including:

[0019] Determine the load balancing algorithm based on the application scenario and obtain the principles of the load balancing algorithm;

[0020] The storage resources are allocated according to the principles of the load balancing algorithm and in conjunction with the load balancer.

[0021] Based on the resource allocation results, obtain various data related to resource allocation, and identify key factors affecting load balancing based on the various data.

[0022] The resource allocation results are dynamically adjusted based on the aforementioned key factors and in conjunction with artificial intelligence technology.

[0023] According to the present invention, a method for optimizing the configuration of a storage resource pool across data centers includes obtaining the performance and usage of the storage resource pool based on the adjustment results, and determining potential storage anomalies and their causes based on the performance and usage.

[0024] Based on the adjustment results, real-time data and feedback mechanisms are collected, and relevant data on the performance and usage of the storage resource pool are obtained based on the real-time data and feedback mechanisms.

[0025] Based on the relevant data, key monitoring indicators are determined, and the health status of the storage resource pool is assessed based on the monitoring indicators.

[0026] Based on the health status, potential storage anomalies are identified, and based on the anomalies and the log information of the storage resource pool, the cause of the anomaly is determined.

[0027] According to the present invention, a method for optimizing the configuration of a storage resource pool across data centers includes obtaining an optimization strategy for the storage resource pool based on potential storage anomalies and their causes, and optimizing the configuration of the storage resource pool across data centers according to the optimization strategy, comprising:

[0028] Based on the potential storage anomalies and their causes, and combined with visualization tools, the status and behavior of the storage resource pool are displayed;

[0029] Determine the correlation between states and behaviors using data analysis tools;

[0030] Optimization strategies for the storage resource pool are obtained based on the correlation between the states and behaviors.

[0031] The storage resource pools across data centers are optimized and configured according to the optimization strategy described above.

[0032] According to the present invention, a method for optimizing the configuration of a storage resource pool across data centers includes periodically auditing the optimized configuration of the storage resource pool using performance testing tools and technologies, and adjusting the optimized configuration of the storage resource pool in real time based on the audit results, comprising:

[0033] Determine the testing cycle, and conduct load and stress tests on the optimized configuration of the storage resource pool based on the testing cycle and in conjunction with performance testing tools and techniques;

[0034] Based on the test results, the optimized configuration of the storage resource pool is audited regularly, and the optimized configuration of the storage resource pool is adjusted in real time based on the audit results.

[0035] The method for optimizing the configuration of storage resource pools across data centers according to the present invention further includes:

[0036] The system is divided into multiple microservices based on microservice architecture and load balancing technology, and requests are distributed using load balancing.

[0037] Data is stored in multiple data centers using a consistent hashing algorithm, and important data is synchronized in real time between different data centers using message queue technology as a data synchronization layer.

[0038] Based on the synchronization results and combined with data replication technology, critical data in the multiple data centers are backed up. Based on the backup results, NAT traversal technology is used to manage the data, thereby realizing a disaster recovery mechanism across data centers.

[0039] According to the present invention, a method for optimizing the configuration of storage resource pools across data centers further includes, before allocating the storage resources according to a load balancing algorithm:

[0040] Obtain the data requirements from the management platform's requesters, determine the data types for scheduling based on the data requirements, and determine the relevant business based on the data types for scheduling.

[0041] Obtain the data retrieval request parameters for relevant business operations, and determine the data scheduling procedure based on the data retrieval request parameters;

[0042] The proportion of scheduling resources for each type of scheduling data under each relevant business is determined based on the data scheduling process.

[0043] Based on the scheduling resource ratio, generate scheduling resource configuration schemes for each type of scheduling data under each relevant service;

[0044] The prior resource allocation ratio and resource efficiency benefits are determined based on the scheduling resource allocation scheme, and the resource optimization decision scheme is determined based on the prior resource allocation ratio and resource efficiency benefits.

[0045] We introduce data loading and data permission constraints for various types of scheduling data, and construct optimization models for each type of scheduling data.

[0046] Based on the resource optimization decision-making scheme for each type of scheduling data, the optimization model for each type of data is used to determine the target values ​​for resource optimization in multiple business scenarios for each type of scheduling data.

[0047] Based on the resource optimization target values ​​for various types of scheduling data across multiple business scenarios, determine the buffer state description parameters and processing state description parameters for storage resources;

[0048] The bidirectional configuration deviation sequence of storage resources in different data centers is determined based on buffer state description parameters and processing state description parameters.

[0049] A feedback gain function is constructed based on the bidirectional configuration deviation sequence, and the storage resources are preloaded and processed according to the bidirectional configuration deviation sequence using a load balancing algorithm.

[0050] The feedback gain function is used to monitor and optimize the data preloading resource allocation results and processing resource results of storage resources in different data centers in real time.

[0051] Compared with the prior art, the beneficial effects of this application are as follows:

[0052] By monitoring and scheduling storage resources across different data centers through a centralized management platform and distributed storage architecture, and allocating them reasonably, the system can identify resource pool anomalies and optimize them according to optimization strategies. This enables resource acquisition across data centers, and the allocation of resources in each data center to different applications or workloads improves the rationality of resource allocation, increases the utilization rate of the resource pool, and enhances the high availability of the resource pool. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0054] Figure 1 This is a flowchart illustrating the method for optimizing the configuration of storage resource pools across data centers provided in an embodiment of the present invention.

[0055] Figure 2 This is a schematic diagram of the process of monitoring and scheduling storage resources in different data centers based on a management platform and a distributed storage architecture, as provided in an embodiment of the present invention. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0057] Example 1:

[0058] This invention provides a method for optimizing the configuration of storage resource pools across data centers, such as... Figure 1 As shown, the method mainly includes the following steps:

[0059] Step 1: Build a centralized management platform, and monitor and schedule storage resources in different data centers based on the management platform and in conjunction with a distributed storage architecture;

[0060] Step 2: Allocate the storage resources according to the load balancing algorithm, and dynamically adjust the resource allocation results using artificial intelligence technology;

[0061] Step 3: Based on the adjustment results, obtain the performance and usage of the storage resource pool, and determine potential storage anomalies and their causes based on the performance and usage.

[0062] Step 4: Obtain optimization strategies for storage resource pools based on the potential storage anomalies and their causes, and optimize the configuration of storage resource pools across data centers according to the optimization strategies;

[0063] Step 5: Regularly audit the optimized configuration of the storage resource pool using performance testing tools and techniques, and adjust the optimized configuration of the storage resource pool in real time based on the audit results.

[0064] In this embodiment, the distributed storage architecture is an architecture that distributes data storage and management across multiple devices or servers.

[0065] In this embodiment, the load balancing algorithm refers to a strategy of distributing tasks and requests among multiple computers or multiple processing units to ensure that these resources and processing capabilities are fully utilized.

[0066] In this embodiment, storage anomalies in the storage resource pool include: overload, overselling, hardware failure, network interruption, and file system errors.

[0067] The beneficial effects of the above technical solution are: by monitoring and scheduling storage resources in different data centers through a centralized management platform and distributed storage architecture, and making reasonable allocations, it can detect abnormal situations in the resource pool and optimize them according to optimization strategies. This enables resource acquisition across data centers. At the same time, the resources in each data center are allocated to different applications or workloads, improving the rationality of resource allocation, increasing the utilization rate of the resource pool, and enhancing the high availability of the resource pool.

[0068] Example 2:

[0069] Based on Embodiment 1, this embodiment of the invention constructs a centralized management platform. This management platform, combined with a distributed storage architecture, monitors and schedules storage resources in different data centers, such as... Figure 2 As shown, it includes:

[0070] S01: Obtain the requirements and objectives of the management platform, and determine the overall architecture of the platform based on the requirements and objectives;

[0071] S02: Based on the overall architecture of the platform and the data acquisition process, obtain the corresponding technology stack, and build a centralized management platform based on the technology stack;

[0072] S03: Construct a distributed storage architecture based on distributed file systems and distributed object storage technologies;

[0073] S04: Obtain monitoring tools and combine them with a distributed storage architecture to monitor storage resources in different data centers;

[0074] S05: Construct a data acquisition and processing system based on ETL tools, and schedule storage resources of different data centers based on the data acquisition and processing system;

[0075] In this embodiment, the overall architecture includes: system architecture, module division, database design, and API design.

[0076] In this embodiment, the technology stack refers to a set of technologies and frameworks required when developing and building software applications, providing a standardized solution to solve problems and achieve specific functions.

[0077] In this embodiment, the distributed file system is a file system that can be accessed over a network and allows data to be shared among multiple computers.

[0078] In this embodiment, distributed object storage technology refers to a technology that enables access, management, and operation of objects stored in a distributed storage system.

[0079] In this embodiment, the distributed storage architecture is an architecture that distributes data storage and management across multiple devices or servers.

[0080] The beneficial effects of the above technical solution are: it monitors and schedules storage resources in different data centers based on a centralized management platform and distributed storage architecture, enables data sharing among multiple data centers, and provides greater elasticity and portability. At the same time, it can monitor and acquire storage resources in all data centers in real time, ensuring data accuracy and availability.

[0081] Example 3:

[0082] Based on Embodiment 2, this embodiment of the invention allocates storage resources according to a load balancing algorithm and dynamically adjusts the resource allocation results using artificial intelligence technology, including:

[0083] Determine the load balancing algorithm based on the application scenario and obtain the principles of the load balancing algorithm;

[0084] The storage resources are allocated according to the principles of the load balancing algorithm and in conjunction with the load balancer.

[0085] Based on the resource allocation results, obtain various data related to resource allocation, and identify key factors affecting load balancing based on the various data.

[0086] The resource allocation results are dynamically adjusted based on the aforementioned key factors and in conjunction with artificial intelligence technology.

[0087] In this embodiment, the load balancing algorithm refers to a strategy of distributing tasks and requests among multiple computers or multiple processing units to ensure that these resources and processing capabilities are fully utilized.

[0088] In this embodiment, the load balancing algorithm follows these principles: In round-robin, each request is assigned to an available processing unit or service instance in a fixed order. In the shortest connection count method, requests are assigned to the processing unit or service instance with the fewest connections. In the IP hash method, requests are assigned to the processing unit or service instance with the same IP address as the client.

[0089] In this embodiment, a load balancer is a technical device used to distribute system load to ensure stable and efficient system operation. It analyzes and selects the best path to distribute traffic from the server, thereby avoiding network congestion and server overload.

[0090] In this embodiment, various data related to resource allocation include: resource availability, demand, and priority.

[0091] In this embodiment, the key factors affecting load balancing include load distribution and network latency.

[0092] The beneficial effects of the above technical solution are: allocating storage resources according to the load balancing algorithm helps to ensure that the service quality of each data center is evenly distributed, avoiding the occurrence of hot spots and cold spots. Furthermore, by combining artificial intelligence technology to dynamically adjust the resource allocation results, resource demand can be estimated more accurately and adjusted accordingly, thereby improving resource utilization and reducing waste.

[0093] Example 4:

[0094] Based on Example 3, this embodiment of the invention obtains the performance and usage of the storage resource pool according to the adjustment results, and determines potential storage anomalies and their causes based on the performance and usage.

[0095] Based on the adjustment results, real-time data and feedback mechanisms are collected, and relevant data on the performance and usage of the storage resource pool are obtained based on the real-time data and feedback mechanisms.

[0096] Based on the relevant data, key monitoring indicators are determined, and the health status of the storage resource pool is assessed based on the monitoring indicators.

[0097] Based on the health status, potential storage anomalies are identified, and based on the anomalies and the log information of the storage resource pool, the cause of the anomaly is determined.

[0098] In this embodiment, the feedback mechanism refers to receiving information from the user during the usage process after a certain process, product, or service has been provided.

[0099] In this embodiment, data related to the performance and usage of the storage resource pool includes: average request latency, throughput, and utilization; for example, data is collected every 5 minutes. When using the storage service, the system displays a feedback form indicating that recent file uploads have been slow. Data collected by the software shows that the average request latency has increased from 5ms to 10ms, throughput has decreased from 100MB / s to 80MB / s, and utilization has increased from 60% to 80%.

[0100] In this embodiment, the monitoring metrics include IOPS, latency, and throughput. Specifically, based on collected data such as average request latency, throughput, and utilization, IOPS (input / output operations per second), latency, and throughput are calculated and determined. The health of the storage resource pool is assessed by comparing these metrics to their normal threshold ranges. For example, the normal range for IOPS is set to 1000-2000, the normal range for latency is 1-5ms, and the normal range for throughput is 80-120MB / s. Based on the previously collected data, IOPS decreased from 1500 to 1200, latency increased from 5ms to 10ms, and throughput decreased from 100MB / s to 80MB / s. Comparing to the normal threshold ranges, it was found that latency exceeded the normal range, and IOPS and throughput were close to the lower limit. Generally, the health status value = the sum of the weights of the corresponding monitoring indicators and the evaluation values. The weights of different indicators are preset, and the sum of the weights of all indicators is 1. If the monitoring value under the corresponding indicator is within the corresponding range, the evaluation value is considered to be 1. Otherwise, if the monitoring value is greater than the maximum value of the corresponding range, the evaluation value = 1 - (maximum value - monitoring value) / maximum value. If the monitoring value is less than the minimum value of the corresponding range, the evaluation value = 1 - (minimum value - monitoring value) / minimum value.

[0101] In this embodiment, based on the assessed health status, and by comparing it with a list of storage anomalies (overload, overselling, hardware failure, network interruption, file system errors, etc.), possible anomalies are identified. For example, increased latency, decreased IOPS, and decreased throughput indicate potential overload or network interruption anomalies. Log information reveals that during the period when users reported slow file upload speeds, the storage resource pool initiated a large data backup task, consuming significant system resources and causing system overload. Therefore, the corresponding cause of the anomaly is: a large data backup task consuming excessive system resources.

[0102] In this embodiment, storage anomalies in the storage resource pool include: overload, overselling, hardware failure, network interruption, and file system errors.

[0103] In this embodiment, the log information of the storage resource pool is used to record the operating status of the storage resource pool, including: the start and stop time of the storage resource pool, and the specific information of each task.

[0104] The beneficial effects of the above technical solution are: by obtaining the performance and usage of the storage resource pool based on the adjustment results, and identifying potential storage anomalies and their causes, the usage status of the resource pool can be determined quickly and accurately, and anomalies can be detected, thereby optimizing the storage resource pool and improving its utilization and performance.

[0105] Example 5:

[0106] Based on Example 4, this embodiment of the invention obtains an optimization strategy for the storage resource pool according to the potential storage anomalies and their causes, and optimizes the configuration of the storage resource pool across data centers according to the optimization strategy, including:

[0107] Based on the potential storage anomalies and their causes, and combined with visualization tools, the status and behavior of the storage resource pool are displayed;

[0108] Determine the correlation between states and behaviors using data analysis tools;

[0109] Optimization strategies for the storage resource pool are obtained based on the correlation between the states and behaviors.

[0110] The storage resource pools across data centers are optimized and configured according to the optimization strategy described above.

[0111] In this embodiment, the state of the storage resource pool refers to the current operating status of the resource pool.

[0112] In this embodiment, the behavior of the storage resource pool includes:

[0113] Dynamically adjust capacity and / or quotas: Automatically adjust the size of the storage resource pool based on changes in demand and load. For example, when the workload decreases, the capacity of the resource pool can be appropriately reduced.

[0114] Adaptive performance adjustment: Automatically adjusts the performance of the storage resource pool based on workload demands and performance requirements. For example, when a new workload is launched, additional IOPS or bandwidth resources can be provided to that workload.

[0115] Continuous monitoring and reporting: Monitor the status and performance of the storage resource pool in real time and generate corresponding reports for analysis and decision-making.

[0116] In this embodiment, the storage resource pool optimization strategy may include:

[0117] Load balancing: Distributing resources to different applications and workloads to ensure they receive the computing and storage resources they need.

[0118] Dynamic adjustment: Automatically adjusts the size and allocation of the storage resource pool based on actual usage. For example, when a task starts consuming a lot of resources, some of the workload can be moved to other available resources to maintain optimal utilization of the resource pool. Conversely, when resources in the pool are idle, the workload can be moved to other resources to save costs.

[0119] The beneficial effects of the above technical solution are: based on potential storage anomalies and their causes, optimization strategies for storage resource pools are obtained, and storage resource pools across data centers are optimized and configured. The optimization strategies can ensure the normal operation of storage resource pools, thereby improving the overall reliability of the system. At the same time, effective optimization strategies can make storage resource pools across data centers more efficient, reduce energy consumption, and improve application response speed.

[0120] Example 6:

[0121] Based on Example 5, this embodiment of the invention periodically audits the optimized configuration of the storage resource pool using performance testing tools and techniques, and adjusts the optimized configuration of the storage resource pool in real time based on the audit results, including:

[0122] Determine the testing cycle, and conduct load and stress tests on the optimized configuration of the storage resource pool based on the testing cycle and in conjunction with performance testing tools and techniques;

[0123] Based on the test results, the optimized configuration of the storage resource pool is audited regularly, and the optimized configuration of the storage resource pool is adjusted in real time based on the audit results.

[0124] In this embodiment, the testing period can be one month or one quarter.

[0125] In this embodiment, load testing is a method to evaluate the performance of a distributed system or application under high load. By simulating multiple users accessing the application at the same time, the behavior of the system when dealing with a large number of concurrent requests can be detected.

[0126] In this embodiment, stress testing is a method to evaluate the stability and performance of a distributed system or application under continuous maximum load. By running the application at a predetermined maximum load level, the scalability and resilience of the system, as well as its ability to function properly in the face of failures, maintenance, or changes in load balancing strategies, can be determined.

[0127] The beneficial effects of the above technical solution are: by regularly auditing the optimized configuration of the storage resource pool based on performance testing tools and technologies, and adjusting the optimized configuration of the storage resource pool in real time, the solution can continuously meet business needs based on the ever-changing business requirements, ensuring that the storage resource pool always maintains the best performance level.

[0128] Example 7:

[0129] Based on Example 6, the embodiments of the present invention further include:

[0130] The system is divided into multiple microservices based on microservice architecture and load balancing technology, and requests are distributed using load balancing.

[0131] Data is stored in multiple data centers using a consistent hashing algorithm, and important data is synchronized in real time between different data centers using message queue technology as a data synchronization layer.

[0132] Based on the synchronization results and combined with data replication technology, critical data in the multiple data centers are backed up. Based on the backup results, NAT traversal technology is used to manage the data, thereby realizing a disaster recovery mechanism across data centers.

[0133] In this embodiment, a microservice architecture breaks down an application into a series of small, independent services that can work together in a loosely coupled manner.

[0134] In this embodiment, load balancing technology is used to distribute incoming requests to multiple processing units (such as servers, proxies, gateways, etc.) to improve system throughput.

[0135] In this embodiment, the consistent hashing algorithm is an algorithm used for distributed hash tables and data distribution management, which aims to provide an efficient and reliable distributed storage solution for a large number of nodes to perform fast lookup, insertion and deletion operations.

[0136] In this embodiment, message queue technology is a method for implementing asynchronous communication in computer application systems. By decoupling the communication process between producers and consumers, message queues can achieve message passing between multiple applications without occupying processing threads.

[0137] In this embodiment, data replication technology refers to creating one or more identical copies in a distributed system to ensure the integrity, reliability, and consistency of data.

[0138] In this embodiment, NAT traversal technology is used to allow communication between devices located on different network protocol stacks.

[0139] In this embodiment, the disaster recovery mechanism refers to the deployment of backup sites or servers in a distributed system to cope with the failure or unavailability of the main site or service, such as master-slave replication or multi-replica replication.

[0140] The beneficial effects of the above technical solution are: by splitting the system into multiple microservices and synchronizing important data in real time between different data centers, a cross-data center disaster recovery mechanism and data backup can be realized, which can effectively disperse risks and avoid data loss when a single data center fails or natural disasters occur, thus ensuring the integrity and security of the data.

[0141] Example 8:

[0142] Based on Embodiment 7, this embodiment of the invention further includes the following before allocating the storage resources according to the load balancing algorithm:

[0143] Obtain the data requirements from the management platform's requesters, determine the data types for scheduling based on the data requirements, and determine the relevant business based on the data types for scheduling.

[0144] Obtain the data retrieval request parameters for relevant business operations, and determine the data scheduling procedure based on the data retrieval request parameters;

[0145] The proportion of scheduling resources for each type of scheduling data under each relevant business is determined based on the data scheduling process.

[0146] Based on the scheduling resource ratio, generate scheduling resource configuration schemes for each type of scheduling data under each relevant service;

[0147] The prior resource allocation ratio and resource efficiency benefits are determined based on the scheduling resource allocation scheme, and the resource optimization decision scheme is determined based on the prior resource allocation ratio and resource efficiency benefits.

[0148] We introduce data loading and data permission constraints for various types of scheduling data, and construct optimization models for each type of scheduling data.

[0149] Based on the resource optimization decision-making scheme for each type of scheduling data, the optimization model for each type of data is used to determine the target values ​​for resource optimization in multiple business scenarios for each type of scheduling data.

[0150] Based on the resource optimization target values ​​for various types of scheduling data across multiple business scenarios, determine the buffer state description parameters and processing state description parameters for storage resources;

[0151] The bidirectional configuration deviation sequence of storage resources in different data centers is determined based on buffer state description parameters and processing state description parameters.

[0152] A feedback gain function is constructed based on the bidirectional configuration deviation sequence, and the storage resources are preloaded and processed according to the bidirectional configuration deviation sequence using a load balancing algorithm.

[0153] The feedback gain function is used to monitor and optimize the data preloading resource allocation results and processing resource results of storage resources in different data centers in real time.

[0154] In this embodiment, the data requirements of the management platform's demanders refer to the management platform's need to acquire, process, and analyze data from different demanders.

[0155] In this embodiment, the scheduling data type refers to data that describes the time, order, frequency, etc. of events or tasks, such as timestamps, time intervals, frequencies, and priorities.

[0156] In this embodiment, the data scheduling process refers to the steps and methods for scheduling data during data processing.

[0157] In this embodiment, the scheduling resource ratio refers to the proportion of available resources allocated according to priority or importance in a resource management system.

[0158] In this embodiment, the scheduling resource allocation scheme is a plan or strategy for managing and allocating resources, which describes how to allocate limited resources to different tasks and activities. The scheduling resource allocation scheme includes: the type and quantity of resources, the priority and urgency of tasks, the availability and constraints of resources, the definition and implementation of workflows and methodologies, and the mechanisms and indicators for monitoring and improvement.

[0159] In this embodiment, prior resource allocation refers to the pre-allocation or division of different types of resources based on various factors when formulating a scheduling resource allocation plan.

[0160] In this embodiment, resource efficiency benefits refer to improving resource utilization and efficiency by optimizing resource allocation and use.

[0161] In this embodiment, the resource optimization decision scheme refers to selecting the optimal resource allocation scheme based on specific circumstances and strategic objectives when faced with multiple resource options.

[0162] In this embodiment, data permission constraints refer to various restrictions and rules imposed on operations such as accessing, modifying, deleting, and sharing data.

[0163] In this embodiment, the multi-service scenario resource optimization target value of scheduling data refers to the resource optimization target value set for each service scenario according to data requirements and priorities under multiple service scenarios. It can be used to measure the utilization rate and optimization effect of data resources.

[0164] In this embodiment, the buffer state description parameters of storage resources refer to various parameters used to describe the buffer state of storage resources, including buffer size, buffer target utilization, buffer time window, and maximum number of times the buffer can be used.

[0165] In this embodiment, the processing status description parameters of the storage resources refer to parameters used to describe various states of the storage resources during the processing process, including the number of files currently being processed, the type of files currently being processed, the access permissions of the files currently being processed, and the modification time of the files currently being processed.

[0166] In this embodiment, the bidirectional configuration deviation sequence of storage resources refers to a sequence describing the configuration deviations of storage resources during deployment and operation. This sequence may include records of configuration errors, as well as configuration changes caused by system failures or human error.

[0167] The beneficial effects of the above technical solution are: by using a feedback gain function to detect and optimize the data preloading resource allocation results and processing resource results of storage resources in different data centers in real time, the feedback gain function can identify and correct configuration deviations, avoid resource waste and unnecessary performance degradation. At the same time, it can detect and correct problems by monitoring the status of the system and storage resources in real time, thereby ensuring the stability and reliability of the system and data.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing the configuration of storage resource pools across data centers, characterized in that, include: Step 1: Build a centralized management platform, and monitor and schedule storage resources in different data centers based on the management platform and in conjunction with a distributed storage architecture; Step 2: Obtain the data requirements from the management platform's requesters, determine the data type for scheduling based on the data requirements, and determine the relevant business based on the data type for scheduling; Obtain the data retrieval request parameters for relevant business operations, and determine the data scheduling procedure based on the data retrieval request parameters; The proportion of scheduling resources for each type of scheduling data under each relevant business is determined based on the data scheduling process. Based on the scheduling resource ratio, generate scheduling resource configuration schemes for each type of scheduling data under each relevant service; The prior resource allocation ratio and resource efficiency benefits are determined based on the scheduling resource allocation scheme, and the resource optimization decision scheme is determined based on the prior resource allocation ratio and resource efficiency benefits. We introduce data loading and data permission constraints for various types of scheduling data, and construct optimization models for each type of scheduling data. Based on the resource optimization decision-making scheme for each type of scheduling data, the optimization model for each type of data is used to determine the target values ​​for resource optimization in multiple business scenarios for each type of scheduling data. Based on the resource optimization target values ​​for various types of scheduling data across multiple business scenarios, determine the buffer state description parameters and processing state description parameters for storage resources; The bidirectional configuration deviation sequence of storage resources in different data centers is determined based on buffer state description parameters and processing state description parameters; wherein, the bidirectional configuration deviation sequence is a set of sequences describing the configuration deviations of storage resources during deployment and operation. A feedback gain function is constructed based on the bidirectional configuration deviation sequence. A load balancing algorithm is used to preload and process the allocation of storage resources according to the bidirectional configuration deviation sequence. The feedback gain function can identify and correct configuration deviations. The data preloading resource allocation results and processing resource results of storage resources in different data centers are detected and optimized in real time by using a feedback gain function; Step 3: Allocate the storage resources according to the load balancing algorithm, and dynamically adjust the resource allocation results using artificial intelligence technology; Step 4: Based on the adjustment results, obtain the performance and usage of the storage resource pool, and determine potential storage anomalies and their causes based on the performance and usage. Step 5: Obtain optimization strategies for storage resource pools based on the potential storage anomalies and their causes, and optimize the configuration of storage resource pools across data centers according to the optimization strategies; Step 6: Regularly audit the optimized configuration of the storage resource pool using performance testing tools and techniques, and adjust the optimized configuration of the storage resource pool in real time based on the audit results.

2. The method for optimizing the configuration of storage resource pools across data centers according to claim 1, characterized in that, Construct a centralized management platform, and monitor and schedule storage resources in different data centers based on the management platform and in conjunction with a distributed storage architecture, including: Obtain the requirements and objectives of the management platform, and determine the overall architecture of the platform based on the requirements and objectives; Based on the overall architecture of the platform and the data acquisition process, the corresponding technology stack is obtained, and a centralized management platform is built based on the technology stack. Construct a distributed storage architecture based on distributed file systems and distributed object storage technologies; Acquire monitoring tools and combine them with a distributed storage architecture to monitor storage resources in different data centers; A data acquisition and processing system is built using ETL tools, and storage resources in different data centers are scheduled based on the data acquisition and processing system.

3. The method for optimizing the configuration of storage resource pools across data centers according to claim 1, characterized in that, The storage resources are allocated according to a load balancing algorithm, and the resource allocation results are dynamically adjusted using artificial intelligence technology, including: Determine the load balancing algorithm based on the application scenario and obtain the principles of the load balancing algorithm; The storage resources are allocated according to the principles of the load balancing algorithm and in conjunction with the load balancer. Based on the resource allocation results, obtain various data related to resource allocation, and identify key factors affecting load balancing based on the various data. The resource allocation results are dynamically adjusted based on the aforementioned key factors and in conjunction with artificial intelligence technology.

4. The method for optimizing the configuration of storage resource pools across data centers according to claim 1, characterized in that, Based on the adjustment results, obtain the performance and usage of the storage resource pool, and determine potential storage anomalies and their causes based on the performance and usage data, including: Based on the adjustment results, real-time data and feedback mechanisms are collected, and relevant data on the performance and usage of the storage resource pool are obtained based on the real-time data and feedback mechanisms. Based on the relevant data, key monitoring indicators are determined, and the health status of the storage resource pool is assessed based on the monitoring indicators. Based on the health status, potential storage anomalies are identified, and based on the anomalies and the log information of the storage resource pool, the cause of the anomaly is determined.

5. The method for optimizing the configuration of storage resource pools across data centers according to claim 1, characterized in that, Based on the potential storage anomalies and their causes, an optimization strategy for the storage resource pool is obtained. This strategy is then used to optimize the configuration of the storage resource pool across data centers, including: Based on the potential storage anomalies and their causes, and combined with visualization tools, the status and behavior of the storage resource pool are displayed; Determine the correlation between states and behaviors using data analysis tools; Optimization strategies for the storage resource pool are obtained based on the correlation between the states and behaviors. The storage resource pools across data centers are optimized and configured according to the optimization strategy described above.

6. The method for optimizing the configuration of storage resource pools across data centers according to claim 1, characterized in that, Regularly audit the optimized configuration of the storage resource pool using performance testing tools and techniques, and adjust the optimized configuration of the storage resource pool in real time based on the audit results, including: Determine the testing cycle, and conduct load and stress tests on the optimized configuration of the storage resource pool based on the testing cycle and in conjunction with performance testing tools and techniques; Based on the test results, the optimized configuration of the storage resource pool is audited regularly, and the optimized configuration of the storage resource pool is adjusted in real time based on the audit results.

7. The method for optimizing the configuration of storage resource pools across data centers according to claim 1, characterized in that, Also includes: The system is divided into multiple microservices based on microservice architecture and load balancing technology, and requests are distributed using load balancing. Data is stored in multiple data centers using a consistent hashing algorithm, and important data is synchronized in real time between different data centers using message queue technology as a data synchronization layer. Based on the synchronization results and combined with data replication technology, critical data in the multiple data centers are backed up. Based on the backup results, NAT traversal technology is used to manage the data, thereby realizing a disaster recovery mechanism across data centers.

Images