Cross-domain multi-data center storage management method for computing and electricity fusion

Abstract

The invention discloses a computing and electricity fusion-oriented cross-domain multi-data center storage management method, which comprises the following steps of: 1) carrying out cross-domain abstract modeling on data center storage resources distributed in different regions: distributing a global unique identifier or path for a data center, and abstracting storage nodes of the data center into a unified logic storage pool; a resource vector is generated according to the storage capacity, the computing power and the power state of the data center and serves as ternary coupling virtualization representation of the data center; 2) performing joint optimization on data storage based on power supply, network bandwidth and storage load, migrating task data determined to be high in concurrency and high in I / O load to a corresponding data center, and reducing energy consumption through cache classification and I / O path optimization; and 3) performing consistency verification on the data copies among the data centers, and performing trend analysis on the load and the electric power of the storage node by using a prediction model based on a time sequence to realize fault tolerance and self-adaptive repair of the storage node.

Description

Technical Field

[0001] This invention belongs to the field of multi-data center storage management technology, and relates to a multi-data center storage system management method for computing and power convergence. Background Technology

[0002] With the explosive growth of data in fields such as large scientific facilities and high-energy physics, traditional computing and storage centers are facing problems such as high energy consumption, uneven geographical distribution, and disconnection from network and power resources.

[0003] How to fully utilize the advantages of green and clean energy in western China and achieve coordinated scheduling of green computing power and storage systems has become a key issue that urgently needs to be addressed.

[0004] Existing multi-datacenter storage management technologies are insufficient in terms of unified abstraction of cross-domain resources, energy-efficient storage scheduling, and data consistency assurance, making it difficult to meet the requirements of high efficiency, low carbon emissions, and stable operation in the context of computing and power convergence. Summary of the Invention

[0005] To address the problems existing in the prior art, the present invention aims to provide a cross-domain multi-datacenter storage management method for computing and power convergence, achieving unified abstraction of cross-domain resources, energy-efficiency-aware storage scheduling, and intelligent data consistency and operation and maintenance, thereby improving the utilization rate of green electricity and enhancing the high-concurrency service capabilities of the storage system. The method includes: cross-domain abstract modeling of storage resources in data centers distributed across different regions; performing a three-element coupled virtualization representation of storage capacity, computing power, and power status; and constructing a unified namespace for global addressing; based on an energy-efficiency-aware scheduling mechanism, incorporating power supply, network bandwidth, and storage load into joint optimization, dynamically adjusting storage data placement, I / O path selection, and hierarchical data migration; and ensuring the consistency and reliable synchronization of data replicas across multiple data centers through cross-domain consistency protocols and intelligent operation and maintenance mechanisms, and achieving fault tolerance and adaptive repair based on AI prediction. This invention achieves efficient energy consumption management, cross-domain collaboration, and highly reliable operation and maintenance in a multi-datacenter environment, effectively improving the green electricity consumption rate and supporting the high-concurrency access needs of large scientific facilities.

[0006] The technical solution of this invention is a cross-domain multi-datacenter storage management method for computing and power convergence, the steps of which include: 1) Perform cross-domain abstract modeling for data center storage resources distributed across different regions: Assign a globally unique identifier or path to the data center, and abstract the storage nodes of the data center into a unified logical storage pool to achieve cross-domain global addressing and transparent access; Generate a resource vector based on the data center's storage capacity, computing power, and power status as a ternary coupled virtualization representation of the data center. 2) Based on power supply, network bandwidth and storage load, data storage is jointly optimized. Data of tasks identified as high concurrency and high I / O load is migrated to the corresponding data center to prioritize the use of clean energy. Energy consumption is reduced through cache grading and I / O path optimization. 3) Perform consistency checks on data replicas across data centers. When a data replica is found to be inconsistent with the corresponding primary replica, repair the data replica by means of log replay or replica reconstruction. Use a time series-based prediction model to perform trend analysis on the load and power of the storage node. When the prediction results indicate that the storage node has potential anomalies or power fluctuations exceed a preset threshold, automatically trigger data replica reconstruction and / or I / O path switching on the storage node to achieve fault tolerance and adaptive repair of the storage node.

[0007] Preferably, the method for jointly optimizing data storage based on power supply, network bandwidth, and storage load, and migrating data identified as high-concurrency, high-I / O-load tasks to the corresponding data center, is as follows: taking the minimization of total energy consumption or the maximization of green energy consumption as the objective function, selecting an optimal data center according to set constraints, and migrating the data identified as high-concurrency, high-I / O-load tasks to the optimal data center; the constraints include network bandwidth, storage load, and data consistency.

[0008] Preferably, with the goal of minimizing total energy consumption while prioritizing the consumption of green electricity, the set of scheduling cycles is denoted as... Data center collection The objective function is to minimize ;in, For data centers In each cycle Non-green power consumption generated by internal storage and I / O tasks For each cycle Internal Data Center The energy consumption of using green electricity to perform the same task These are the weighting coefficients; the constraints when solving the objective function include: a) storage capacity constraints, i.e., in any scheduling cycle... Inside, placed in the data center The sum of the sizes of all objects on the data center cannot exceed the data center's size. Maximum available capacity b) Bandwidth constraint, i.e., within any scheduling period Inside, data center The actual total bandwidth consumption cannot exceed its maximum bandwidth limit. c) Delay constraints, i.e., for each request Response latency Cannot exceed the maximum latency limit .

[0009] Preferably, maximizing green electricity consumption means that the increase in the proportion of green electricity supply to the data center is greater than a set proportion threshold while the electricity price is lower than a set threshold.

[0010] Preferably, data identified as high-concurrency, high-I / O-load tasks are migrated to the corresponding data center through a hierarchical migration mechanism, and the client's I / O request routing is dynamically adjusted. The hierarchical migration mechanism is as follows: based on the access frequency and importance of the data, the data is divided into three levels: hot, warm, and cold, with priority given to migrating hot data to maximize energy efficiency benefits while ensuring the service quality of frequently accessed paths.

[0011] Preferably, the method for verifying the consistency of data replicas among data centers is as follows: select a data center responsible for processing write requests and generating operation logs, use the data of this data center as the primary replica data, and replicate it to other data centers; periodically compare the checksum of the primary replica data with the checksum of each data replica, and perform consistency verification based on the checksum.

[0012] Preferably, a distributed metadata management system is used to collect physical resource information and real-time status of storage nodes in each data center; the physical resource information includes storage capacity, performance, and type.

[0013] The present invention provides a multi-datacenter storage system management method for a computing-electronics convergence context, comprising the following steps: 1. Cross-domain abstract modeling: Cross-domain abstract modeling is performed on data center storage resources distributed in different regions. Storage capacity, computing power, and power status are represented by a three-element coupling virtualization, and a unified namespace is constructed to achieve global addressing. 2. Energy Efficiency Aware Scheduling: Based on an energy efficiency awareness scheduling mechanism, the power supply, network bandwidth, and storage load of each data center are jointly optimized to dynamically adjust storage data placement, I / O path selection, and hierarchical data migration. 3. Intelligent Operation and Maintenance Assurance: Through cross-domain consistency protocols and intelligent operation and maintenance mechanisms, the consistency and reliable synchronization of data replicas across multiple data centers are ensured, and fault tolerance and adaptive repair are achieved based on AI prediction.

[0014] The cross-domain abstract modeling is specifically achieved through distributed metadata management and business semantic-level resource modeling, so that storage nodes in different regions are presented to users as a unified logical storage pool.

[0015] The ternary coupled virtualization representation includes: collecting and updating in real time the storage capacity, computing power utilization, and real-time power status (including the proportion of green power supply) of each data center, and generating a three-dimensional resource vector for unified management.

[0016] When the energy efficiency sensing and scheduling mechanism detects in real time that a data center has sufficient green power supply and the best price, it uses a data hierarchical migration strategy to migrate data copies of high-concurrency, high-I / O load tasks to that data center to achieve priority consumption of green power.

[0017] The cross-domain consistency protocol employs a combination of primary replica log replication and multi-replica consistency verification to ensure data consistency in high-concurrency scientific computing access scenarios.

[0018] The intelligent operation and maintenance mechanism includes: using an AI prediction model to predict the load, network latency, and power fluctuations of storage nodes and generating a potential risk assessment report; when the AI ​​prediction model detects a potential fault or power fluctuations exceeding a threshold, automatically triggering data replica reconstruction and / or I / O path switching operations to achieve adaptive repair.

[0019] The advantages of this invention are as follows: 1) Achieve unified abstraction and transparent access to storage resources across multiple data centers; 2) Improve the utilization rate of green electricity and reduce the overall energy consumption of the storage system; 3) Improve the consistency and availability of cross-domain storage systems to ensure the stable operation of high-concurrency scientific computing. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall architecture of the multi-data center storage management method described in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the logical structure of cross-domain storage resource abstraction and unified namespace as described in an embodiment of the present invention.

[0022] Figure 3 This is a schematic diagram of the energy efficiency-aware storage scheduling process described in an embodiment of the present invention.

[0023] Figure 4 This is a flowchart illustrating the cross-domain consistency and intelligent operation and maintenance mechanism described in this embodiment of the invention. Detailed Implementation

[0024] The present invention will now be described in further detail with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0025] Example 1: Cross-domain storage resource abstraction and unified naming See Figure 1 and Figure 2This embodiment uses distributed metadata management and semantic-level resource modeling to abstract storage nodes in different regions into a unified logical storage pool, enabling cross-domain global addressing and transparent access. To ensure feasibility, distributed metadata management is used to collect the capacity, performance, and operating status of each data center; the ternary coupled virtualization representation is used to receive computing power status (such as CPU / GPU utilization) and power status (such as real-time power supply and green electricity ratio) and abstract them together with storage capacity into a resource vector; a unified namespace provides a globally unique identifier or path, allowing upper-layer applications to access the system without being aware of the underlying location.

[0026] Overall framework of multi-datacenter storage management method Figure 1 As shown, the bottom layer (physical layer) includes two data centers (DC1, DC2) with different geographical locations; the middle layer (management layer) includes a three-element coupled virtualization module, a distributed metadata management system, and a unified namespace service; the scheduling layer includes an energy efficiency-aware scheduler; the upper layer (service layer) includes a client / large scientific facility application that accesses data through a unified namespace interface; and the operation and maintenance layer includes an intelligent operation and maintenance module that connects to each data center and an AI prediction model.

[0027] This embodiment 1 aims to achieve unified management and transparent access to storage resources across multiple regions, including the following:

[0028] Distributed Metadata Management System (MDMS): Deploy a highly available distributed metadata management system. This system is responsible for collecting physical resource information (capacity, performance, type) and real-time status of storage nodes in various data centers (DC1, DC2, DC3...).

[0029] The three-dimensional coupled virtualization representation: MDMS receives computing power status (such as CPU / GPU utilization) from the computing power scheduling layer (such as Kubernetes) and power status (such as real-time power supply and green electricity ratio) from the power scheduling layer (such as EMS). The system abstracts the three dimensions of storage capacity, computing power, and power into a three-dimensional resource vector R=(S,C,E) to virtualize each data center.

[0030] Unified Namespace (UNS): Based on a unified interface of object storage or file system, a global UNS is built. Users can access data through a unified, globally unique ID or path. The underlying data placement and I / O routing are transparent to upper-layer applications. Business semantic-level resource modeling allows the system to match the most suitable storage resources (such as performance-level or capacity-level) according to the application type (such as high-energy physics experimental data or gene sequencing data).

[0031] Example 2: Energy-Aware Storage Scheduling and Data Migration See Figure 3This embodiment, based on the joint optimization of power supply, network bandwidth, and storage load, migrates high-concurrency, high-I / O load task data to the corresponding center when surplus green electricity is detected, prioritizing the consumption of clean energy, and reducing energy consumption through tiered caching and I / O path optimization. The joint optimization, without altering the subject matter of this invention, employs a comprehensive assessment of real-time power status, link bandwidth, and current load to ensure that migration and scheduling are performed while meeting service performance requirements. When conditions are met, a tiered migration strategy prioritizes frequently accessed data to ensure a balance between energy efficiency gains and business continuity. I / O path optimization dynamically selects paths among multiple candidate data centers, directing new read / write requests to locations with sufficient green electricity and moderate service load.

[0032] This embodiment 2 is based on ternary coupling information to perform energy efficiency optimization scheduling; it mainly includes the following contents.

[0033] Joint optimization model: The scheduling mechanism sets the objective function to minimize total energy consumption or maximize green energy consumption, with constraints including network bandwidth, storage load (I / O latency) and data consistency.

[0034] Under the premise of meeting the constraints of storage capacity, network bandwidth, I / O latency, and data replica consistency of each data center, the scheduling mechanism of this invention aims to minimize total energy consumption while prioritizing the consumption of green electricity. Let the set of scheduling periods be denoted as... Data center collection The objective function can then be expressed as: among all feasible scheduling schemes, minimize In each cycle Inside: Data Center Non-green power consumption generated by storage and I / O tasks; Data Center Energy consumption for performing the same task using green electricity; Weighting coefficients are used to balance the importance of total energy consumption and green energy consumption.

[0035] By using the above objective function, without violating bandwidth, storage load (I / O latency) and data consistency constraints, high-load tasks are prioritized to be scheduled to centers with sufficient green electricity and satisfactory network conditions, thereby achieving energy consumption optimization and priority consumption of green electricity.

[0036] The parameters corresponding to the constraints appear in the form of inequality constraints and are not directly written into the objective function; they can also be added to the objective through penalty terms in the extended form, but in this invention, they are mainly given in the form of explicit constraints, which mainly include the following three items.

[0037] 1. Storage capacity constraints: set up For the first The size of each data object For object In the cycle Should it be placed in a data center? The decision variables (0 / 1). For data centers If the storage capacity limit is: Meaning: In any scheduling cycle Inside, placed in the data center The sum of the sizes of all objects on the data center cannot exceed the maximum available capacity of the data center. .

[0038] 2. Bandwidth constraints: set up For object Bandwidth usage For data centers If the upper limit of the inbound and outbound bandwidth is: Meaning: In any scheduling cycle Inside, data center The actual total bandwidth consumption cannot exceed its maximum bandwidth limit. .

[0039] 3. Delay constraints: set up For the first Response latency for each request If the system's allowed latency is the upper limit, then: Meaning: every request Response latency Neither can exceed the maximum allowed latency limit of the system. .

[0040] Based on the above constraints and objective function, the overall meaning should be: to minimize total energy consumption or maximize green energy consumption while satisfying the three types of constraints.

[0041] Green electricity priority consumption strategy: When a significant increase in the proportion of green electricity supply in a western data center (such as DC-West) is detected and the electricity price is below a threshold, the scheduler will assess the data required for the current high computing load tasks (such as training tasks) and migrate hot data copies from centers with low green electricity ratios (such as DC-East) to DC-West through a data tiered migration mechanism.

[0042] I / O path optimization: Dynamically adjust the routing of client I / O requests, directing new read and write requests to data centers with sufficient green electricity and moderate load. The above migration and routing are only performed under the premise of meeting joint optimization constraints to ensure a balance between performance and energy efficiency.

[0043] Data tiered migration: Based on access frequency and importance, data is categorized into three levels: hot, warm, and cold. During scheduling, hot data is migrated first to maximize energy efficiency while ensuring service quality for frequently accessed paths.

[0044] Example 3: Cross-Domain Consistency Maintenance and Intelligent Operation and Maintenance See Figure 4 This embodiment employs a protocol combining log replication and multi-replica verification to ensure data consistency across multiple data centers. After a write request generates an operation log at the primary replica, it is replicated to other replicas via an efficient link, and consistency checks are performed periodically between replicas. When inconsistencies are detected, repairs are initiated through log replay or replica reconstruction. To improve operational stability and predictability, a time-series-based predictive model is used to analyze trends in storage node load and power fluctuations. When prediction results indicate potential anomalies or fluctuations exceeding preset thresholds, replica reconstruction and / or I / O path switching are automatically triggered, achieving fault tolerance and adaptive repair.

[0045] This embodiment 3 utilizes protocols and intelligent prediction to ensure the high reliability of the system, including the following:

[0046] Cross-domain consistency protocol: An improved version of the Primary-Backup protocol. One data center is designated as the primary replica, responsible for handling write requests and generating operation logs. The logs are replicated to other data centers (Backup) via efficient network links. The multi-replica consistency verification module periodically compares the checksums of each replica; if an inconsistency is found, log replay or replica reconstruction is immediately initiated.

[0047] Intelligent Operation and Maintenance (AI Prediction): Train an AI model based on time series analysis. Input parameters include: historical storage node load, real-time network latency, 24-hour power load forecasts provided by the power company, and weather data (affecting solar / wind power).

[0048] Adaptive Repair: 1) Fault prediction: When the AI ​​model predicts that the load or I / O latency of a certain storage node will exceed the preset threshold within the next 30 minutes, the operation and maintenance mechanism will immediately mark it as "warning" and automatically switch new I / O requests to healthy nodes.

[0049] 2) Fault tolerance: When a power fluctuation in a data center is detected that may cause a large-scale outage, the system quickly switches the identity of all primary replicas of that data center to a data center with stable green power and normal load, and completes replica repair according to the consistency protocol to ensure service continuity.

[0050] Although specific embodiments of the invention have been disclosed for illustrative purposes to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A cross-domain multi-datacenter storage management method for computing and power convergence, comprising the following steps: 1) Perform cross-domain abstract modeling for data center storage resources distributed across different regions: Assign a globally unique identifier or path to the data center, and abstract the storage nodes of the data center into a unified logical storage pool to achieve cross-domain global addressing and transparent access; Generate a resource vector based on the data center's storage capacity, computing power, and power status as a ternary coupled virtualization representation of the data center. 2) Based on power supply, network bandwidth and storage load, data storage is jointly optimized. Data of tasks identified as high concurrency and high I / O load is migrated to the corresponding data center to prioritize the use of clean energy. Energy consumption is reduced through cache grading and I / O path optimization. 3) Perform consistency checks on data replicas across data centers. When a data replica is found to be inconsistent with the corresponding primary replica, repair the data replica by means of log replay or replica reconstruction. Use a time series-based prediction model to perform trend analysis on the load and power of the storage node. When the prediction results indicate that the storage node has potential anomalies or power fluctuations exceed a preset threshold, automatically trigger data replica reconstruction and / or I / O path switching on the storage node to achieve fault tolerance and adaptive repair of the storage node.

2. The method according to claim 1, characterized in that, The method for jointly optimizing data storage based on power supply, network bandwidth, and storage load, and migrating data identified as high-concurrency, high-I / O-load tasks to the corresponding data center, is as follows: with the objective function of minimizing total energy consumption or maximizing green energy consumption, an optimal data center is selected according to the set constraints, and the data identified as high-concurrency, high-I / O-load tasks is migrated to the optimal data center; the constraints include network bandwidth, storage load, and data consistency.

3. The method according to claim 2, characterized in that, With the goal of minimizing total energy consumption while prioritizing the consumption of green electricity, let the set of scheduling cycles be denoted as . Data center collection The objective function is to minimize ;in, For data centers In each cycle Non-green power consumption generated by internal storage and I / O tasks For each cycle Internal Data Center The energy consumption of using green electricity to perform the same task These are the weighting coefficients; the constraints when solving the objective function include: a) storage capacity constraints, i.e., in any scheduling cycle... Inside, placed in the data center The sum of the sizes of all objects on the data center cannot exceed the data center's size. Maximum available capacity b) Bandwidth constraint, i.e., within any scheduling period Inside, data center The actual total bandwidth consumption cannot exceed its maximum bandwidth limit. c) Delay constraints, i.e., for each request Response latency Cannot exceed the maximum latency limit .

4. The method according to claim 2, characterized in that, Maximizing green electricity consumption means that the increase in the proportion of green electricity supply to the data center is greater than a set threshold while the electricity price is lower than a set threshold.

5. The method according to claim 1, 2, or 3, characterized in that, Data identified as high-concurrency, high-I / O-load tasks will be migrated to the corresponding data center through a tiered migration mechanism, and the client's I / O request routing will be dynamically adjusted. The tiered migration mechanism is as follows: based on the access frequency and importance of the data, the data is divided into three levels: hot, warm, and cold. Hot data will be migrated first to maximize energy efficiency benefits while ensuring the service quality of frequently accessed paths.

6. The method according to claim 1, characterized in that, The method for verifying the consistency of data replicas among data centers is as follows: Select a data center that is responsible for processing write requests and generating operation logs, use the data of this data center as the primary replica data, and replicate it to other data centers; periodically compare the checksum of the primary replica data with the checksum of each data replica, and perform consistency verification based on the checksum.

7. The method according to claim 1, 2, or 3, characterized in that, A distributed metadata management system is used to collect physical resource information and real-time status of storage nodes in various data centers; the physical resource information includes storage capacity, performance, and type.

Images