Data-in-data electric power data storage and backup method and system based on use popularity

Abstract

The invention relates to a method and a system for storing and backing up electric power data of a data-in-data station based on use popularity, and belongs to the technical field of data-in-data stations, and the method comprises the following steps: classifying electric power business data into hot data, temperature data and cold data based on preset access frequency, time window and data volume index; according to a classification result and a business importance degree, storing hot data in an SSD cloud disk or a memory database, storing temperature data in a distributed column database, and compressing cold data and then archiving the cold data to an object storage or a tape library; dynamically configuring differentiated backup strategies for different types of data; automatic scheduling of storage and backup is realized through a cloud platform component, and hot switching of master and slave databases, OSS archived data recovery by a DataWorks tool and a remote multi-copy recovery mechanism are supported; and migrating data in advance in combination with a historical access prediction popularity trend, and triggering cold data compression and archiving when storage resources exceed 80%. The method is suitable for power industry scenes, the storage cost is optimized, and the data calling efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data middle platform technology, and more specifically, to a data middle platform power data storage and backup method and system based on usage popularity. Background Art

[0002] At present, the data middle platform, as an enterprise-level data capability reuse platform, has been widely used in various industries, including the power industry. It integrates data resources inside and outside the enterprise, provides data collection, integration, analysis and application services, and supports enterprises to realize data empowerment and business innovation. In the power industry, the application of the data middle platform is particularly critical. It can help power grid companies realize centralized management, standardized processing and in-depth analysis of data, and provide strong data support for the intelligent operation of the power grid. With the continuous expansion and deepening of business in various industries, the amount of data gathered by the middle platform is growing rapidly, but different business needs have significant differences in the demand for data and the frequency of use. At present, the middle platform has not yet implemented differentiated data operations. At a time when storage resources are becoming increasingly tight, the storage space capacity of the middle platform continues to be high, which brings great pressure to the operation and maintenance of the enterprise middle platform.

[0003] Existing technologies, such as the Chinese patent application with publication number "CN119377234A", disclose a data tiered storage method, including: stratifying the data storage architecture into a hot data layer, a warm data layer, and a cold data layer; regularly ingesting data from the hot data layer and loading it into the DuckDB internal table of the warm data layer; converting and processing the data in the warm data layer in DuckDB to form cold standby target data; transferring the cold standby target data to the cold data layer; wherein the hot data ingestion and loading tasks, the warm data conversion tasks, and the data cold standby tasks are coded according to the Dagster task framework, the code is built and published as a container image, and task scheduling configuration, task triggering, and task execution are performed. Warm data uses DuckDB for data conversion and low-frequency queries; the cold data layer uses object storage to store historical data in Parquet format to meet the needs of low-cost and efficient analysis. Tiered storage combined with cloud-native task scheduling technology solves the complexity of traditional task scheduling and the inefficiency of resource allocation.

[0004] The problem with the above-mentioned existing technology is that it only divides hot and cold data based on time windows (such as hot data within two years) and access frequency, without introducing business importance indicators, which may result in critical data not being allocated to high-performance storage, affecting business continuity; the lack of a backup mechanism may lead to insufficient backup of high-frequency data or excessive backup of low-frequency data, resulting in low resource utilization. Summary of the Invention

[0005] In order to solve the above technical problems, the present invention proposes a data center power data storage and backup method and system based on usage popularity.

[0006] The technical solutions of the present invention are as follows:

[0007] The present invention proposes a method for storing and backing up power data in a data center based on usage popularity, comprising the following steps:

[0008] Based on preset access frequency, time window and data volume indicators, business data is divided into hot data, warm data and cold data;

[0009] Based on the data classification results, combined with the data's business importance and storage costs, hot data, warm data, and cold data are stored in high-performance storage media, balanced storage media, and low-cost archiving media respectively;

[0010] Dynamically configure differentiated backup strategies for different types of business data;

[0011] Automated scheduling of data storage, backup, and recovery is achieved through cloud platform components, and data availability is guaranteed based on a master-slave architecture and off-site multi-copy mechanism.

[0012] As a preferred embodiment, the business data is divided into hot data, warm data and cold data based on the preset access frequency, time window and data volume indicators, wherein the classification rules for structured transaction data are:

[0013] Hot data: the number of visits within 7 days is greater than 100,000 and the data volume is greater than 0;

[0014] Warm data: Meets any of the following conditions:

[0015] The number of visits within 7 days is ≤ 100,000 and the number of visits within 30 days is > 0;

[0016] The data volume is greater than 0 and the number of visits within 30 days is greater than 0;

[0017] Cold data: Number of visits within 30 days = 0 or data volume = 0.

[0018] As a preferred embodiment, the business data is divided into hot data, warm data and cold data based on the preset access frequency, time window and data volume indicators, wherein the classification rules for unstructured data are:

[0019] Hot data: the number of calls in the past day is greater than 0 and the number of calls is less than the number of calls in the past 7 days;

[0020] Warm data: Meets any of the following conditions:

[0021] Number of calls in the past day = number of calls in the past 7 days;

[0022] The number of calls in the past day = 0 and the number of calls in the past 7 days > 0;

[0023] Cold data: Number of calls in the past 7 days = 0.

[0024] As a preferred embodiment, the importance of the data services is divided into the following levels:

[0025] Important data: data related to real-time monitoring of the power grid or core business operations;

[0026] General data: data related to historical measurements or periodic statistics of equipment;

[0027] Unimportant data: log, debugging, or temporary backup data.

[0028] As a preferred embodiment, the hot data, warm data, and cold data are stored in high-performance storage media, balanced storage media, and low-cost archiving media respectively; wherein:

[0029] High-performance storage media include: SSD cloud disks and in-memory databases;

[0030] Balanced storage media include: distributed column database;

[0031] Low-cost archival media include object storage and tape libraries.

[0032] As a preferred embodiment, the differential backup strategy includes: for structured transactional data backup strategy:

[0033] Hot data:

[0034] Important data: Perform full backup every day and retain backup files for the past 30 days;

[0035] General data: Perform full backup every two days, and retain the last seven backup files;

[0036] Non-important data: Perform a full backup every three days, and retain the last seven backup files;

[0037] Temperature data:

[0038] Important data: Perform full backup every day and retain the backup files of the last 20 times;

[0039] General data: Perform a full backup every three days, and retain the five most recent backup files;

[0040] Non-important data: Perform full backup every week and keep the most recent two backup files;

[0041] Cold data:

[0042] Important data: Perform full backups daily and retain the most recent 15 backup files;

[0043] General data: Perform full backup every week and keep the last three backup files;

[0044] Non-important data: Perform a full backup every month and retain the two most recent backup files.

[0045] As a preferred embodiment, the automated scheduling of data storage, backup and recovery is achieved through the cloud platform component, wherein the implementation of data recovery includes:

[0046] Restore transactional data through hot switching of master and slave databases;

[0047] Use DataWorks to restore OSS archived data to the online database.

[0048] Trigger an off-site copy through the cloud platform console to restore unstructured data.

[0049] As a preferred embodiment, the method includes a dynamic optimization step:

[0050] Predict data popularity based on historical access frequency and migrate predicted hot data to high-performance storage in advance.

[0051] When the storage resource usage exceeds 80%, the cold data compression and archiving process is automatically triggered.

[0052] As a preferred embodiment, the method, when applied to power data, includes:

[0053] Classify real-time power grid monitoring data as hot data, store it in an in-memory database, and perform daily incremental backups;

[0054] Classify equipment measurement historical data as warm data, store it in a distributed database, and back up the full data weekly;

[0055] Log data is classified as cold data, compressed, stored in OSS, and archived quarterly.

[0056] On the other hand, the present invention also provides a data center power data storage and backup system based on usage popularity, comprising:

[0057] The data classification module divides business data into hot data, warm data, and cold data based on preset access frequency, time window, and data volume indicators;

[0058] The storage module stores hot data, warm data, and cold data in high-performance storage media, balanced storage media, and low-cost archiving media, respectively, based on data classification results, combined with data business importance and storage costs.

[0059] The backup module dynamically configures differentiated backup strategies for different types of business data;

[0060] The data recovery and call module realizes the automatic scheduling of data storage, backup and recovery through cloud platform components, and ensures data availability based on the master-slave architecture and multi-replica mechanism in different locations.

[0061] The present invention has the following beneficial effects:

[0062] 1. Tiered storage reduces costs and increases efficiency: Aiming at the data diversity in the enterprise middle platform, we precisely match data popularity classification with multi-level storage media (SSD / distributed library / OSS) to store lower-interest data in low-cost storage media, reducing storage costs by more than 40%. At the same time, high-performance storage media ensures real-time access to high-interest data.

[0063] 2. Dynamic backup resource optimization: Differentiated backup strategies are implemented based on data popularity and business value, and innovative elastic backup mechanisms are designed. For example, high-frequency full backup of important hot data and low-frequency archiving of cold data are implemented. Different backup strategies eliminate the backup of unnecessary data, reducing the amount of redundant backup data by 50% and shortening the backup window.

[0064] 3. Intelligent prediction improves availability: Based on historical access popularity prediction and storage compression triggering mechanism, data migration status can be predicted in advance, achieving dynamic balance of resource utilization and reducing the complexity of manual operation and maintenance.

[0065] 4. Deep adaptation to power scenarios: Classification rules are designed based on the characteristics of real-time grid monitoring, equipment measurement, and log data. The master-slave architecture and remote multi-copy mechanism ensure the availability of core power business data. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0067] Figure 1 Schematic diagram of the process of the present invention;

[0068] Figure 2 This is a flowchart of structured transaction data storage and master-slave hot standby recovery;

[0069] Figure 3 Flowchart for stream computing, backup and recovery of structured business detail data;

[0070] Figure 4 Provide a flowchart for layered processing and cross-platform backup of structured offline analytical data;

[0071] Figure 5 Provides a flowchart for structured service shared data API encapsulation and full database backup and recovery;

[0072] Figure 6 A flowchart for storage, backup and recovery of measurement and analysis result data on the application side;

[0073] Figure 7 Flowchart for multi-region redundant storage, backup and recovery of unstructured data. DETAILED DESCRIPTION

[0074] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0075] It should be understood that the step numbers used herein are only for convenience of description and are not intended to limit the order in which the steps are to be executed.

[0076] It should be understood that the terms used in the present specification are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the present specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise.

[0077] The terms “include” and “comprising” indicate the presence of described features, integers, steps, operations, elements and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0078] The term "and / or" refers to and includes any and all possible combinations of one or more of the associated listed items.

[0079] Example 1:

[0080] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following will be combined with the specific embodiments of the present application and refer to the attached Figure 1 , clearly and completely describe the technical solution of the present invention.

[0081] To solve the problems of the prior art, the present invention provides a method for storing and backing up power data in a data center based on usage popularity, comprising the following steps:

[0082] Based on preset access frequency, time window and data volume indicators, business data is divided into hot data, warm data and cold data;

[0083] Based on the current status of the data center, the hot, warm, and cold data evaluation indicators are categorized into five indicators: data table creation time, data table access time, data table access count, data table read count, and data table entry count. Based on the results of the previous data survey, rules were formulated based on the survey data table creation time, data table access time, data table access count within 1 day, 3 days, 7 days, and 30 days, data table read count within 1 day, 3 days, 7 days, and 30 days, and data table entry count. The data table proportions and data capacity proportions for the three dimensions of hot, warm, and cold were determined under various combination rules, resulting in a data classification strategy for the three dimensions of hot, warm, and cold.

[0084] Based on the preset access frequency, time window, and data volume indicators, business data is divided into hot data, warm data, and cold data. The classification rules for structured transactional data are as follows:

[0085] Hot data: the number of visits within 7 days is greater than 100,000 and the data volume is greater than 0;

[0086] Warm data: Meets any of the following conditions:

[0087] The number of visits within 7 days is ≤ 100,000 and the number of visits within 30 days is > 0;

[0088] The data volume is greater than 0 and the number of visits within 30 days is greater than 0;

[0089] Cold data: Number of visits within 30 days = 0 or data volume = 0.

[0090] Based on the preset access frequency, time window and data volume indicators, business data is divided into hot data, warm data and cold data, wherein the classification rules for unstructured data are:

[0091] Hot data: the number of calls in the past day is greater than 0 and the number of calls is less than the number of calls in the past 7 days;

[0092] Warm data: Meets any of the following conditions:

[0093] Number of calls in the past day = number of calls in the past 7 days;

[0094] The number of calls in the past day = 0 and the number of calls in the past 7 days > 0;

[0095] Cold data: Number of calls in the past 7 days = 0.

[0096] In this embodiment, the data classification strategy divides data into structured data, measurement data, and unstructured data, where:

[0097] Structured data includes: transactional data, business detail query, offline data and service sharing data.

[0098] Transactional data is stored in a PostgreSQL database. The amount of data in the table in the database is relatively small, and the access frequency is relatively high. The data usage popularity classification strategy is shown in Table 1:

[0099] Table 1 Structured transaction data heat classification strategy table

[0100]

[0101]

[0102] For business detail queries, the structured business detail query data stored in the Alibaba OTS component is determined based on the collected access count indicator values. Data with a query frequency higher than once per minute (equivalent to 1,440 accesses per day) in the past day and accessed daily are considered hot data. Furthermore, data with a query frequency greater than 300,000 in the past 30 days is considered warm data. The data usage heat classification strategy is shown in Table 2:

[0103] Table 2 Structured business details query popularity classification strategy table

[0104]

[0105] Offline data is stored in MaxCompute, Alibaba's big data computing service. Currently, the data center is primarily used for data storage and computing. Based on previous research on different data dimensions of components, the data usage popularity classification strategy is shown in Table 3:

[0106] Table 3 Structured offline data heat classification strategy table

[0107]

[0108]

[0109] For service shared data stored in Alibaba RDS / DRDS and ADB databases, structured service shared data is primarily classified as hot data based on whether it is accessed regularly on average per week and whether the number of accesses has increased linearly over the past 30 days. Warm data is classified as cold data based on whether it has been accessed within the past 30 days and whether there are only access records for the past three days. The data is classified as hot data based on its heat classification strategy as shown in Table 4:

[0110] Table 4 Structured service shared data popularity classification strategy table

[0111]

[0112] Measurement data includes: transactional data, offline data, business detail query and service sharing data, including:

[0113] For measurement transaction data, the amount of data in the database tables storing transaction data is relatively small, and the access frequency is relatively high. Based on the previous research on different data dimensions of components, the data usage heat classification strategy is shown in Table 5:

[0114] Table 5. Measurement transaction data heat classification strategy

[0115]

[0116]

[0117] Offline measurement data is stored in the MaxCompute component of Alibaba's big data computing service. Based on the previous research on different data dimensions of the component, the data usage popularity classification strategy is shown in Table 6:

[0118] Table 6: Measurement offline data heat classification strategy

[0119]

[0120] For measurement-related business detail data, based on the collected access count indicator values, we primarily determine if the data query frequency in the past day is higher than one access per minute per day (equivalent to 1440 accesses per day) and the data is accessed every day. This data is considered hot data, and whether the number of operations in the past 30 days is greater than 300,000 is also considered warm data. The data usage heat classification strategy is shown in Table 7:

[0121] Table 7 Measurement business data heat classification strategy table

[0122]

[0123]

[0124] For measurement service shared data, based on the previous component data dimension survey, the data usage popularity classification strategy is shown in Table 8:

[0125] Table 8: Classification strategy for shared data popularity of measurement services

[0126]

[0127] Unstructured data: Unstructured data is stored in Alibaba OSS components. Unstructured data mainly includes program packages, images, audio, and text data. The data usage popularity classification strategy is shown in Table 9:

[0128] Table 9 Unstructured data heat classification strategy table

[0129]

[0130] Based on the data classification results, combined with the data's business importance and storage costs, hot data, warm data, and cold data are stored in high-performance storage media, balanced storage media, and low-cost archiving media respectively;

[0131] The importance of data services is divided into the following levels:

[0132] Important data: data related to real-time monitoring of the power grid or core business operations;

[0133] General data: data related to historical measurements or periodic statistics of equipment;

[0134] Unimportant data: log, debugging, or temporary backup data.

[0135] Dynamically configure differentiated backup strategies for different types of business data;

[0136] The hot data, warm data, and cold data are stored in high-performance storage media, balanced storage media, and low-cost archiving media respectively; wherein:

[0137] High-performance storage media include: SSD cloud disks and in-memory databases;

[0138] Balanced storage media include: distributed column database;

[0139] Low-cost archival media include object storage and tape libraries.

[0140] The differentiated backup strategy includes:

[0141] The backup strategy for structured transactional data is as follows: Transactional data is recommended to be stored in a database that facilitates service calls and real-time data interaction. In principle, data is collected and stored in the database using real-time access, and the database is deployed using a master-slave architecture. Based on the popularity of data usage and the importance of the business, the strategy is formulated as follows:

[0142] Hot data:

[0143] Important data: Perform full backup every day and retain backup files for the past 30 days;

[0144] General data: Perform full backup every two days, and retain the last seven backup files;

[0145] Non-important data: Perform a full backup every three days, and retain the last seven backup files;

[0146] Temperature data:

[0147] Important data: Perform full backup every day and retain the backup files of the last 20 times;

[0148] General data: Perform a full backup every three days, and retain the five most recent backup files;

[0149] Non-important data: Perform full backup every week and keep the most recent two backup files;

[0150] Cold data:

[0151] Important data: Perform full backups daily and retain the most recent 15 backup files;

[0152] General data: Perform full backup every week and keep the last three backup files;

[0153] Non-important data: Perform a full backup every month and retain the two most recent backup files.

[0154] The backup strategy for structured business detail data is as follows: Business detail data is recommended to be stored in a database that facilitates time-series arrangement and query, and facilitates real-time incremental data writing. In principle, data is collected and stored in the database in real-time access. Based on the popularity of data usage and the importance of the business, the strategy is formulated as follows:

[0155] Hot data:

[0156] For important data, in principle, incremental data backup is performed daily, full data is backed up every three days, and the corresponding daily incremental backup data is deleted.

[0157] For general data, in principle, incremental data backup is performed daily, full data is backed up every 5 days, and the corresponding daily incremental backup data is deleted.

[0158] For unimportant data, in principle, incremental data backup is performed daily, full data is backed up every 7 days, and the corresponding daily incremental backup data is deleted.

[0159] Temperature data:

[0160] For important data, in principle, incremental data backup is performed daily, full data is backed up every 5 days, and the corresponding daily incremental backup data is deleted.

[0161] For general data, in principle, incremental data backup is performed daily, full data is backed up every 10 days, and the corresponding daily incremental backup data is deleted.

[0162] For unimportant data, in principle, incremental data backup is performed daily, full data is backed up every 15 days, and the corresponding daily incremental backup data is deleted.

[0163] Cold data:

[0164] For important data, in principle, incremental data backup is performed daily, full data is backed up every 7 days, and the corresponding daily incremental backup data is deleted.

[0165] For general data, in principle, incremental data backup is performed daily, full data is backed up every 15 days, and the corresponding daily incremental backup data is deleted.

[0166] For unimportant data, in principle, incremental data backup is performed daily, full data is backed up every 21 days, and the corresponding daily incremental backup data is deleted.

[0167] The backup strategy for structured offline analysis data is as follows: Offline analysis data is recommended to be stored in a database that facilitates large-scale analysis and calculations and facilitates data archiving. In principle, data is collected and stored using offline access. Based on data usage and business importance, the following strategy is formulated (the update frequency of hot data is mainly determined by usage needs, and the number of years of partitioned data retained in incremental tables is mainly determined by usage needs):

[0168] Hot data:

[0169] For important data, in principle, the full update table T-1 is updated once a day, backed up once a day, and retains a 7-day data copy; the incremental update table T-1 is updated once a day, backed up once a day, and retains the incremental backup for the past 5 years. Data older than 5 years is archived and backed up regularly by year or month.

[0170] For general data, in principle, the full update table T-1 is updated once a day, backed up once a day, and retains 5 days of data copies; the incremental update table T-1 is updated once a day, backed up once a day, and retains the incremental backups for the past 3 years. Data older than 3 years is archived and backed up regularly by year or month.

[0171] For unimportant data, in principle, the full update table T-1 is updated once a day, backed up once a day, and retains 3 days of data copies; the incremental update table T-1 is updated once a day, backed up once a day, and retains the incremental backups for the past 3 years. Data older than 3 years is archived and backed up regularly by year or month.

[0172] Temperature data:

[0173] For important data, in principle, the full update table T-1 is updated once a day, backed up once a day, and retains 5 days of data copies; the incremental update table T-1 is updated once a day, backed up once a day, and retains the incremental backups for the past 5 years. Data older than 5 years is archived and backed up regularly by year or month.

[0174] For general data, in principle, full update tables are updated once every three days, backed up once every three days, and the last three copies of data are retained; incremental update tables are updated incrementally once every three days, backed up incrementally every three days, and the last three years of incremental backups are retained. Data older than three years are archived and backed up regularly by year or month.

[0175] For unimportant data, in principle, full update tables are updated once every 7 days, backed up once every 7 days, and the most recent data copy is retained; incremental update tables are updated once every 7 days, backed up once every 7 days, and the incremental backup for the past year is retained. Data older than 1 year is archived and backed up regularly by year or month.

[0176] Cold data:

[0177] For important data, in principle, full update tables are updated once every three days, backed up once every three days, and the last three copies of data are retained; incremental update tables are updated incrementally every three days, backed up incrementally every three days, and the last three years of incremental backups are retained. Data older than three years are archived and backed up regularly by year or month.

[0178] For general data, in principle, full update tables are updated once a month, backed up once a month, and the most recent data copy is retained; incremental update tables are updated once a month, backed up once a month, and the incremental backup for the past year is retained. Data older than one year is archived and backed up regularly by year or month.

[0179] For unimportant data, in principle, full update tables are manually updated in full, manually backed up as needed, and the most recent data copy is retained; incremental update tables are incrementally updated once a month, with incremental backups once a month, and the nearly 6 months of incremental backups are retained. Data older than 6 months is archived and backed up regularly by year or month.

[0180] The backup strategy for structured service shared data is as follows: It is recommended that service shared data be stored in a database that facilitates easy integration interfaces and frequent data queries. In principle, data should be collected and stored in a timely manner based on demand. Because service shared data primarily replicates the results of offline analysis, it is recommended that data be stored in different database instances based on data usage popularity, and default data backups be performed for all instances. Based on data usage popularity and business importance, the strategy is as follows:

[0181] Hot data:

[0182] For important data, in principle, data updates are guaranteed to be at least once a day, and full database backups are performed three times a week.

[0183] For general data, in principle, data updates are guaranteed to be at least once a day, and a full database backup is performed once a week.

[0184] For unimportant data, in principle, data updates are guaranteed to be at least once every three days, and a full database backup is performed once a month.

[0185] Temperature data:

[0186] For important data, in principle, data updates are guaranteed to be at least once every two days, and full database backups are performed three times a week.

[0187] For general data, in principle, data updates are guaranteed to be at least once every three days, and full database backup is performed once a week.

[0188] For unimportant data, in principle, data updates should be carried out at least once a week, and a full database backup should be performed once a month.

[0189] Cold data:

[0190] For important data, in principle, data updates are guaranteed to be at least once every three days, and full database backups are performed three times a week.

[0191] For general data, in principle, data updates should be carried out at least once a week, and a full database backup should be performed once a week.

[0192] For unimportant data, static data is guaranteed in principle, manually updated as needed, and a full database backup is performed once a month.

[0193] The backup strategy for structured service shared data is as follows: Detailed measurement business data is recommended to be stored in a database that facilitates time-series arrangement and query, and facilitates real-time incremental data writing. In principle, data is collected and stored in the database in real-time access. Based on the popularity of data usage and the importance of the business, the strategy is formulated as follows:

[0194] Hot data:

[0195] For important data, incremental backup is carried out daily in principle, and 7 days of backup data is retained. Data older than 1 year is compressed and archived regularly by year or month.

[0196] For general data, in principle, incremental backup is performed daily, and 5 days of backup data is retained. Data older than 1 year is regularly compressed and archived by year or month.

[0197] For unimportant data, in principle, incremental data backup is performed daily, and 3 days of backup data is retained. Data older than 1 year is regularly compressed and archived by year or month.

[0198] Temperature data:

[0199] For important data, incremental backup is carried out daily in principle, and 7 days of backup data is retained. Data older than 6 months is compressed and archived regularly by year or month.

[0200] For general data, in principle, incremental backup is performed daily, and 5 days of backup data is retained. Data older than 6 months is regularly compressed and archived by year or month.

[0201] For unimportant data, in principle, incremental data backup is performed daily, and 3 days of backup data is retained. Data older than 6 months is regularly compressed and archived by year or month.

[0202] Cold data:

[0203] For important data, incremental backup is carried out daily in principle, and 7 days of backup data is retained. Data older than 3 months is compressed and archived regularly by year or month.

[0204] For general data, in principle, incremental backup is performed daily, and 5 days of backup data is retained. Data older than 3 months is regularly compressed and archived by year or month.

[0205] For unimportant data, in principle, incremental data backup is performed daily, and 3 days of backup data is retained. Data older than 3 months is regularly compressed and archived by year or month.

[0206] For the storage and backup strategy of measured offline analysis data, refer to the incremental update data in structured offline analysis data.

[0207] For the storage and backup strategy of measurement service shared data, refer to the structured service shared data strategy.

[0208] The measurement and analysis result data is mainly stored in the application-side analytical database. Its storage and backup strategies refer to the structured service shared data strategy.

[0209] Unstructured data is primarily stored in databases that facilitate object storage queries, and in principle, all interactions are conducted through a unified interface. Based on the popularity of data usage and business importance, the following strategies are formulated:

[0210] Hot data: In principle, file backup is performed daily.

[0211] Warm data: In principle, file backup is performed weekly.

[0212] Cold data: In principle, file backup is performed monthly.

[0213] For unstructured measurement data, the following strategies are formulated based on data usage popularity and business importance:

[0214] Hot data: In principle, file backup is performed daily.

[0215] For important data, data older than 10 years are packaged and compressed and stored by year or month.

[0216] For general data, data older than 5 years are packaged and compressed and stored by year or month.

[0217] For unimportant data, data older than three years are packaged and compressed and stored by year or month.

[0218] Warm data: In principle, file backup is performed weekly.

[0219] For important data, data older than three years are packaged and compressed and stored by year or month.

[0220] For general data, data older than two years are packaged and compressed and stored by year or month.

[0221] For unimportant data, data older than one year is packaged and compressed and stored by year or month.

[0222] Cold data:

[0223] For important data, file backup is carried out monthly in principle, and data older than one year is packaged and compressed and stored by year or month.

[0224] For general data, file backup is carried out monthly in principle, and data older than 6 months is packaged and compressed and stored by year or month.

[0225] For unimportant data, file backup is performed manually on demand, and data older than three months is packaged and compressed and stored by year or month.

[0226] Automated scheduling of data storage, backup, and recovery is achieved through cloud platform components, and data availability is guaranteed based on a master-slave architecture and off-site multi-copy mechanism.

[0227] The cloud platform components are used to implement automated scheduling of data storage, backup, and recovery, wherein data recovery is implemented in the following ways:

[0228] Restore transactional data through hot switching of master and slave databases;

[0229] Use DataWorks to restore OSS archived data to the online database.

[0230] Trigger an off-site copy through the cloud platform console to restore unstructured data.

[0231] When applied to power data, the method includes:

[0232] Classify real-time power grid monitoring data as hot data, store it in an in-memory database, and perform daily incremental backups;

[0233] Classify equipment measurement historical data as warm data, store it in a distributed database, and back up the full data weekly;

[0234] Log data is classified as cold data, compressed, stored in OSS, and archived quarterly.

[0235] In this embodiment, the solution for implementing automated scheduling of data storage, backup, and recovery through Alibaba Cloud's proprietary cloud components is as follows:

[0236] Structured data:

[0237] Transactional data:

[0238] like Figure 2 As shown, structured transaction data is generated by the interaction between the homology maintenance module and the source business system, and is synchronously stored in the Polar-DB cloud database based on the SG-CIM unified data model. The business middle platform calls transaction data by building microservices and provides a shared interface to the outside world.

[0239] There are two main ways to back up structured transaction data. One is to configure a Polar-DB backup policy, perform scheduled daily backups through the MINI Du Kang platform, and store the backups in OSS object storage. The other is to set up a master and slave database when creating a Polar-DB database. Through automatic synchronization between the master and slave databases, the slave database serves as a hot backup for the master database data. If the master database becomes unavailable, the slave database is switched to the master state first. If the slave database also becomes unavailable, the Polar-DB database can be restored from the OSS backup file through the MINI Du Kang platform.

[0240] Business details:

[0241] like Figure 3 As shown, structured business detail data is consolidated from the Data Hub into the OTS table storage database using the BLINK stream computing component based on a unified model. The business middle platform uses microservices to call OTS business detail data and provide a shared interface.

[0242] Backing up structured business detail data can be achieved by synchronizing OTS data to OSS object storage using the DataWorks DI tool. In the event of an OTS failure or human-caused data loss, the DataWorks DI tool can also be used to restore backup data from OSS to OTS.

[0243] Offline analysis data:

[0244] like Figure 4As shown, structured offline analysis data is aggregated from the business middleware and source business systems to the ODPS source layer using the DataWorks DI tool. Depending on the source data update mode, full or incremental access can be selected. Streaming data such as measurement and business details written to the Datahub is consumed in the ODPS source layer through data synchronization by the Datahub component.

[0245] ODPS source layer offline analysis data is integrated into ODPS shared layer model tables and standard tables based on the State Grid's unified data model SG-CIM through the data development tasks of the DataWorks tool. It is then processed into analysis results through business analysis logic and finally synchronized to analytical databases such as RDS / DRDS / ADB through the DataWorks DI tool to support data service sharing.

[0246] ODPS offline analysis data is regularly backed up or archived to the centralized backup center, OSS, using the DataWorks DI tool. When archived data needs to be queried temporarily, it can be referenced in ODPS as a table. In the event of an ODPS failure or human-intentioned data loss, the DataWorks DI tool can be used to restore the data to the corresponding ODPS table and partition after the ODPS service is restored.

[0247] Services Shared Data:

[0248] like Figure 5 As shown in the figure, structured service shared data is stored in analytical databases such as RDS / DRDS / ADB. The analysis results are accessed through the DataWorks DI tool, and based on the DataWorks data service encapsulation, business applications and business middle platforms are provided with API interfaces.

[0249] The RDS database uses the MINI Dukang operation and maintenance platform to create a full-database backup strategy and perform regular full-database backups. If RDS experiences data loss due to database failure or human error, the MINI Dukang operation and maintenance platform can be used to restore the entire database. Because analysis results are derived from the ODPS analysis layer, if RDS experiences data loss, the latest data can be restored using the DataWorks DI tool.

[0250] Measurement data:

[0251] Business details:

[0252] It is recommended that detailed measurement business data be stored in a database that facilitates time-series arrangement and query, and facilitates real-time incremental data writing. In principle, data is collected and stored in the database in real-time access mode. Based on the popularity of data use and the importance of the business, the following strategies are formulated:

[0253] Hot data:

[0254] For important data, incremental backup is carried out daily in principle, and 7 days of backup data is retained. Data older than 1 year is compressed and archived regularly by year or month.

[0255] For general data, in principle, incremental backup is performed daily, and 5 days of backup data is retained. Data older than 1 year is regularly compressed and archived by year or month.

[0256] For unimportant data, incremental backup is carried out daily in principle, and 3 days of backup data is retained. Data older than 1 year is regularly compressed and archived by year or month.

[0257] Temperature data:

[0258] For important data, incremental backup is carried out daily in principle, and 7 days of backup data is retained. Data older than 6 months is compressed and archived regularly by year or month.

[0259] For general data, in principle, incremental backup is performed daily, and 5 days of backup data is retained. Data older than 6 months is regularly compressed and archived by year or month.

[0260] For unimportant data, incremental backup is carried out daily in principle, and 3 days of backup data is retained. Data older than 6 months is regularly compressed and archived by year or month.

[0261] Cold data:

[0262] For important data, in principle, incremental data backup is performed daily, and 7 days of backup data is retained. Data older than 3 months is regularly compressed and archived by year or month.

[0263] For general data, in principle, incremental backup is performed daily, and 5 days of backup data is retained. Data older than 3 months is regularly compressed and archived by year or month.

[0264] For unimportant data, incremental backup is carried out daily in principle, and 3 days of backup data is retained. Data older than 3 months is regularly compressed and archived by year or month.

[0265] Offline analysis data:

[0266] The storage, backup, and recovery solutions for measurement offline analysis data are the same as those for structured offline analysis data.

[0267] Services Shared Data:

[0268] The storage, backup, and recovery solutions for measurement service shared data are the same as those for structured service shared data.

[0269] Analysis result data:

[0270] like Figure 6As shown, the measurement and analysis result data is obtained by the application calling the enterprise middle platform data service interface, and after being processed and analyzed by program logic, it is deposited in the application-side analytical database such as RDS / DRDS / ADB for business application calls.

[0271] The RDS database uses the MINI Dukang operation and maintenance platform to create a full-database backup strategy and perform regular full-database backups. If RDS experiences data loss due to database failure or human error, the MINI Dukang operation and maintenance platform can be used to restore the entire database. Because analysis results are derived from the ODPS analysis layer, if RDS experiences data loss, the latest data can be restored using the DataWorks DI tool.

[0272] Unstructured data:

[0273] like Figure 7 As shown in the figure, the unstructured platform is integrated with the object storage component OSS based on the unstructured data management component of the big data platform, providing a unified standard interface to the outside world to enable uploading and downloading of unstructured data in the OSS component.

[0274] The OSS object storage component is backed up through the proprietary cloud backup console ASR-BR, which backs up unstructured data to another cloud in a different location within the same city. At the same time, the data is copied to the off-cloud miniOSS object storage component in a different location within the cloud. File recovery is also achieved through the recovery function of the backup console ASR-BR.

[0275] Unstructured measurement data:

[0276] The storage, backup, and recovery solutions for unstructured measurement data are the same as those for unstructured data.

[0277] Example 2:

[0278] The data classification module divides business data into hot data, warm data, and cold data based on preset access frequency, time window, and data volume indicators;

[0279] The storage module stores hot data, warm data, and cold data in high-performance storage media, balanced storage media, and low-cost archiving media, respectively, based on data classification results, combined with data business importance and storage costs.

[0280] The backup module dynamically configures differentiated backup strategies for different types of business data;

[0281] The data recovery and call module realizes the automatic scheduling of data storage, backup and recovery through cloud platform components, and ensures data availability based on the master-slave architecture and multi-replica mechanism in different locations.

[0282] In the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can represent: a, b, c, a and b, a and c, b and c or a and b and c, where a, b, c can be single or multiple.

[0283] Those skilled in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented using a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0284] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0285] In the several embodiments provided in this application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of this application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program code.

[0286] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention's description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for storing and backing up power data in a data center based on usage popularity, characterized in that: The following steps are involved: Based on preset access frequency, time window and data volume indicators, business data is divided into hot data, warm data and cold data; Based on the data classification results, combined with the data's business importance and storage costs, hot data, warm data, and cold data are stored in high-performance storage media, balanced storage media, and low-cost archiving media respectively; Dynamically configure differentiated backup strategies for different types of business data; Automated scheduling of data storage, backup, and recovery is achieved through cloud platform components, and data availability is guaranteed based on a master-slave architecture and off-site multi-copy mechanism.

2. The method for storing and backing up power data in a data center based on usage popularity according to claim 1, characterized in that: Based on the preset access frequency, time window and data volume indicators, business data is divided into hot data, warm data and cold data, wherein the classification rules for structured transaction data are: Hot data: the number of visits within 7 days is greater than 100,000 and the data volume is greater than 0; Warm data: Meets any of the following conditions: The number of visits within 7 days is ≤ 100,000 and the number of visits within 30 days is > 0; The data volume is greater than 0 and the number of visits within 30 days is greater than 0; Cold data: Number of visits within 30 days = 0 or data volume = 0.

3. The method for storing and backing up power data in a data center based on usage popularity according to claim 1, characterized in that: Based on the preset access frequency, time window and data volume indicators, business data is divided into hot data, warm data and cold data, wherein the classification rules for unstructured data are: Hot data: the number of calls in the past day is greater than 0 and the number of calls is less than the number of calls in the past 7 days; Warm data: Meets any of the following conditions: Number of calls in the past day = number of calls in the past 7 days; The number of calls in the past day = 0 and the number of calls in the past 7 days > 0; Cold data: Number of calls in the past 7 days = 0.

4. The method for storing and backing up power data in a data center based on usage popularity according to claim 1, characterized in that: The importance of data services is divided into the following levels: Important data: data related to real-time monitoring of the power grid or core business operations; General data: data related to historical measurements or periodic statistics of equipment; Unimportant data: log, debugging, or temporary backup data.

5. The method for storing and backing up power data in a data center based on usage popularity according to claim 1, characterized in that: The hot data, warm data, and cold data are stored in high-performance storage media, balanced storage media, and low-cost archiving media respectively; wherein: High-performance storage media include: SSD cloud disks and in-memory databases; Balanced storage media include: distributed column database; Low-cost archival media include object storage and tape libraries.

6. The method for storing and backing up power data in a data center based on usage popularity according to claim 1, characterized in that: The differentiated backup strategy includes: for structured transactional data backup strategy: Hot data: Important data: Perform full backup every day and retain backup files for the past 30 days; General data: Perform full backup every two days, and retain the last seven backup files; Non-important data: Perform a full backup every three days, and retain the last seven backup files; Temperature data: Important data: Perform full backup every day and retain the backup files of the last 20 times; General data: Perform a full backup every three days, and retain the five most recent backup files; Non-important data: Perform full backup every week and keep the most recent two backup files; Cold data: Important data: Perform full backups daily and retain the most recent 15 backup files; General data: Perform full backup every week and keep the last three backup files; Non-important data: Perform a full backup every month and retain the two most recent backup files.

7. The method for storing and backing up power data in a data center based on usage popularity according to claim 1, characterized in that: The cloud platform components are used to implement automated scheduling of data storage, backup, and recovery, wherein data recovery is implemented in the following ways: Restore transactional data through hot switching of master and slave databases; Use DataWorks to restore archived data from the Object Storage Service (OSS) to the online database. Trigger an off-site copy through the cloud platform console to restore unstructured data.

8. The method for storing and backing up power data in a data center based on usage popularity according to claim 1, characterized in that: The method comprises the steps of dynamic optimization: Predict data popularity based on historical access frequency and migrate predicted hot data to high-performance storage in advance. When the storage resource usage exceeds 80%, the cold data compression and archiving process is automatically triggered.

9. The method for storing and backing up power data in a data center based on usage popularity according to claim 1, characterized in that: When applied to power data, the method includes: Classify real-time power grid monitoring data as hot data, store it in an in-memory database, and perform daily incremental backups; Classify equipment measurement historical data as warm data, store it in a distributed database, and back up the full data weekly; Log data is classified as cold data, compressed, stored in OSS, and archived quarterly.

10. A data center power data storage and backup system based on usage popularity, characterized in that: include: The data classification module divides business data into hot data, warm data, and cold data based on preset access frequency, time window, and data volume indicators; The storage module stores hot data, warm data, and cold data in high-performance storage media, balanced storage media, and low-cost archiving media, respectively, based on data classification results, combined with data business importance and storage costs. The backup module dynamically configures differentiated backup strategies for different types of business data; The data recovery and call module realizes the automatic scheduling of data storage, backup and recovery through cloud platform components, and ensures data availability based on the master-slave architecture and multi-replica mechanism in different locations.

Images