Electric power cold and hot data storage system and method

Abstract

The invention discloses an electric power cold and hot data storage system. The system comprises a data access frequency acquisition module for acquiring electric power data access frequencies of a hot database and cold data; the power data migration module is used for migrating the power data to the cold database for storage if the access frequency of the power data in the hot database is smaller than the preset data access frequency, and migrating the power data to the hot database for storage if the access frequency of the power data in the cold database is larger than or equal to the preset data access frequency; the preset data access frequency adjusting module is used for predicting the power data access frequency trend through a random forest and adjusting the preset data access frequency in the power data migration module, and the data storage optimizing module is used for distributing storage resources by monitoring the use information stored in the hot database and the cold database in real time and combining the access frequency of the power data. By dynamically adjusting the migration threshold value of the cold and hot data, the data storage strategy is ensured to be always matched with the actual access mode.

Description

Technical Field

[0001] The present invention relates to the technical field of power data storage, and in particular, to a power cold and hot data storage system and method. Background Art

[0002] In modern power systems, the management and storage of power data face huge challenges. As the power system develops towards intelligence and informatization, a large amount of real-time monitoring data, transaction data, and historical data are continuously generated and stored. These data include load forecasting data, real-time transaction data, historical operation data, and power grid equipment status information, covering all aspects of the operation of the power system. However, currently, many storage systems lack the intelligent cold and hot data management ability and cannot dynamically adjust the storage location of data according to the change of data access frequency. Manual configuration of storage strategies and fixed migration thresholds are not only inefficient but also easily lead to waste of storage resources and degradation of system performance. Summary of the Invention

[0003] Aiming at the deficiencies of the prior art, the present invention provides a power cold and hot data storage system and method.

[0004] A power cold and hot data storage system provided by the present invention includes the following:

[0005] Data access frequency acquisition module (11): used to acquire the power data access frequencies of the hot database and the cold data;

[0006] Power data migration module (12): used to compare the power data access frequency with a preset data access frequency. If the access frequency of the power data in the hot database is less than the preset data access frequency, the power data is migrated to the cold database for storage. If the access frequency of the power data in the cold database is greater than or equal to the preset data access frequency, the power data is migrated to the hot database for storage;

[0007] Preset data access frequency adjustment module (13): used to predict the trend of the power data access frequency through random forest and dynamically adjust the preset data access frequency in the power data migration module (12);

[0008] Data storage optimization module (14): automatically allocates storage resources by real-time monitoring the usage information of the hot database and the cold database storage and combining with the access frequency of the power data.

[0009] Preferably, the power data migration module (12) includes,

[0010] Access frequency judgment unit (121): used to judge according to the preset data access frequency. If the access frequency of the power data in the hot database is less than the preset data access frequency, the power data is migrated to the cold database for storage; if the access frequency of the power data in the cold database is greater than the preset data access frequency, the power data is migrated to the hot database for storage;

[0011] Data migration trigger unit (122): used to trigger data migration operations according to the judgment results of the access frequency judgment unit; when the access frequency of a certain power data in the hot database is less than the preset data access frequency, trigger the operation of migrating the power data to the cold database; when the access frequency of a certain power data in the cold database is greater than or equal to the preset data access frequency, trigger the operation of migrating the power data to the hot database;

[0012] Data migration execution unit (123): used to execute data migration operations, migrating power data from the hot database to the cold database, or from the cold database to the hot database;

[0013] Migration status monitoring unit (124): used to monitor the status information of the data migration process in real time, including migration progress, migration data volume, and abnormal information that appears during the migration process, and handle the abnormal information.

[0014] Preferably, the migration status monitoring unit (124) includes,

[0015] Migration progress monitoring subunit (1241): used to track the progress of data migration in real time, record the amount of data that has been migrated and the amount of data remaining to be migrated, and display the migration progress in the form of a percentage or a progress bar;

[0016] Migration data volume statistics subunit (1242): used to count the total amount of data involved in the migration process, the amount of data that has been migrated, and the migration rate per unit time to obtain the migration data volume;

[0017] Abnormal detection subunit (1243): used to detect abnormal information that appears during the migration process in real time, including network interruption, storage device failure, and data verification failure.

[0018] Abnormal handling subunit (1244): used to execute corresponding handling measures according to the detected abnormal information, including retrying the migration operation, switching the migration path, and notifying the administrator to intervene;

[0019] Migration log recording subunit (1245): used to record the detailed log information during the migration process, including the migration start time, end time, migration data volume, abnormal information, and handling results;

[0020] Migration status visualization subunit (1246): Used to visually display the migration progress, data volume statistics, and exception information in the form of charts or dashboards.

[0021] Preferably, the preset data access frequency adjustment module (13) includes

[0022] Feature extraction unit (131): Used to extract features from the power data access frequency monitored by the data access frequency acquisition module (11), including time series features, periodic features, and access pattern features;

[0023] Model prediction unit (132): Used to input the extracted features into the trained power data access frequency prediction model to predict the power data access frequency trend within a preset time range;

[0024] Frequency adjustment unit (133): Used to dynamically adjust the preset data access frequency in the power data migration module (12) according to the predicted power data frequency access frequency trend.

[0025] Preferably, the data storage optimization module (14) includes

[0026] Storage usage monitoring unit (141): Used to periodically collect the storage usage information of the hot database and the cold database, including storage space, CPU utilization rate, and memory utilization rate;

[0027] Storage resource allocation unit (142): Used to formulate a storage allocation strategy according to the storage usage information and data access frequency of the hot database and the cold database, including storage expansion and data compression;

[0028] Execution unit (143): Used to execute the storage allocation strategy and automatically complete the allocation of storage resources.

[0029] Optimization effect evaluation unit (144): Used to evaluate the execution effect information of the storage optimization strategy in real time, including storage resource utilization rate, response time, and data access efficiency, and send the execution effect information to the storage resource allocation unit (142) to form a closed-loop optimization mechanism.

[0030] A method for storing electric power cold and hot data includes the following steps: collecting the access frequencies of electric power data in the hot database and the cold data; comparing the access frequencies of electric power data with a preset data access frequency. If the access frequency of the electric power data in the hot database is less than the preset data access frequency, the electric power data is migrated to the cold database for storage. If the access frequency of the electric power data in the cold database is greater than or equal to the preset data access frequency, the electric power data is migrated to the hot database for storage; predicting the trend of the access frequency of electric power data through a random forest to dynamically adjust the preset data access frequency; automatically allocating storage resources by monitoring the usage information of the hot database and the cold database in real time and combining the access frequencies of the electric power data.

[0031] The present invention discloses an electric power cold and hot data storage system, and the beneficial effects thereof are as follows:

[0032] By monitoring the storage usage of the hot database and the cold database in real time, the efficient utilization of storage resources is ensured. By dynamically adjusting the migration threshold of cold and hot data, it is ensured that the data storage strategy always matches the actual access pattern, avoiding performance degradation caused by unreasonable static threshold settings. Combining the data access frequency and the storage resource usage situation, storage resources are automatically allocated, reducing manual intervention and improving management efficiency. Through dynamic adjustment and prediction functions, it can adapt to these dynamic changes and ensure that the system is always in an optimal state. By predicting the access frequency of data through the random forest algorithm, the storage strategy of cold and hot data is adjusted in advance to avoid performance fluctuations caused by changes in the access pattern. Through real-time monitoring and dynamic adjustment, system performance degradation or failures caused by insufficient or overloaded storage resources are avoided. Through intelligent algorithms and automated modules, decisions are automatically made according to the data access pattern and the storage resource usage situation, reducing the need for manual intervention and lowering the operation and maintenance costs and workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0034] Figure 1 It is an architecture diagram of an electric power cold and hot data storage system provided by the present invention;

[0035] Figure 2 It is a structural diagram of the electric power data migration module unit in an electric power cold and hot data storage system provided by the present invention;

[0036] Figure 3It is the subunit structure diagram of the migration status monitoring unit in the power data migration module provided by the present invention;

[0037] Figure 4 It is the unit structure diagram of the preset data access frequency adjustment module provided by the present invention;

[0038] Figure 5 It is the unit structure diagram of the data storage optimization module provided by the present invention; Detailed implementation manners

[0039] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0040] To better understand the above technical solutions, the above technical solutions will be described in detail below in conjunction with the accompanying drawings of the specification and specific implementation manners.

[0041] Refer to Figure 1 , a power cold and hot data storage system provided by the present invention includes the following:

[0042] Data access frequency acquisition module 11: used to acquire the power data access frequencies of the hot database and cold data; specifically, acquire the access logs of the power data in the hot database and cold database, including the number of accesses and the access time intervals, and calculate the power data access frequencies of each piece of power data according to the access logs.

[0043] Among them, the hot database uses high-performance, high-throughput and low-latency storage devices, such as RAM caches, multi-core CPUs and network interfaces configured with low latency and high bandwidth; the cold database uses large-capacity, low-cost and medium-performance storage devices, such as HDDs, tape storage and hierarchical storage devices;

[0044] Power data migration module 12: used to compare the power data access frequency with the preset data access frequency. If the access frequency of the power data in the hot database is less than the preset data access frequency, the power data is migrated to the cold database for storage. If the access frequency of the power data in the cold database is greater than the preset data access frequency, the power data is migrated to the hot database for storage;

[0045] Preset data access frequency adjustment module 13: used to predict the trend of the power data access frequency through random forest and dynamically adjust the preset data access frequency in the power data migration module 12;

[0046] Data storage optimization module 14: Automatically allocate storage resources by monitoring the usage information of the hot database and cold database in real time and combining with the access frequency of power data.

[0047] A power cold and hot data storage system provided by the present invention can store frequently accessed data in high-performance devices such as RAM caches and multi-core CPUs, which can significantly reduce data access latency and improve data processing speed, thereby enhancing the responsiveness and overall performance of the system. Infrequently accessed data is migrated to large-capacity and low-cost storage devices such as HDDs or tape storage. This can save storage costs without affecting critical operations, while ensuring sufficient space for rapid access to hot data. Use the random forest algorithm to predict data access trends and dynamically adjust the threshold for data migration. This enables the system to adapt to different workload patterns, utilize existing resources more effectively, and potentially reduce unnecessary data migrations; by monitoring storage usage and adjusting resource allocation as needed. This self-adjusting ability helps maintain the optimal storage environment and support changes in business requirements.

[0048] Reference Figure 2 , in the preferred embodiment, the power data migration module 12 includes,

[0049] Access frequency judgment unit 121: Used to judge according to the preset data access frequency. If the access frequency of the power data in the hot database is less than the preset data access frequency, the power data is migrated to the cold database for storage. If the access frequency of the power data in the cold database is greater than the preset data access frequency, the power data is migrated to the hot database for storage;

[0050] Data migration trigger unit 122: Used to trigger data migration operations according to the judgment results of the access frequency judgment unit; when the access frequency of a certain power data in the hot database is less than the preset data access frequency, trigger the operation of migrating the power data to the cold database; when the access frequency of a certain power data in the cold database is greater than the preset data access frequency, trigger the operation of migrating the power data to the hot database;

[0051] Data migration execution unit 123: Used to execute data migration operations, migrating power data from the hot database to the cold database, or from the cold database to the hot database;

[0052] Migration status monitoring unit 124: Used to monitor the status information of the data migration process in real time, including migration progress, migration data volume, and abnormal information that occurs during the migration process, and handle abnormal information.

[0053] Reference Figure 3 , in the preferred embodiment, the migration status monitoring unit 124 includes,

[0054] Migration progress monitoring subunit 1241: It is used to track the progress of data migration in real time, record the amount of data that has been migrated and the remaining amount of data to be migrated, and display the migration progress in the form of a percentage or a progress bar;

[0055] Migrated data volume statistics subunit 1242: It is used to count the total amount of data involved in the migration process, the amount of data that has been migrated, and the migration rate per unit time, and obtain the migrated data volume;

[0056] Abnormality detection subunit 1243: It is used to detect abnormal information that appears during the migration process in real time, including network interruption, storage device failure, and data verification failure.

[0057] Abnormality handling subunit 1244: It is used to execute corresponding handling measures according to the detected abnormal information, including retrying the migration operation, switching the migration path, and notifying the administrator to intervene;

[0058] Migration log recording subunit 1245: It is used to record detailed log information during the migration process, including the migration start time, end time, migrated data volume, abnormal events, and handling results;

[0059] Migration status visualization subunit 1246: It is used to visually display the migration progress, data volume statistics, and abnormal information in the form of charts or dashboards.

[0060] Reference Figure 4 , in a preferred embodiment, the preset data access frequency adjustment module 13 includes,

[0061] Feature extraction unit 131: It is used to extract features from the power data access frequency monitored by the data access frequency acquisition module 11, including time series features, periodic features, and access pattern features;

[0062] Model prediction unit 132: It is used to input the extracted features into the trained power data access frequency prediction model to predict the trend of the power data access frequency within a preset time range; among them, historical power data access frequency data is collected, including timestamps, access frequencies, data types, etc., and a power data access frequency prediction model is constructed using random forests according to the historical power data access frequency data.

[0063] Frequency adjustment unit 133: It is used to dynamically adjust the preset data access frequency in the power data migration module 12 according to the predicted trend of the power data access frequency.

[0064] Specifically, when the prediction result indicates that the overall data access frequency decreases, the preset data access frequency is appropriately reduced to reduce the storage pressure on the hot database; when the prediction result indicates that the overall data access frequency increases, the preset data access frequency is appropriately increased to ensure that high-frequency data can be migrated to the hot database in a timely manner.

[0065] Reference Figure 5 , in the preferred embodiment, the data storage optimization module 14 includes

[0066] Storage usage monitoring unit 141: used to periodically collect the storage usage information of the hot database and the cold database, including storage space, CPU utilization rate, and memory utilization rate; specifically, identify the peaks and valleys of resource usage based on the collected storage usage information;

[0067] Storage resource allocation unit 142: used to formulate a storage allocation strategy according to the storage usage information and data access frequency of the hot database and the cold database, including storage expansion and data compression;

[0068] Execution unit 143: used to execute the storage allocation strategy and automatically complete the allocation of storage resources.

[0069] Specifically, if it is identified that the resource usage period is a peak period, or it is detected that some log data in the cold database has not been accessed for a long time and occupies a large amount of space, use lightweight compression algorithms such as Snappy or Zlib to compress the data in the cold database; if it is identified that the resource usage period is a peak period or it is detected that the storage space utilization rate of the hot database exceeds the preset threshold, automatically add new storage nodes or partitions, such as adding high-performance storage devices SSD to process hot data, and enhance the load balancing ability to ensure that requests are evenly distributed to all servers; if it is identified that the resource usage period is a trough period, automatically reduce unnecessary resources of the cold database and the hot database, such as shutting down unused servers or virtual machines, reducing storage allocation, and migrating infrequently accessed data to lower-cost HDD or tape storage.

[0070] Optimization effect evaluation unit 144: used to evaluate the execution effect information of the storage optimization strategy in real time, including storage resource utilization rate, response time, and data access efficiency, and send the execution effect information to the storage resource allocation unit 142 to form a closed-loop optimization mechanism.

[0071] Specifically, calculate resource utilization rates such as storage space usage percentage, read and write I / O load, analyze the peak and trough distribution of storage resources, and identify whether there is resource waste or shortage; calculate the average response time, P95, and P99 latency of requests, compare the response time changes before and after optimization, and evaluate the strategy effect; count the throughput of data reading and writing, and check whether the migration of cold data and hot data causes performance bottlenecks

[0072] A storage method for electric power hot and cold data includes the following steps: collecting the access frequencies of electric power data in the hot database and the cold data; comparing the access frequencies of electric power data with the preset data access frequency. If the access frequency of the electric power data in the hot database is less than the preset data access frequency, the electric power data is migrated to the cold database for storage. If the access frequency of the electric power data in the cold database is greater than the preset data access frequency, the electric power data is migrated to the hot database for storage; predicting the trend of the access frequency of electric power data through random forest and dynamically adjusting the preset data access frequency; automatically allocating storage resources by monitoring the usage information of the hot database and the cold database in real time and combining with the access frequency of the electric power data.

Claims

1. A power-based cold and heat data storage system, characterized in that, The following are included: Data access frequency acquisition module (11): used to acquire the power data access frequencies of the hot database and cold data; Power data migration module (12): used to compare the power data access frequency with a preset data access frequency. If the access frequency of the power data in the hot database is less than the preset data access frequency, the power data is migrated to the cold database for storage. If the access frequency of the power data in the cold database is greater than or equal to the preset data access frequency, the power data is migrated to the hot database for storage; Preset data access frequency adjustment module (13): used to predict the trend of the power data access frequency through random forest and dynamically adjust the preset data access frequency in the power data migration module (12); Data storage optimization module (14): by monitoring the usage information of the hot database and cold database storage in real time and combining with the access frequency of the power data, automatically allocate storage resources.

2. The power cold and heat data storage system according to claim 1, characterized in that, The power data migration module (12) includes: Access frequency judgment unit (121): used to judge according to the preset data access frequency. If the access frequency of the power data in the hot database is less than the preset data access frequency, the power data is migrated to the cold database for storage. If the access frequency of the power data in the cold database is greater than the preset data access frequency, the power data is migrated to the hot database for storage; Data migration trigger unit (122): used to trigger the data migration operation according to the judgment result of the access frequency judgment unit; When the access frequency of a certain power data in the hot database is less than the preset data access frequency, trigger the operation of migrating the power data to the cold database; when the access frequency of a certain power data in the cold database is greater than or equal to the preset data access frequency, trigger the operation of migrating the power data to the hot database; Data migration execution unit (123): used to execute the data migration operation, migrating the power data from the hot database to the cold database, or from the cold database to the hot database; Migration status monitoring unit (124): used to monitor the status information of the data migration process in real time, including migration progress, amount of migrated data, and abnormal information that appears during the migration process, and handle the abnormal information.

3. The power cold and heat data storage system according to claim 2, characterized in that, The migration status monitoring unit (124) includes: Migration progress monitoring subunit (1241): used to track the progress of the data migration in real time, record the amount of data that has been migrated and the amount of data remaining to be migrated, and display the migration progress in the form of a percentage or progress bar; Migrated data volume statistics subunit (1242): used to count the total amount of data involved in the migration process, the amount of data that has been migrated, and the migration rate per unit time to obtain the migrated data volume; Abnormality detection subunit (1243): used to detect abnormal information that appears during the migration process in real time, including network interruption, storage device failure, and data verification failure. Abnormality handling subunit (1244): used to execute corresponding handling measures according to the detected abnormal information, including retrying the migration operation, switching the migration path, and notifying the administrator to intervene; Migration Logging Sub-unit (1245): Used to record detailed log information during the migration process, including the start time, end time, amount of migrated data, exception information, and processing results; Migration Status Visualization Sub-unit (1246): Used to visually display the migration progress, data volume statistics, and exception information in the form of charts or dashboards.

4. The power cold and heat data storage system according to claim 2, characterized in that, The preset data access frequency adjustment module (13) includes: Feature Extraction Unit (131): Used to extract features from the power data access frequencies monitored by the data access frequency acquisition module (11), including time series features, periodic features, and access pattern features; Model Prediction Unit (132): Used to input the extracted features into a trained power data access frequency prediction model to predict the trend of power data access frequencies within a preset time range; Frequency Adjustment Unit (133): Used to dynamically adjust the preset data access frequency in the power data migration module (12) according to the predicted trend of power data access frequencies.

5. A power cold and heat data storage system according to claim 1, characterized in that The data storage optimization module (14) includes: Storage Usage Monitoring Unit (141): Used to periodically collect storage usage information of the hot database and cold database, including storage space, CPU utilization, and memory utilization; Storage Resource Allocation Unit (142): Used to formulate a storage allocation strategy according to the storage usage information and data access frequencies of the hot database and cold database, including storage expansion and data compression; Execution Unit (143): Used to execute the storage allocation strategy and automatically complete the allocation of storage resources. Optimization Effect Evaluation Unit (144): Used to evaluate the execution effect information of the storage optimization strategy in real time, including storage resource utilization, response time, and data access efficiency, and send the execution effect information to the storage resource allocation unit (142) to form a closed-loop optimization mechanism.

6. A storage method for electric power cooling and heating data using an electric power cooling and heating data storage system as described in any one of claims 1-5, characterized in that, It includes the following: Collect the power data access frequencies of the hot database and cold database; compare the power data access frequencies with the preset data access frequency. If the access frequency of the power data in the hot database is less than the preset data access frequency, migrate the power data to the cold database for storage. If the access frequency of the power data in the cold database is greater than or equal to the preset data access frequency, migrate the power data to the hot database for storage; Predict the trend of power data access frequencies through random forest and dynamically adjust the preset data access frequency; Automatically allocate storage resources by monitoring the usage information of the hot database and cold database in real time and combining with the access frequencies of the power data.

Images