Large power grid supply and demand adjusting system based on multi-source storage collaboration

Abstract

The invention provides a large power grid supply and demand regulation system based on multi-source storage collaboration. The system integrates a real-time database storage module, a non-real-time database storage module, a temporary storage module, a data monitoring module, a data classification module and a subsequent optimization module. The system temporarily stores data through the temporary storage module and supports access, and the data monitoring module monitors and marks data access conditions in real time. And the data classification module automatically classifies the data according to the access frequency and imports the data into a real-time or non-real-time database, so that the classification effect of the data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing for large power grid supply and demand regulation based on multi-source-storage collaboration, and in particular to a large power grid supply and demand regulation system based on multi-source-storage collaboration. Background Art

[0002] With the expansion of power grid scale and the diversification of energy structure, the supply and demand regulation of large power grids faces unprecedented challenges. In order to meet these challenges, multi-source storage synergy has become an effective solution. Multi-source storage synergy can integrate different types of energy storage resources, realize the optimal allocation and efficient utilization of resources, and thus improve the stability and flexibility of the power grid.

[0003] The supply and demand regulation of large power grids with multi-source storage coordination refers to the dynamic regulation of the supply and demand relationship of the power grid by coordinating multiple energy storage resources. This regulation method can flexibly adjust the energy storage and release strategies according to the energy demand and supply conditions at different times and locations, thereby ensuring the stable operation and efficient utilization of the power grid. The supply and demand regulation of large power grids with multi-source storage coordination has the advantages of fast response speed, wide regulation range, and high resource utilization, which is of great significance for improving the reliability and economy of the power grid.

[0004] However, existing technologies often rely on empirical judgment when dividing data, which lacks scientificity and accuracy. This division method not only fails to meet the actual use needs of different enterprises, but also may lead to waste of resources and reduced efficiency.

[0005] To this end, we propose a large power grid supply and demand regulation system based on multi-source storage collaboration Summary of the invention

[0006] The purpose of the present invention is to provide a large power grid supply and demand regulation system based on multi-source storage collaboration to solve the above-mentioned deficiencies in the prior art.

[0007] In order to achieve the above object, the present invention provides the following technical solution: comprising a real-time database storage module and a non-real-time database storage module, characterized in that the large power grid supply and demand regulation system based on multi-source storage collaboration also includes:

[0008] A temporary storage module, when the large power grid supply and demand regulation system based on multi-source storage collaboration stores various types of data, the data is temporarily stored through the temporary storage module, and the temporary storage module supports user access and search;

[0009] A data monitoring module, which is connected to the temporary storage module, the real-time database storage module and the non-real-time database storage module through an interface. When a user accesses and searches for data in the temporary storage module, the data monitoring module is used to monitor and mark the accessed data, and record the number of accesses and access time within a time interval;

[0010] A data classification module, which is used to import data into a real-time database storage module or a non-real-time database storage module according to the number of accesses to the data in the temporary database within a time interval, and when the number of times a type of data is classified into the real-time database storage module / non-real-time database storage module exceeds a threshold, the subsequent storage of the type of data will be directly stored in the real-time database storage module / non-real-time database storage module instead of the temporary storage module;

[0011] The subsequent optimization module is used to monitor the number of accesses to data in the real-time database and the non-real-time database within a time interval, and according to the number of data accesses within a time interval, transfer the data in the real-time database to the non-real-time database, compress the data in the non-real-time database and mark it as long-term data.

[0012] As a further description of the above technical solution: the temporary storage module is constructed using a distributed file system, wherein the distributed file system includes Hadoop HDF and Ceph to ensure high availability and scalability of data.

[0013] As a further description of the above technical solution: the data monitoring module is built using the stream processing framework of Apache Flink to achieve real-time monitoring and marking of data in the temporary storage module, and the data monitoring module also integrates log collection and analysis tools to record and analyze data access logs, wherein the log collection and analysis tools include but are not limited to Logstash and ELK Stack.

[0014] As a further description of the above technical solution: the data classification module uses a machine learning algorithm to classify the data so as to automatically import the data into a real-time database storage module or a non-real-time database storage module according to the number of accesses.

[0015] As a further description of the above technical solution: the subsequent optimization module uses a data migration tool to realize data transmission between the real-time database and the non-real-time database, wherein data compression uses Hadoop's MapReduce for batch processing to improve data compression efficiency and quality, and the data migration tool includes but is not limited to the Apache Airflow data migration tool.

[0016] As a further description of the above technical solution: the specific steps of monitoring data by the data monitoring module are:

[0017] Step A: The data monitoring module captures data access events in the temporary storage module through the data source interface. These events include but are not limited to data reading, writing, querying and other operations;

[0018] Step B: After capturing data access events, the data monitoring module uses the built-in real-time monitoring engine (based on event-driven architecture) to process these events in real time. For each access, the data monitoring module will record the access time and number of accesses.

[0019] 9. As a further description of the above technical solution: the method for classifying data by the data classification module is divided into the following steps:

[0020] Step C: The data classification module first obtains the access statistics of various types of data within a period of time from the data monitoring module, including the number of accesses and access time distribution.

[0021] Step D: Based on the data access count information, the data classification module uses a preset classification algorithm (threshold classification method based on access frequency) to classify the data. If the access count of a certain type of data exceeds the set threshold, it is regarded as hot data and is decided to be imported into the real-time database storage module; otherwise, it is regarded as non-hot data and imported into the non-real-time database storage module. After the classification decision is completed, the data classification module will perform the corresponding data migration operation to migrate the data from the temporary storage module to the corresponding storage module.

[0022] As a further description of the above technical solution: the specific steps of the subsequent optimization module optimizing the real-time data storage module and the non-real-time data storage module are:

[0023] Step E: The subsequent optimization module first analyzes the data access patterns of the real-time database and the non-real-time database, including but not limited to counting the number of accesses and access time intervals of each data;

[0024] Step F: Based on the analysis results, the module will migrate the data in the real-time database that is no longer active (data with a sudden decrease in the number of accesses) to the non-real-time database; at the same time, for the data in the non-real-time database that has been stored for a long time and has a very low access frequency, the module will compress it and mark it as long-term data.

[0025] In the above technical solution, the large power grid supply and demand regulation system based on multi-source storage coordination provided by the present invention has the following beneficial effects:

[0026] Intelligent data division meets diverse needs: The system can automatically divide data based on actual conditions without relying on experience. Through the collaborative work of the data monitoring module and the data classification module, the system can accurately count the access frequency of various types of data, and automatically import the data into a real-time database or a non-real-time database based on these frequencies. This intelligent data division method meets the different needs of different companies for data access speed, storage cost, and data management complexity, and improves the flexibility and efficiency of data processing.

[0027] Optimize storage resource utilization: The subsequent optimization module monitors data access patterns and migrates data that is no longer active in the real-time database to the non-real-time database, effectively freeing up storage space in the real-time database and ensuring that data with high access frequency can be responded to quickly. At the same time, for data in the non-real-time database that has been stored for a long time and has a very low access frequency, the system compresses it and marks it as long-term data, further saving storage space and improving the utilization efficiency of storage resources.

[0028] Improve data access speed and compression efficiency: The system uses high-performance real-time database and non-real-time database, combined with efficient data transmission protocol and data compression technology, which significantly improves data access speed and compression efficiency. The real-time database can ensure fast access and analysis of high-real-time data, while the non-real-time database provides large data storage and complex query functions. At the same time, Hadoop's MapReduce framework is used for data compression, achieving high efficiency and quality assurance of batch processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0030] Figure 1 A system block diagram of the large power grid supply and demand regulation system based on multi-source and storage collaboration provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0032] Please refer to the system block diagram of the present invention. An embodiment of the present invention provides a technical solution: a large power grid supply and demand regulation system based on multi-source storage collaboration, including a real-time database storage module and a non-real-time database storage module. Data with high real-time requirements are stored in the real-time database for quick access and analysis. The real-time database can process high-speed data streams and provide low-latency responses; for data with low real-time requirements, it is stored in non-real-time databases. These databases can store large amounts of data and provide data query and analysis functions. It should also be noted that the real-time database storage module is built with a high-performance in-memory database In-Memory Database to ensure the real-time and high-speed processing capabilities of the data, and the data is transmitted in real time from the temporary storage module or other data sources to the real-time database storage module through the efficient data transmission protocol Kafka;

[0033] The non-real-time database storage module is built using the relational database MySQL or the columnar database HBase to support the storage of large amounts of data and complex queries;

[0034] The large power grid supply and demand regulation system based on multi-source storage coordination also includes:

[0035] A temporary storage module, when a large power grid supply and demand regulation system based on multi-source storage collaboration stores various types of data, the temporary storage module is used to temporarily store the data, and the temporary storage module supports user access and search, wherein the temporary storage module is constructed using a distributed file system, wherein the distributed file system includes Hadoop HDF and Ceph to ensure data availability and scalability;

[0036] It should be noted that users can access and search for data in the temporary storage module through a web-based user interface (UI) or application programming interface (API). These interfaces are developed and used using the Spring Boot framework. The temporary storage module can also integrate a data cache layer such as Redis to increase data access speed and reduce the pressure of direct access to the underlying storage, thereby improving the efficiency of searching for data in the temporary data storage module.

[0037] A data monitoring module, which is connected to the temporary storage module, the real-time database storage module and the non-real-time database storage module through an interface. When a user accesses and searches for data in the temporary storage module, the data monitoring module is used to monitor and mark the accessed data, and record the number of accesses and access time within a time interval. The data monitoring module is constructed using the stream processing framework of Apache Flink to achieve real-time monitoring and marking of data in the temporary storage module, and the data monitoring module also integrates log collection and analysis tools to record and analyze data access logs, wherein the log collection and analysis tools include but are not limited to Logstash and ELK Stack, and the monitoring data is visualized through tools such as Elasticsearch or Kibana, so that the data access situation can be intuitively understood;

[0038] It should be noted here that the specific steps of data monitoring by the data monitoring module are as follows:

[0039] Step A: The data monitoring module captures data access events in the temporary storage module through the data source interface. These events include but are not limited to data reading, writing, querying and other operations;

[0040] Step B: After capturing data access events, the data monitoring module uses the built-in real-time monitoring engine (based on event-driven architecture) to process these events in real time. For each access, the data monitoring module will record the access time and access count;

[0041] The time interval here can be one week or one month. It needs to be set according to the specific situation. It should not be too long or too short. It is recommended to set it to one week, half a month or one month.

[0042] A data classification module, which is used to import data into a real-time database storage module or a non-real-time database storage module according to the number of accesses to the data in the temporary database within a time interval, and when the number of times a type of data is classified into the real-time database storage module / non-real-time database storage module exceeds a threshold, the subsequent storage of the type of data will be directly stored in the real-time database storage module / non-real-time database storage module instead of the temporary storage module;

[0043] The data classification module uses machine learning algorithms to classify data so as to automatically import the data into the real-time database storage module or the non-real-time database storage module according to the number of accesses. The classification results are verified and adjusted through a rule-based engine (Drools) to ensure the accuracy and rationality of the classification;

[0044] The method for classifying data by the data classification module is divided into the following steps:

[0045] Step C: The data classification module first obtains the access statistics of various types of data within a period of time from the data monitoring module, including the number of accesses and access time distribution.

[0046] Step D: Based on the access frequency information of the data, the data classification module uses a preset classification algorithm (threshold classification method based on access frequency) to classify the data. If the access frequency of a certain type of data exceeds the set threshold, it is regarded as hot data and is imported into the real-time database storage module; otherwise, it is regarded as non-hot data and imported into the non-real-time database storage module. After the classification decision is completed, the data classification module will perform the corresponding data migration operation to migrate the data from the temporary storage module to the corresponding storage module;

[0047] It should be noted here that in the data classification module, the preset threshold can be set as the number of accesses to the data within a specific time interval (such as a week, half a month or a month). For example, a threshold is set to 1,000 accesses, that is, if a certain type of data is accessed more than 1,000 times within a specified time interval, then this type of data is considered hot data and should be directly imported into the real-time database storage module. On the contrary, if the number of accesses is lower than this threshold, it is considered non-hot data and imported into the non-real-time database storage module. When the number of times a certain type of data is classified for the first time and imported into the real-time database storage module or the non-real-time database storage module exceeds another preset threshold (such as 5 times), the system records this classification decision, and when encountering the same type of data in the future, it directly skips the temporary storage module and imports it into the previously determined storage module;

[0048] It should also be noted that in order to determine whether the subsequent data belongs to the classified "same type", the data classification module can identify it based on the metadata of the data (such as data format, source, identifier, etc.). For example, if the data contains specific identifiers (such as data ID, unit symbol, field name, etc.), it can be determined whether the newly received data is the same as the classified data type by matching these identifiers. Once it is identified that the data belongs to the classified type, the system will directly import the data into the corresponding real-time database storage module / non-real-time database storage module based on the previous decision results;

[0049] As more and more data is directly classified and imported into the real-time database storage module or the non-real-time database storage module, the data in the temporary storage module will gradually decrease. When the system detects that the amount of data in the temporary storage module is lower than a preset threshold (such as the amount of remaining data is less than 10% of the total capacity, or no new data has been temporarily stored for a period of time), and confirms that all remaining data has been successfully migrated to the corresponding storage module, the system can automatically trigger the operation of deleting the temporary storage module. This operation can be achieved by releasing the resources occupied by the temporary storage module, deleting related files or database tables;

[0050] A subsequent optimization module is used to monitor the number of accesses to data in the real-time database and the non-real-time database within a time interval, and according to the number of data accesses within a time interval, transfer the data in the real-time database to the non-real-time database, compress the data in the non-real-time database and mark it as long-term data. The subsequent optimization module uses a data migration tool to realize data transmission between the real-time database and the non-real-time database, wherein data compression uses Hadoop's MapReduce for batch processing to improve data compression efficiency and quality. The data migration tool includes but is not limited to the Apache Airflow data migration tool;

[0051] The specific steps for the subsequent optimization module to optimize the real-time data storage module and the non-real-time data storage module are as follows:

[0052] Step E: The subsequent optimization module first analyzes the data access patterns of the real-time database and the non-real-time database, including but not limited to counting the number of accesses and access time intervals of each data;

[0053] Step F: Based on the analysis results, the module will migrate the data that is no longer active in the real-time database (data with a sudden decrease in access times) to the non-real-time database; at the same time, the module will compress the data in the non-real-time database that has been stored for a long time and has a very low access frequency, and mark it as long-term data;

[0054] It should be noted that in the subsequent optimization module, in addition to migrating the no longer active data in the real-time database to the non-real-time database and compressing the data in the non-real-time database that has been stored for a long time and has a very low access frequency, there is another situation that needs to be considered: when the access frequency of data in the non-real-time database storage module suddenly increases, the subsequent optimization module will use data migration tools (such as Apache Airflow) to achieve data migration;

[0055] In terms of data compression, the subsequent optimization module makes full use of Hadoop's MapReduce framework for batch processing to improve the efficiency and quality of data compression. MapReduce can process large-scale data sets in parallel, decompose complex data compression tasks into multiple simple tasks, and execute them in a distributed computing environment. This not only speeds up data compression, but also ensures the quality of compressed data. At the same time, for data sets marked as long-term data, the module will store them on lower-cost long-term storage media, which can not only meet the needs of long-term data preservation, but also effectively control storage costs.

[0056] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A large power grid supply and demand regulation system based on multi-source storage collaboration, including a real-time database storage module and a non-real-time database storage module, characterized in that: The large power grid supply and demand regulation system based on multi-source storage collaboration also includes: A temporary storage module, when the large power grid supply and demand regulation system based on multi-source storage collaboration stores various types of data, the data is temporarily stored through the temporary storage module, and the temporary storage module supports user access and search; A data monitoring module, which is connected to the temporary storage module, the real-time database storage module and the non-real-time database storage module through an interface. When a user accesses and searches for data in the temporary storage module, the data monitoring module is used to monitor and mark the accessed data, and record the number of accesses and access time within a time interval; A data classification module, which is used to import data into a real-time database storage module or a non-real-time database storage module according to the number of accesses to the data in the temporary database within a time interval, and when the number of times a type of data is classified into the real-time database storage module / non-real-time database storage module exceeds a threshold, the subsequent storage of the type of data will be directly stored in the real-time database storage module / non-real-time database storage module instead of the temporary storage module; The subsequent optimization module is used to monitor the number of accesses to data in the real-time database and the non-real-time database within a time interval, and according to the number of data accesses within a time interval, transfer the data in the real-time database to the non-real-time database, compress the data in the non-real-time database and mark it as long-term data.

2. The large power grid supply and demand regulation system based on multi-source storage collaboration according to claim 1 is characterized in that: The temporary storage module is constructed using a distributed file system, wherein the distributed file system includes Hadoop HDF and Ceph to ensure the availability and scalability of data.

3. The large power grid supply and demand regulation system based on multi-source storage collaboration according to claim 2 is characterized in that: The data monitoring module is built using the stream processing framework of Apache Flink to achieve real-time monitoring and marking of data in the temporary storage module, and the data monitoring module also integrates log collection and analysis tools to record and analyze data access logs, wherein the log collection and analysis tools include but are not limited to Logstash and ELK Stack.

4. The large power grid supply and demand regulation system based on multi-source storage collaboration according to claim 3 is characterized in that: The data classification module uses a machine learning algorithm to classify the data so as to automatically import the data into a real-time database storage module or a non-real-time database storage module according to the number of accesses.

5. The large power grid supply and demand regulation system based on multi-source storage collaboration according to claim 4 is characterized in that: The subsequent optimization module uses a data migration tool to realize data transmission between the real-time database and the non-real-time database, wherein data compression uses Hadoop's MapReduce for batch processing to improve data compression efficiency and quality, and the data migration tool includes but is not limited to the Apache Airflow data migration tool.

6. The large power grid supply and demand regulation system based on multi-source storage collaboration according to claim 5 is characterized in that: The data monitoring module specifically monitors the data in the following steps: Step A: The data monitoring module captures data access events in the temporary storage module through the data source interface. These events include but are not limited to data reading, writing, querying and other operations; Step B: After capturing data access events, the data monitoring module uses the built-in real-time monitoring engine (based on event-driven architecture) to process these events in real time. For each access, the data monitoring module will record the access time and number of accesses.

7. The large power grid supply and demand regulation system based on multi-source storage collaboration according to claim 6 is characterized in that: The method for classifying data by the data classification module is divided into the following steps: Step C: The data classification module first obtains the access statistics of various types of data within a period of time from the data monitoring module, including the number of accesses and access time distribution. Step D: Based on the data access count information, the data classification module uses a preset classification algorithm (threshold classification method based on access frequency) to classify the data. If the access count of a certain type of data exceeds the set threshold, it is regarded as hot data and is decided to be imported into the real-time database storage module; otherwise, it is regarded as non-hot data and imported into the non-real-time database storage module. After the classification decision is completed, the data classification module will perform the corresponding data migration operation to migrate the data from the temporary storage module to the corresponding storage module.

8. The large power grid supply and demand regulation system based on multi-source storage collaboration according to claim 7 is characterized in that: The specific steps of the subsequent optimization module optimizing the real-time data storage module and the non-real-time data storage module are as follows: Step E: The subsequent optimization module first analyzes the data access patterns of the real-time database and the non-real-time database, including but not limited to counting the number of accesses and access time intervals of each data; Step F: Based on the analysis results, the module will migrate the data in the real-time database that is no longer active (data with a sudden decrease in the number of accesses) to the non-real-time database; at the same time, for the data in the non-real-time database that has been stored for a long time and has a very low access frequency, the module will compress it and mark it as long-term data.

Images