Tiered storage system and method based on time sequence library

Abstract

The invention discloses a hierarchical storage system and method based on a time sequence library. The system comprises a real-time library, the time sequence library and a relation library, the real-time library is connected with a sampling unit and a storage unit; the sampling unit reads the real-time electric power data from the real-time library regularly or when a preset triggering condition is met, and the real-time electric power data is subjected to data preprocessing and then sent to the storage unit; the storage unit stores the real-time power data to the time sequence library and the relation library; the Idkey processing unit extracts ID information and Key information of the real-time power data and synchronizes the ID information and the Key information to a time sequence library and a relation library; through cooperative work of the real-time library, the time sequence library and the relation library, the integrity and consistency of data are ensured, the time sequence library optimizes time sequence data, power data can be quickly inquired and analyzed, and the requirements of real-time monitoring and quick decision making are met.

Description

Technical Field

[0001] The present invention belongs to the field of data storage management, and in particular relates to a hierarchical storage system and method based on a time series library. Background Art

[0002] In the power dispatching automation system, real-time database and relational database are usually used to store power data. The real-time database has high-speed writing and reading capabilities, and can record the operation status of the power grid in real time, providing instant feedback and decision support for power grid operation and maintenance. The relational database is used for long-term storage and management of large amounts of historical data, which can be used to analyze and review the past operation of the power system, perform fault diagnosis, performance analysis, and future planning.

[0003] As the amount of single-table data in the real-time database and relational database continues to increase, the query and statistics of historical data by the on-site systems of some projects have become slower, and occasionally the real-time database retrieval time exceeds the relational database query time. Summary of the invention

[0004] The present invention provides a hierarchical storage system and method based on a time series library to solve the problem of low efficiency in querying historical data in an electric power dispatching automation system.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is: The first aspect of the present invention provides a hierarchical storage system based on a time series library, comprising a real-time library, a time series library and a relationship library; the real-time library is connected to a sampling unit and a storage unit; the sampling unit reads real-time power data from the real-time library regularly or when a preset trigger condition is reached, and performs data preprocessing on the real-time power data and then sends it to the storage unit; the storage unit stores the real-time power data in the time series library and the relationship library; The IDkey processing unit extracts the ID information and Key information of the real-time power data, and synchronizes the ID information and Key information to the timing library and the relationship library; the relationship library constructs a relationship table based on the ID information and Key information, and stores the real-time power data in the relationship table; the timing library stores the real-time power data in chronological order; the timing library regularly cleans up the real-time power data that exceeds the set time limit.

[0006] Furthermore, the real-time library includes a SCADA model library and a HIS model library for storing real-time power data, and the real-time power data is metadata information and real-time operation data of a power system test point.

[0007] Furthermore, the preset triggering conditions include: an electric power parameter of the electric power system exceeds a preset first threshold, a fault alarm occurs in the electric power system, the accumulated power of the electric power system reaches a second threshold, or a data collection instruction is received.

[0008] Furthermore, the sampling unit preprocesses the real-time power data including: The sampling unit records the timestamp of the real-time power data, verifies the integrity of the real-time power data based on the timestamp, and obtains the timestamp of the missing real-time power data; Fitting a function to the real-time power data, and filling in the missing real-time power data according to the fitting function of the real-time power data; After removing noise and outliers from the real-time power data, the real-time power data is normalized or standardized.

[0009] Furthermore, it also includes an extreme value statistical unit, and the real-time library, relationship library and timing library are electrically connected to the extreme value statistical unit; the extreme value statistical unit performs statistical calculations on the real-time power data according to a set time window to obtain extreme value indicators, and the extreme value indicators include maximum load, minimum load, average load and peak load time; anomaly detection is performed based on the extreme value indicators.

[0010] Furthermore, it also includes a data retrieval unit; the real-time library, relationship library and timing library are electrically connected to the data retrieval unit; the data retrieval unit receives a retrieval request instruction, and searches the timing library first according to the retrieval request instruction. If there is no query result corresponding to the retrieval request instruction, search the relationship library to obtain the query result.

[0011] Furthermore, the client is electrically connected to the relationship library and the timing library through a universal interface; the client is configured with a metadata structure dynamic library and a universal interface program dynamic library; The metadata structure dynamic library contains time series library type enumeration information, maximum value class statistics information, idkey data structure class information and time range information; The timing library plug-in is called from the general interface program dynamic library according to the configuration file of the timing library, and the real-time power data is read from the timing library through the timing library plug-in.

[0012] A second aspect of the present invention provides a hierarchical storage method based on a time series library, comprising: Reading real-time power data from the real-time database periodically or when a preset trigger condition is reached, and sending the real-time power data to the storage unit after data preprocessing; storing the real-time power data in the timing database and the relationship database through the storage unit; The ID information and Key information of the real-time power data are extracted, and the ID information and Key information are synchronized to the timing library and the relationship library; the relationship library constructs a relationship table based on the ID information and Key information, and stores the real-time power data in the relationship table; the timing library stores the real-time power data in chronological order; the timing library regularly cleans up the real-time power data that exceeds the set time limit.

[0013] Further, a search request instruction is received, and a search is preferentially performed in the time series library according to the search request instruction. If there is no query result corresponding to the search request instruction, a search is performed in the relationship library to obtain the query result.

[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention reads real-time power data from the real-time library periodically or when a preset trigger condition is reached, and stores the real-time power data in the timing library and the relationship library; the timing library is optimized for time series data, and can quickly query and analyze power data to meet the needs of real-time monitoring and rapid decision-making; the relationship library is used for long-term storage and management of historical data, which is convenient for historical data analysis, trend analysis and long-term planning.

[0015] The IDkey processing unit of the present invention extracts the ID information and Key information of the real-time power data, and synchronizes the ID information and Key information to the timing library and the relationship library; the relationship library constructs a relationship table based on the ID information and the Key information, and stores the real-time power data in the relationship table; the timing library stores the real-time power data in chronological order; the timing library regularly cleans up the real-time power data that exceeds the set time limit; the integrity and consistency of the data are ensured through the collaborative work of the real-time library, the timing library and the relationship library. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A structural diagram of a hierarchical storage system provided in Example 1 of the present invention; Figure 2 A timing library interface calling process diagram is provided for embodiment 1 of the present invention. DETAILED DESCRIPTION

[0017] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0018] Example 1

[0019] like Figure 1 and Figure 2 As shown, this implementation provides a hierarchical storage system based on a timing library, including a real-time library, a timing library and a relationship library; the real-time library includes a SCADA model library and a HIS model library for storing real-time power data, and the real-time power data is metadata information and real-time operation data of a power system test point.

[0020] The real-time library is connected to a sampling unit and a storage unit; the sampling unit reads real-time power data from the real-time library periodically or when a preset trigger condition is reached, and the preset trigger condition includes: a power parameter of the power system exceeds a preset first threshold, a fault alarm occurs in the power system, the accumulated power of the power system reaches a second threshold, or a data collection instruction is received.

[0021] The sampling unit preprocesses the real-time power data including: The sampling unit records the timestamp of the real-time power data, verifies the integrity of the real-time power data based on the timestamp, and obtains the timestamp of the missing real-time power data; Fitting a function to the real-time power data, and filling in the missing real-time power data according to the fitting function of the real-time power data; After removing noise and outliers from the real-time power data, the real-time power data is normalized or standardized.

[0022] The real-time power data is pre-processed and then sent to the storage unit; the storage unit stores the real-time power data in the timing library and the relationship library.

[0023] The IDkey processing unit extracts the ID information and Key information of the real-time power data, and synchronizes the ID information and Key information to the timing library and the relationship library; the relationship library constructs a relationship table based on the ID information and Key information, and stores the real-time power data in the relationship table; the timing library stores the real-time power data in chronological order; the timing library regularly cleans up the real-time power data that exceeds the set time limit.

[0024] The real-time library, relationship library and timing library are electrically connected to the maximum value statistical unit; the maximum value statistical unit performs statistical calculations on the real-time power data according to the set time window to obtain maximum value indicators, and the maximum value indicators include maximum load, minimum load, average load and peak load time; anomaly detection is performed based on the maximum value indicators.

[0025] The real-time library, relationship library and timing library are electrically connected to a data retrieval unit; the data retrieval unit receives a retrieval request instruction and searches the timing library first according to the retrieval request instruction. If there is no query result corresponding to the retrieval request instruction, the relationship library is searched to obtain the query result.

[0026] The client is electrically connected to the relationship library and the timing library through a universal interface; the client is configured with a metadata structure dynamic library and a universal interface program dynamic library; the metadata structure dynamic library contains timing library type enumeration information, maximum value class statistical information, idkey data structure class information and time range information; according to the configuration file of the timing library, the timing library plug-in is called from the universal interface program dynamic library, and real-time power data is read from the timing library through the timing library plug-in.

[0027] Example 2

[0028] like Figure 1 As shown, this embodiment provides a hierarchical storage method based on a time series library. The hierarchical storage method described in Example 1 is applied to a hierarchical storage system. The hierarchical storage method includes: Reading real-time power data from the real-time database periodically or when a preset trigger condition is reached, and sending the real-time power data to the storage unit after data preprocessing; storing the real-time power data in the timing database and the relationship database through the storage unit; The ID information and Key information of the real-time power data are extracted, and the ID information and Key information are synchronized to the timing library and the relationship library; the relationship library constructs a relationship table based on the ID information and Key information, and stores the real-time power data in the relationship table; the timing library stores the real-time power data in chronological order; the timing library regularly cleans up the real-time power data that exceeds the set time limit.

[0029] Receive a search request instruction, and search the time series library first according to the search request instruction. If there is no query result corresponding to the search request instruction, search the relationship library to obtain the query result.

[0030] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, etc.) containing computer-usable program code. M , optical storage, etc.).

[0031] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0032] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0033] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0034] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A hierarchical storage system based on a time series library, characterized in that: It includes a real-time library, a time sequence library and a relationship library; the real-time library is connected to a sampling unit and a storage unit; the sampling unit reads real-time power data from the real-time library regularly or when a preset trigger condition is reached, and performs data preprocessing on the real-time power data before sending it to the storage unit; The storage unit stores the real-time power data into the time series library and the relationship library; The IDkey processing unit extracts the ID information and Key information of the real-time power data, and synchronizes the ID information and Key information to the timing library and the relationship library; The relationship library constructs a relationship table based on the ID information and the Key information, and stores the real-time power data in the relationship table; The time series library stores the real-time power data in chronological order; the time series library regularly clears the real-time power data that exceeds a set time limit.

2. The hierarchical storage system according to claim 1, characterized in that: The real-time library includes a SCADA model library and a HIS model library for storing real-time power data, and the real-time power data is metadata information and real-time operation data of a power system test point.

3. The hierarchical storage system according to claim 1, characterized in that: The preset trigger conditions include: the power parameter of the power system exceeds a preset first threshold, a fault alarm occurs in the power system, the accumulated power of the power system reaches a second threshold, or a data collection instruction is received.

4. The hierarchical storage system according to claim 1, characterized in that: The sampling unit preprocesses the real-time power data including: The sampling unit records the timestamp of the real-time power data, verifies the integrity of the real-time power data based on the timestamp, and obtains the timestamp of the missing real-time power data; Fitting a function to the real-time power data, and filling in the missing real-time power data according to the fitting function of the real-time power data; After removing noise and outliers from the real-time power data, the real-time power data is normalized or standardized.

5. The hierarchical storage system according to claim 1, characterized in that: It also includes an extreme value statistical unit, and the real-time library, relationship library and timing library are electrically connected to the extreme value statistical unit; the extreme value statistical unit performs statistical calculations on the real-time power data according to a set time window to obtain extreme value indicators, and the extreme value indicators include maximum load, minimum load, average load and peak load time; anomaly detection is performed based on the extreme value indicators.

6. The hierarchical storage system according to claim 1, characterized in that: It also includes a data retrieval unit; the real-time library, relationship library and timing library are electrically connected to the data retrieval unit; the data retrieval unit receives a retrieval request instruction, and searches the timing library first according to the retrieval request instruction. If there is no query result corresponding to the retrieval request instruction, search the relationship library to obtain the query result.

7. The hierarchical storage system according to claim 1, characterized in that: The client is electrically connected to the relationship library and the timing library through a universal interface; the client is configured with a metadata structure dynamic library and a universal interface program dynamic library; The metadata structure dynamic library contains time series library type enumeration information, maximum value class statistics information, idkey data structure class information and time range information; The timing library plug-in is called from the general interface program dynamic library according to the configuration file of the timing library, and the real-time power data is read from the timing library through the timing library plug-in.

8. The hierarchical storage method of the hierarchical storage system according to any one of claims 1 to 7, characterized in that: include: Reading real-time power data from the real-time database periodically or when a preset trigger condition is reached, and sending the real-time power data to the storage unit after data preprocessing; The real-time power data is stored in the time series library and the relationship library through the storage unit; Extract the ID information and key information of real-time power data, and synchronize the ID information and key information to the timing library and relationship library; The relationship library constructs a relationship table based on the ID information and the Key information, and stores the real-time power data in the relationship table; The time series library stores the real-time power data in chronological order; the time series library regularly clears the real-time power data that exceeds a set time limit.

9. The hierarchical storage method according to claim 8, characterized in that: Receive a search request instruction, and search the time series library first according to the search request instruction. If there is no query result corresponding to the search request instruction, search the relationship library to obtain the query result.

Images