Time Series Data Storage Management Method, Medium and Concentrator Based on Aggregation Processing

Abstract

The present invention relates to a time-series data storage management method, medium and concentrator based on aggregation processing. After processing each time-series data collected by each electric meter according to a preset acquisition time format to obtain preset formatted time-series data, each electric meter in the power distribution area is traversed, and judgment processing is performed based on each obtained preset formatted time-series data and the database of the concentrator in the power distribution area, so that the time-series data with the same date are placed together, enabling these time-series data to be aggregated within the dimension of one day. In this way, the number of data records for dispersed recording of each time-series data can be reduced, and the storage space for the number of data records can be decreased. Since the time-series data are aggregated according to the date (day), the dimension of the index is reduced, and thus the number of indexes for querying data will be greatly reduced, thereby improving the query efficiency for time-series data.

Description

Technical Field

[0001] The present invention relates to the field of electric power, and particularly to a method, medium and concentrator for storing and managing time series data based on aggregation processing. Background Art

[0002] In the field of electric power, a large amount of time series data is generated during the operation of each electric meter in a transformer substation area. The concentrator in the same transformer substation area needs to store this time series data. As time goes by, more and more time series data is generated, which poses higher requirements for the storage performance of the concentrator.

[0003] In order to cope with the challenge of the storage performance of the acquisition system caused by the large amount of generated time series data, currently, mainly the time series data is stored in the document database of the concentrator. Each piece of document data stores one piece of time series data, and indexes are created on the key fields. The tables are divided according to the time field to improve the query performance of querying time series data using the document database. Then, the statistical analysis data obtained based on these time data is stored separately.

[0004] However, the existing time series data storage method has deficiencies: Since the number of electric meters in the transformer substation area is large, and each electric meter collects data at a certain collection frequency every day, the amount of time series data generated every day is very large. The concentrator needs to create more indexes to facilitate querying the required time series data, which thus occupies a large amount of storage space and reduces the index efficiency. Summary of the Invention

[0005] The first technical problem to be solved by the present invention is to provide a method for storing and managing time series data based on aggregation processing in view of the above-mentioned prior art. The method for storing and managing time series data based on aggregation processing performs aggregated storage on the time series data generated by the electric meter, reduces the number of data storage records and the number of indexes, reduces the data storage space, and improves the query performance for time series data.

[0006] The second technical problem to be solved by the present invention is to provide a readable storage medium. A computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the method for storing and managing time series data based on aggregation processing is implemented.

[0007] The third technical problem to be solved by the present invention is to provide a concentrator. The concentrator includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the method for storing and managing time series data based on aggregation processing is implemented.

[0008] The technical solution adopted by the present invention to solve the first technical problem is: a time-series data storage management method based on aggregation processing, which is characterized by including the following steps:

[0009] Step S1, process the time-series data of each electric meter collected according to a preset collection time format to obtain preset formatted time-series data corresponding to each electric meter; wherein, the preset formatted time-series data includes time-series data and collection time, and the collection time includes collection date and collection moment, and each electric meter corresponds to at least one preset formatted time-series data;

[0010] Step S2, traverse each electric meter in the substation area and make a judgment process according to each obtained preset formatted time-series data and the database of the concentrator in the substation area:

[0011] When any preset formatted time-series data has the same collection date as any time-series data in the database, insert the time-series data of the any preset formatted time-series data into the time-series data array of the corresponding electric meter, and transfer to Step S3; otherwise, insert the time-series data of the any preset formatted time-series data into the database of the concentrator as an independent time-series data group, and transfer to Step S3;

[0012] Step S3, take the time-series data array inserted with the preset formatted time-series data as the latest time-series data group of the corresponding electric meter.

[0013] Improved, in the present invention, the time-series data storage management method based on aggregation processing further includes: performing regular statistical analysis processing on the time-series data in each latest time-series data group to obtain statistical analysis data.

[0014] Further, in the time-series data storage management method based on aggregation processing, the statistical analysis data is at least one of the maximum value, minimum value and average value of the time-series data.

[0015] Further improved, in the present invention, the time-series data storage management method based on aggregation processing further includes:

[0016] Step a1, obtain data query request information;

[0017] Step a2, make a process according to the data query request information:

[0018] When the data query request information is a time-series data query request, disassemble the time-series data located in the latest time-series data group and corresponding to the time-series data query request into time-series data in a complete format, and transfer to step a3; otherwise, feedback information indicating that data query cannot be provided; wherein, the time-series data query request has the time-series data to be queried, and the time-series data of the time-series data query request has the same data structure form as the disassembled time-series data in the complete format.

[0019] Step a3: Provide the disassembled time-series data in the complete format to the requester that sends the data query request information.

[0020] Improved, in this invention, the time-series data storage management method based on aggregation processing further includes: obtaining the time-series data acquisition success rate of the electric meter according to the latest time-series data group of the obtained electric meter within each day and the acquisition interval of the electric meter within each day; wherein, the time-series data acquisition success rate of the electric meter is the ratio between the total number of the latest time-series data in the latest time-series data group of the electric meter within each day and the total number of acquisitions of the electric meter within each day, and the total number of acquisitions of the electric meter within each day is the integer value of the ratio between the total duration of each day and the acquisition interval.

[0021] Further, in this invention, the time-series data storage management method based on aggregation processing further includes: pre-generating a target number of empty time-series data groups in the database of the concentrator; wherein, the target number is the total number of time-series data collected by the electric meter within one day according to the acquisition interval, and the number of built-in time-series data in the empty time-series data group in its initial state is zero; and inserting the time-series data collected by the electric meter into the corresponding empty time-series data group.

[0022] Improved again, in this invention, the time-series data storage management method based on aggregation processing further includes:

[0023] Take out the acquisition time part of the preset formatted time-series data in a form consistent with the unit of the acquisition interval;

[0024] Convert the taken-out acquisition time part into a form consistent with the acquisition interval unit to obtain the converted acquisition duration;

[0025] Calculate the ratio between the converted acquisition duration and the acquisition interval, and perform integer processing on the ratio to obtain the integer ratio;

[0026] And use the integer ratio as the position of the preset formatted time-series data in the updated time-series data group.

[0027] Improved, in the time-series data storage management method based on aggregation processing, the acquisition interval is 15 min.

[0028] The technical solution adopted by the present invention to solve the second technical problem is as follows: a readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, any one of the described time-series data storage management methods based on aggregation processing is implemented.

[0029] The technical solution adopted by the present invention to solve the third technical problem is as follows: a concentrator, characterized by including a memory, a processor, and a computer program stored on the memory and operable on the processor, and when the processor executes the computer program, any one of the described time-series data storage management methods based on aggregation processing is implemented.

[0030] Compared with the prior art, the advantages of the present invention are as follows: in this invention, for the time-series data storage management method based on aggregation processing, after processing each piece of time-series data collected by each electric meter according to a preset collection time format to obtain preset formatted time-series data, then traverse each electric meter in the substation area and make a judgment and processing based on each obtained preset formatted time-series data and the database of the concentrator in this substation area, so as to put the time-series data with the same date together, enabling these time-series data to be aggregated within the dimension of one day. In this way, the number of data records for scattered recording of each piece of time-series data can be reduced, and the storage space for the number of data records can be decreased; since the time-series data is aggregated according to the date (day), the dimension of the index is reduced, and thus the number of indexes for querying data will be greatly reduced, thereby improving the query efficiency for time-series data. In addition, by performing aggregation processing on these time-series data, the amount of duplicate attribute data in this large amount of time-series data can be reduced, saving storage space. Description of the Drawings

[0031] Figure 1 It is a schematic flowchart of the time-series data storage management method based on aggregation processing in an embodiment of the present invention. Detailed Embodiment

[0032] The present invention will be further described in detail below in conjunction with the embodiments of the drawings.

[0033] This embodiment provides a time-series data storage management method based on aggregation processing. Specifically, as shown in Figure 1 The time-series data storage management method based on aggregation processing in this embodiment includes the following steps S1 to S3:

[0034] Step S1, process each piece of time-series data of each electric meter collected according to a preset collection time format to respectively obtain preset formatted time-series data corresponding to each electric meter; wherein, the preset formatted time-series data includes time-series data and a collection time, and the collection time includes a collection date and a collection moment, and each electric meter corresponds to at least one preset formatted time-series data;

[0035] Specifically in this embodiment, the preset acquisition time format here is yyyy-MM-dd. For example, for the electricity meter 1, the time-series data of the electricity meter 1 is as shown in Table 1 below:

[0036] Table 1

[0037]

[0038] It can be known that the time-series data group of the electricity meter 1 contains 6 preset formatted time-series data, which are: the first preset formatted time-series data is {voltage: 230, current: 120, acquisition time 0:00:00}; the second preset formatted time-series data is {voltage: 230, current: 120, acquisition time 0:15:00}; the other four are shown in Table 1;

[0039] Step S2, traverse each electricity meter in the substation area and make a judgment and processing according to each obtained preset formatted time-series data and the database of the concentrator in the substation area:

[0040] When any one of the preset formatted time-series data has the same acquisition date as any time-series data in the database, insert the time-series data of the any one of the preset formatted time-series data into the time-series data array of the corresponding electricity meter, and transfer to step S3; otherwise, insert the time-series data of the any one of the preset formatted time-series data as an independent time-series data group into the database of the concentrator, and transfer to step S3; where, "when any one of the preset formatted time-series data has the same acquisition date as any time-series data in the database, insert the time-series data of the any one of the preset formatted time-series data into the time-series data array of the corresponding electricity meter" is to perform aggregation processing according to the dimension of days;

[0041] Step S3, take the time-series data array inserted with the preset formatted time-series data as the latest time-series data group of the corresponding electricity meter.

[0042] For the need of analyzing the time-series data in these time-series data groups, the time-series data storage management method based on aggregation processing in this embodiment also performs regular statistical analysis processing on the time-series data in each latest time-series data group to obtain statistical analysis data. For example, the statistical analysis data here is at least one of the maximum value, minimum value and average value of the time-series data.

[0043] To meet the actual needs of querying time-series data, the time-series data storage management method based on aggregation processing in this embodiment further makes the following improvements: Step a1, obtain data query request information; Step a2, process according to the data query request information: when the data query request information is a time-series data query request, disassemble the time-series data located in the latest time-series data group and corresponding to the time-series data query request into time-series data in a complete format, and transfer to Step a3; otherwise, feedback information indicating that data query cannot be provided; wherein, the time-series data query request has the time-series data to be queried, and the time-series data of the time-series data query request has the same data structure form as the disassembled time-series data in the complete format; Step a3, provide the disassembled time-series data in the complete format to the requester who sends the data query request information.

[0044] To meet the actual needs of mastering the success rate of time-series data acquisition, in this embodiment, the success rate of time-series data acquisition of the electric meter is further obtained according to the latest time-series data group of the obtained electric meter within each day and the acquisition interval of the electric meter within each day; wherein, the success rate of time-series data acquisition of the electric meter is the ratio between the total number of the latest time-series data in the latest time-series data group of the electric meter within each day and the total number of acquisitions of the electric meter within each day, and the total number of acquisitions of the electric meter within each day is the integer value of the ratio between the total duration of each day and the acquisition interval. For example, the acquisition interval here is 15 min.

[0045] To avoid the problem of insufficient reserved space for the time-series data to be inserted in the early stage, the time-series data storage management method based on aggregation processing in this embodiment further makes the following improvements: pre-generate a target number of empty time-series data groups in the database of the concentrator; wherein, the target number is the total number of time-series data collected by the electric meter within one day according to the acquisition interval, and the number of built-in time-series data in the empty time-series data group in its initial state is zero; and insert the time-series data collected by the electric meter into the corresponding empty time-series data group.

[0046] To accurately determine the position of the time-series data to be inserted in the time-series data group, this embodiment determines the position by taking the following steps b1 to b measures, that is:

[0047] Step b1, extract the acquisition time part of the preset formatted time-series data in a form consistent with the unit of the acquisition interval;

[0048] Step b2, convert the extracted acquisition time part into a form consistent with the acquisition interval unit to obtain the converted acquisition duration;

[0049] Step b3, calculate the ratio between the converted acquisition duration and the acquisition interval, and perform rounding processing on the ratio to obtain the rounded ratio; and,

[0050] Step b4: Use the rounded ratio as the position of the preset formatted time series data in the updated time series data group.

[0051] For example, for meter 1, it is known that the unit of the acquisition interval is min. Then, extract the acquisition time part of its fifth time series data, which is 60 min (i.e., 1 * 60 = 60). Since the acquisition interval is 15 min, the rounded ratio of the converted acquisition duration of 60 min to the acquisition interval of 15 min is 4. Then, use the rounded ratio 4 as the position of the preset formatted time series data {voltage: 210, current: 223, acquisition time 1:00:00} in the updated time series data group.

[0052] This embodiment also provides a readable storage medium. Specifically, a computer program is stored on the readable storage medium, and when the computer program is executed by a processor, the above-described time series data storage management method based on aggregation processing is implemented.

[0053] This embodiment also provides a concentrator, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the above-described time series data storage management method based on aggregation processing is implemented.

[0054] Although the preferred embodiments of the present invention have been described in detail above, it should be clearly understood that various changes and modifications can be made to the present invention for those skilled in the art. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for storing and managing time-series data based on aggregation processing, characterized in that It includes the following steps S1 to S3: Step S1: Process the time-series data of each electricity meter collected according to a preset collection time format to obtain preset formatted time-series data corresponding to each electricity meter. Among them, the preset formatted time-series data includes time-series data and collection time, and the collection time includes collection date and collection moment. Each electricity meter corresponds to at least one preset formatted time-series data; Step S2: Traverse each electricity meter in the area and make a judgment and processing based on each obtained preset formatted time-series data and the database of the concentrator in this area: When any preset formatted time-series data has the same collection date as any time-series data in the database, insert the time-series data of this preset formatted time-series data into the time-series data array of the corresponding electricity meter, and go to Step S3; otherwise, insert the time-series data of this preset formatted time-series data into the database of the concentrator as an independent time-series data group, and go to Step S3; Step S3: Use the time-series data array inserted with the preset formatted time-series data as the latest time-series data group of the corresponding electricity meter; Among them, this time-series data storage management method based on aggregation processing further includes: Obtain the time-series data collection success rate of the electricity meter according to the latest time-series data group of the electricity meter within each day and the collection interval of the electricity meter within each day. Among them, the time-series data collection success rate of the electricity meter is the ratio between the total number of the latest time-series data in the latest time-series data group of the electricity meter within each day and the total number of collections of the electricity meter within each day. The total number of collections of the electricity meter within each day is the integer value of the ratio between the total duration of each day and the collection interval; Pre-generate a target number of empty time-series data groups in the database of the concentrator. Among them, the target number is the total number of time-series data collected by the electricity meter according to the collection interval within one day, and the number of built-in time-series data in the empty time-series data group in its initial state is zero; and insert the time-series data collected by the electricity meter into the corresponding empty time-series data group.

2. The method for storing and managing time-series data based on aggregation processing according to claim 1, wherein It further includes: Perform regular statistical analysis processing on the time-series data in each latest time-series data group to obtain statistical analysis data.

3. The method for storing and managing time-series data based on aggregation processing according to claim 2, wherein The statistical analysis data is at least one of the maximum value, minimum value, and average value of the time-series data.

4. The method for storing and managing time series data based on aggregation processing according to claim 1, wherein It further includes steps a1 to a3: Step a1: Obtain data query request information; Step a2: Make a processing according to the data query request information: When the data query request information is a time-series data query request, disassemble the time-series data located in the latest time-series data group and corresponding to this time-series data query request into time-series data in a complete format, and go to Step a3; Otherwise, feedback information that data query cannot be provided. Among them, the time-series data query request has the time-series data to be queried, and the time-series data of the time-series data query request has the same data structure form as the disassembled time-series data in a complete format; Step a3: Provide the disassembled time-series data in a complete format to the requester who sends the data query request information.

5. The time-series data storage management method based on aggregation processing according to any one of claims 1 to 4, characterized in that Extract the acquisition time part of the preset formatted time series data in a form consistent with the unit of the acquisition interval; Convert the extracted acquisition time part into a form consistent with the acquisition interval unit to obtain the converted acquisition duration; Calculate the ratio between the converted acquisition duration and the acquisition interval, and round the ratio to obtain the rounded ratio; And use the rounded ratio as the position of the preset formatted time series data in the updated time series data group.

6. The method for managing the storage of time series data based on aggregation processing according to claim 1, wherein The acquisition interval is 15 minutes.

7. A readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the time series data storage management method based on aggregation processing according to any one of claims 1 to 6.

8. Concentrator, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the time series data storage management method based on aggregation processing according to any one of claims 1 to 6.

Images