Time series data file management method, device, equipment and storage medium thereof

Abstract

The embodiments of the present application belong to the field of data management technology, and relate to a method, apparatus, device, and storage medium for time series data file management, including obtaining the table name information of the target storage table to generate a management root directory as the file directory name; obtaining the cache entry time of each cached time series data to be managed; partitioning all time series data to be managed based on the cache entry time and a preset table partitioning expression; generating a management subdirectory under the management root directory; generating corresponding data management files under all management subdirectories according to the partition processing results and the table structure information of the target storage table; obtaining all time series data to be managed, and writing them into the corresponding data management files in the form of a data stream, thereby completing the file management of the time series data to be managed. This method facilitates improving the compression ratio of time series data, reducing disk space usage, increasing the write rate of time series data, and scientifically managing time series data.

Description

Technical Field

[0001] The present application relates to the field of data management technology, and in particular to a method, apparatus, device and storage medium for time series data file management. Background Art

[0002] Time series data, that is, time series data, is mainly collected and generated by various types of real-time monitoring, inspection and analysis equipment in the power industry, chemical industry, etc. The typical characteristics of these industrial data are: high generation frequency, dependence on collection time, multiple measurement points and large amount of information.

[0003] In the field of smart IoT, with the increasing popularity of smart devices and smart gateways, more and more sensors or point meters are able to upload data anytime and anywhere. This data is often time-series data. Currently, each monitoring point can generate multiple data items per second, each of which must correspond to a unique time. Conventional real-time monitoring systems have thousands of monitoring points, each generating data every second, which can generate tens of GB of data per day. Therefore, how to improve the compression ratio of time series data, reduce disk space usage, increase the write rate of time series data, and scientifically manage time series data have become urgent issues in time series data storage and management. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to propose a method, device, equipment and storage medium for time series data file management, so as to improve the compression ratio of time series data, reduce disk space usage, increase the write rate of time series data, and scientifically manage time series data.

[0005] To solve the above technical problems, the present invention provides a method for managing time series data files, which adopts the following technical solutions:

[0006] A method for documenting and managing time series data includes the following steps:

[0007] Obtaining table name information and table structure information of a target storage table, wherein the target storage table is used to store time series data to be managed;

[0008] In a preset file management server, the table name information of the target storage table is used as the file directory name to generate a management root directory;

[0009] According to a preset real-time data transmission mode, the time series data to be managed is cached in real time into the target storage table, and a cache entry time is obtained, wherein the cache entry time is the time when each time series data to be managed is cached into the target storage table;

[0010] The cache entry time is passed as a calculation parameter into a preset table partitioning expression, and partition processing is performed on all time series data to be managed cached in the target storage table to obtain the partition processing result;

[0011] Obtain all return values ​​output by the preset table partition expression after partition processing;

[0012] Under the management root directory, all returned values ​​are used as file directory names to generate management subdirectories corresponding to each partition;

[0013] Generate a data management file in a management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table;

[0014] All the time series data to be managed cached in the target storage table are obtained, and all the time series data to be managed are written into the corresponding data management file in the form of data stream, thereby completing the file management of the time series data to be managed.

[0015] Furthermore, the step of caching the time series data to be managed into the target storage table in real time according to the preset real-time data transmission mode specifically includes:

[0016] When each time series data to be managed is cached and stored, the cache storage time is obtained;

[0017] Using the cache entry time as a sorting field, sort all time series data to be managed in the order of cache entry time to obtain a sorting result;

[0018] All time series data to be managed are cached one by one into the target storage table according to the sorting results.

[0019] Furthermore, before executing the step of passing the cache storage time as a calculation parameter into a preset table partition expression, the method further includes:

[0020] Set the number of rows of time series data to be managed in each partition, and use the number of rows of time series data to be managed in each partition as the row number threshold;

[0021] Setting the target data format of the return value, wherein the return value represents the corresponding output result after the table partition expression is executed;

[0022] After executing the step of passing the cache storage time as a calculation parameter into a preset table partition expression, the method further includes:

[0023] Step A: Set the initial number of rows of time series data to be managed in each partition to 0;

[0024] Step B: When the cache entry time of the time series data to be managed is passed into the preset table partition expression as a calculation parameter, the passed-in calculation parameters are obtained in sequence and the number of the passed-in calculation parameters is counted. Each time an passed-in calculation parameter is obtained, the initial row number information is accumulated by 1.

[0025] Step C: until the accumulated result corresponding to the initial number of rows is the same as the row number threshold corresponding to the target partition, the pre-partitioning process for the target partition is completed;

[0026] Step D: Repeat steps B to C to complete the pre-partitioning process for each partition.

[0027] Furthermore, the step of partitioning all the time series data to be managed that is cached in the target storage table specifically includes:

[0028] Determine the time series data to be managed corresponding to each calculation parameter in each partition based on the calculation parameters obtained by pre-partitioning each partition;

[0029] The target storage table is partitioned based on the calculation parameters obtained by pre-partitioning the partitions and the time series data to be managed corresponding to each calculation parameter in each partition.

[0030] Furthermore, before executing the step of obtaining all return values ​​output by the preset table partitioning expression after partitioning processing, the method further includes:

[0031] Get the minimum number of time series data in each partition after partitioning;

[0032] Get the maximum number of time series data in each partition after partitioning;

[0033] Based on the preset partition processing log record file, identify the time series data with the most partition processing times in each partition, and the partition processing times corresponding to the time series data with the most partition processing times;

[0034] The return value output by the current partition is composed of the partition calculation value corresponding to the current partition, the minimum number of time series data in the current partition, the maximum number of time series data in the current partition, and the partition processing count corresponding to the time series data with the most processing times in the current partition. The target data format of the return value is the concatenation of the partition calculation value corresponding to the current partition, the minimum number of time series data in the current partition, the maximum number of time series data in the current partition, and the partition processing count corresponding to the time series data with the most processing times in the current partition. The step of obtaining all return values ​​output by the preset table partition expression after partition processing specifically includes:

[0035] Get the partition calculation value corresponding to each partition, the minimum number of time series data in each partition, the maximum number of time series data in each partition, and the partition processing times corresponding to the time series data with the most processing times in each partition;

[0036] According to the preset splicing symbol, the partition calculation value, the minimum number of the time series data, the maximum number of the time series data, and the partition processing times corresponding to the time series data with the most processing times corresponding to the same partition are spliced ​​to obtain the splicing format data corresponding to each partition;

[0037] The concatenated format data corresponding to each partition is used as the return value of each partition.

[0038] Furthermore, the table structure information of the target data table includes column field information, index field information, and the total number of data rows in the table. The data management file includes a basic management file and a partition management file. The step of generating a data management file in a management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table specifically includes:

[0039] Generating the partition management file according to the partition processing result, wherein the partition management file includes a partition expression file, a partition field maximum value file, and a partition field minimum value file;

[0040] The basic management file is generated according to the table structure information of the target storage table, wherein the basic management file includes a check file, a column information file, a count file, an index file, a column data file, and a column field mark file.

[0041] Furthermore, before executing the step of writing all the time series data to be managed into the corresponding data management file in the form of a data stream, the method further includes:

[0042] Compress each time series data to be managed based on the LZ4 compression algorithm to generate compressed data blocks with the same number as the time series data to be managed. The compressed data blocks are composed of the data to be compressed and the preset header data information. The preset header data information is represented by 9-bit bytes, specifically a 9-bit byte composed of a UInt8 integer sequence and two UInt32 integer sequences. The data to be compressed is the time series data to be managed that is currently being compressed.

[0043] The step of writing all time series data to be managed into the corresponding data management file in the form of a data stream specifically includes:

[0044] According to the column field information in the table corresponding to all the time series data to be managed, write all the time series data to be managed into the column data file in the basic management file;

[0045] The step of writing all time series data to be managed into the corresponding data management file in the form of a data stream also includes:

[0046] According to the storage locations of all the time series data to be managed in the column data file, the compressed data blocks generated according to each time series data to be managed are written to the target location in the column data file, wherein there is a unique correspondence between the storage location corresponding to each time series data to be managed and the target location corresponding to its corresponding compressed data block.

[0047] In order to solve the above technical problems, the present application also provides a time series data file management device, which adopts the following technical solutions:

[0048] A time series data file management device, comprising:

[0049] a table information acquisition module, configured to acquire table name information and table structure information of a target storage table, wherein the target storage table is used to store time series data to be managed;

[0050] A management root directory generation module is used to generate a management root directory in a preset file management server using the table name information of the target storage table as the file directory name;

[0051] A time series data caching module is used to cache the time series data to be managed in real time into the target storage table according to a preset real-time data transmission mode, and obtain a cache storage time, wherein the cache storage time is the time when each time series data to be managed is cached into the target storage table;

[0052] The data partition processing module in the table is used to pass the cache storage time as a calculation parameter into the preset table partition expression, perform partition processing on all the time series data to be managed cached in the target storage table, and obtain the partition processing result;

[0053] A partition processing return value acquisition module is used to obtain all return values ​​output by the preset table partition expression after partition processing;

[0054] A management subdirectory generation module is used to generate management subdirectories corresponding to each partition under the management root directory using all returned values ​​as file directory names;

[0055] A data management file generation module, configured to generate a data management file in a management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table;

[0056] The time series data writing module is used to obtain all the time series data to be managed cached in the target storage table, and write all the time series data to be managed into the corresponding data management file in the form of data stream to complete the file management of the time series data to be managed.

[0057] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0058] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned time series data file management method when executing the computer-readable instructions.

[0059] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0060] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the time series data file management method as described above.

[0061] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0062] The time series data file management method described in the embodiment of the present application obtains the table name information and table structure information of the target storage table, wherein the target storage table is used to store the time series data to be managed; uses the table name information of the target storage table as the file directory name in the preset file management server to generate a management root directory; caches the time series data to be managed in real time into the target storage table according to the preset data real-time transmission method, and obtains the cache entry time, wherein the cache entry time is the time when each time series data to be managed is cached into the target storage table; passes the cache entry time as a calculation parameter into the preset table partition expression, and performs the partitioning of the target storage. All time series data to be managed cached in the storage table are partitioned and processed to obtain the partition processing results; all return values ​​output by the preset table partition expression after partition processing are obtained; under the management root directory, all return values ​​are used as file directory names to generate management subdirectories corresponding to each partition; according to the partition processing results and the table structure information of the target storage table, data management files are generated under the management subdirectories corresponding to each partition; all time series data to be managed cached in the target storage table are obtained, and all time series data to be managed are written into the corresponding data management files in the form of data streams, completing the file management of the time series data to be managed. This method is convenient for improving the compression ratio of time series data, reducing disk space usage, increasing the write rate of time series data, and scientifically managing time series data. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0064] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0065] Figure 2 A flowchart of an embodiment of a method for documenting and managing time series data according to the present application;

[0066] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;

[0067] Figure 4 yes Figure 2 A flowchart of a specific embodiment of step 205 is shown;

[0068] Figure 5 A schematic structural diagram of an embodiment of a time series data file management device according to the present application;

[0069] Figure 6 A schematic structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0070] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0071] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0072] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0073] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing communication links between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0074] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0075] Terminal devices 101, 102, and 103 can be various electronic devices with display screens and support web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV), laptop computers, desktop computers, etc.

[0076] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0077] It should be noted that the time series data file management method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the time series data file management device is generally set in the server / terminal device.

[0078] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0079] Continue to refer Figure 2 , shows a flow chart of an embodiment of a time series data file management method according to the present application. The time series data file management method includes the following steps:

[0080] Step 201: Acquire table name information and table structure information of a target storage table, wherein the target storage table is used to store time series data to be managed.

[0081] Step 202: In a preset file management server, the table name information of the target storage table is used as the file directory name to generate a management root directory.

[0082] Specifically, assuming that the surface information of the target storage table is tb_like, the table name tb_like is directly used as the file directory name of the management root directory in the file manager.

[0083] Step 203: cache the time series data to be managed into the target storage table in real time according to the preset real-time data transmission mode, and obtain the cache entry time, wherein the cache entry time is the time when each time series data to be managed is cached into the target storage table.

[0084] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes:

[0085] Step 301: When each time series data to be managed is cached and stored, the cache storage time is obtained;

[0086] Step 302: Using the cache entry time as a sorting field, sort all time series data to be managed in the order of cache entry time to obtain a sorting result;

[0087] Step 303: All time series data to be managed are cached one by one into the target storage table according to the sorting result.

[0088] The time series data to be managed is sorted by the cache entry time of the time series data to be managed, and is added to the target storage table one by one according to the cache entry time, which avoids storage confusion of the time series data and facilitates scientific management of the time series data.

[0089] In step 204 , the cache entry time is passed as a calculation parameter into a preset table partitioning expression, partitioning is performed on all time series data to be managed that are cached in the target storage table, and a partitioning result is obtained.

[0090] In this embodiment, before executing the step of passing the cache storage time as a calculation parameter into the preset table partitioning expression, the method further includes: setting the number of rows of time series data to be managed in each partition, and using the number of rows of time series data to be managed in each partition as a row number threshold; setting the target data format of the return value, wherein the return value represents the corresponding output result after the table partitioning expression is executed.

[0091] In this embodiment, after executing the step of passing the cache storage time as a calculation parameter into a preset table partition expression, the method further includes:

[0092] Step A: Set the initial number of rows of time series data to be managed in each partition to 0;

[0093] Step B: When the cache entry time of the time series data to be managed is passed into the preset table partition expression as a calculation parameter, the passed-in calculation parameters are obtained in sequence and the number of the passed-in calculation parameters is counted. Each time an passed-in calculation parameter is obtained, the initial row number information is accumulated by 1.

[0094] Step C: until the accumulated result corresponding to the initial number of rows is the same as the row number threshold corresponding to the target partition, the pre-partitioning process for the target partition is completed;

[0095] Step D: Repeat steps B to C to complete the pre-partitioning process for each partition.

[0096] Through pre-partitioning processing, a prior basis is provided for partitioning the time series data to be managed, so that when the final partitioning is performed later, the time series data to be managed can be partitioned according to the pre-partitioning processing results, which facilitates scientific management of time series data.

[0097] In this embodiment, the step of partitioning all the time series data to be managed cached in the target storage table specifically includes: determining the time series data to be managed corresponding to each calculation parameter in each partition based on the calculation parameters obtained by pre-partitioning the partitions; and partitioning the target storage table based on the calculation parameters obtained by pre-partitioning the partitions and the time series data to be managed corresponding to each calculation parameter in each partition.

[0098] Step 205: Obtain all return values ​​output by the preset table partitioning expression after partitioning processing.

[0099] In this embodiment, before executing the step of obtaining all return values ​​output by the preset table partitioning expression after partition processing, the method further includes: obtaining the minimum number of time series data in each partition after partition processing; obtaining the maximum number of time series data in each partition after partition processing; based on a preset partition processing log record file, identifying the time series data with the most partition processing times in each partition, and the number of partition processing times corresponding to the time series data with the most partition processing times.

[0100] In this embodiment, the return value output by the current partition is composed of the partition calculation value corresponding to the current partition, the minimum number of the time series data in the current partition, the maximum number of the time series data in the current partition, and the partition processing times corresponding to the time series data with the most processing times in the current partition.

[0101] Specifically, the target data format of the return value is the concatenation of the partition calculation value corresponding to the current partition, the minimum number of the time series data in the current partition, the maximum number of the time series data in the current partition, and the partition processing times corresponding to the time series data with the most processing times in the current partition.

[0102] Continue to refer Figure 4 , Figure 4 yes Figure 2 The flowchart of a specific embodiment of step 205 shown includes:

[0103] Step 401: Obtain the partition calculation value corresponding to each partition, the minimum number of time series data in each partition, the maximum number of time series data in each partition, and the partition processing times corresponding to the time series data with the most processing times in each partition;

[0104] Step 402: Splice the partition calculation value, the minimum number of the time series data, the maximum number of the time series data, and the partition processing times corresponding to the time series data with the most processing times corresponding to the same partition according to a preset splicing symbol to obtain splicing format data corresponding to each partition;

[0105] Step 403: Use the splicing format data corresponding to each partition as the return value of each partition.

[0106] Specifically, the generation rules of the return value of each partition are as follows: the partition calculation value corresponding to the current partition_the minimum number of time series data in the current partition_the maximum number of time series data in the current partition_the partition processing times corresponding to the time series data with the most processing times in the current partition. Among them, the partition calculation value corresponding to the current partition is the value directly calculated using the partition expression, for example: the partition expression is toYYYYMM(timestamp), where timestamp represents the cache entry time of the time series data to be managed, that is, the calculation parameter. There is a column in the accessed data named timestamp and the value is 2022-12-01, then the partition calculation value corresponding to the current partition is 202212; the minimum number of time series data in the current partition represents the minimum ID value of the time series data cached in the current partition, for example, the minimum number of time series data in the current partition is 11; the maximum number of time series data in the current partition represents the current partition. The maximum ID value of the time series data cached in the partition. For example, the maximum number of time series data in the current partition is 22. The partition processing count corresponding to the time series data with the most processing times in the current partition indicates the maximum number of times the time series data in the current partition has been merged. The initial value is 0. The unit is partition. If different partitions are merged, the merge count of all time series data in the new partition formed after the merger is cumulatively added by 1. For example, if the maximum number of a certain time series data is 9, then the number of times the time series data has been merged is 9, that is, it has been merged from different partitions, that is, there are 10 partitions merged, and the final return value of the current partition is: 202212_11_22_9.

[0107] Similarly, the generation rule of the return value of each partition can also be: the partition calculation value corresponding to the current partition_the minimum number of the previous partition merged by the current partition_the maximum number of the previous partition merged by the current partition_the partition processing times corresponding to the time series data with the most processing times in the current partition. Obviously, the partition calculation value corresponding to the current partition is still the value calculated directly using the partition expression. For example, the partition expression is toYYYYMM(timestamp), where timestamp represents the cache storage time of the time series data to be managed, that is, the calculation parameter. There is a column named timestamp in the accessed data and its value is 2022-12-01. The partition calculation value corresponding to the current partition is 202212. The minimum number of the previous partition merged by the current partition and the maximum number of the previous partition merged by the current partition are no longer represented by the number of the time series data in the current partition, but by the partition number. For example, the current partition is formed by merging the previous partitions, where the partition numbers of the previous partitions include partition 1, partition 3, partition 4, and partition 8. The current partition is formed by merging 4 previous partitions. The minimum number of the previous partition merged by the current partition is 1, and the maximum number of the previous partition merged by the current partition is 8. Assuming that the current partition is formed by merging two of the four previous partitions and then synthesizing them, the number of partition processing times corresponding to the time series data with the most processing times in the current partition is 3, that is, one of the previous partitions is the first partition processing, the middle partition formed by merging the two partitions is the second partition processing, and the current partition formed by the synthesis is the third partition processing. The final return value of the current partition is: 202212_1_8_3; assuming that the current partition is formed by merging one of the four previous partitions with other partitions one by one, the number of partition processing times corresponding to the time series data with the most processing times in the current partition is 4, that is, one of the previous partitions is partition 1, the first partition processing is performed, and then the middle partition formed by merging partition 1 with partition 3 is the second partition processing, and then partition 4 is synthesized to form the middle partition for the third partition processing. Finally, partition 8 is synthesized to form the current partition for the fourth partition processing. The final return value of the current partition is: 202212_1_8_4.

[0108] Step 206: Under the management root directory, all returned values ​​are used as file directory names to generate management subdirectories corresponding to each partition.

[0109] Assume that the return value of the current partition is 202212_1_8_4, then 202212_1_8_4 is used as the file directory name of the management subdirectory corresponding to the current partition.

[0110] The file directory name of the management subdirectory is set according to the return value of each partition. This makes the naming of the file directory associated with the cache storage time of the time series data and the partition processing results. It also ensures the uniformity and standardization of the file directory name, facilitating the scientific management of time series data in the future.

[0111] Step 207: Generate a data management file in the management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table.

[0112] In this embodiment, the table structure information of the target data table includes column field information, index field information, and the total number of data rows in the table.

[0113] In this embodiment, the data management file includes a basic management file and a partition management file.

[0114] In this embodiment, the step of generating a corresponding data management file in the management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table specifically includes: generating the partition management file according to the partition processing result, wherein the partition management file includes a partition expression file, a partition field maximum value file, and a partition field minimum value file; generating the basic management file according to the table structure information of the target storage table, wherein the basic management file includes a check file, a column information file, a count file, an index file, a column data file, and a column field mark file.

[0115] Specifically, the verification file stores the sizes of other files and the hash relationship between the file names and file sizes, which is used to quickly verify the integrity and correctness of the files. For example, if the size of the count file is 20kb, the verification file will store the content [count file: 20kb].

[0116] The column information file stores all column field information of the time series data. The count file stores the total number of data rows in the current partition directory. The index file is used to store sparse indexes. The column data file stores the data of a certain column in a compressed format. The column field file tag file stores the offset information of the data in the column data file. The partition expression file stores the partition calculation value corresponding to the partition expression under the current partition. The partition field minimum value file and partition field maximum value file store the minimum and maximum numbers of the time series data in the current partition. At the same time, they also store the minimum and maximum partition numbers of the previous partition after the current partition is merged.

[0117] By generating corresponding data management files and using them to represent the data in the corresponding target storage table, managers only need to use the data management files to maintain and manage time series data in the future, without having to check line by line through the target storage table. This simplifies the maintenance process and reduces the maintenance workload to a certain extent, making it easier to manage time series data scientifically.

[0118] Step 208: Acquire all the time series data to be managed that are cached in the target storage table, and write all the time series data to be managed into the corresponding data management file in the form of a data stream, thereby completing the file management of the time series data to be managed.

[0119] In this embodiment, before executing the step of writing all the time series data to be managed into the corresponding data management file in the form of a data stream, the method also includes: compressing each time series data to be managed based on the LZ4 compression algorithm to generate compressed data blocks with the same number as the time series data to be managed, wherein the compressed data blocks are compressed by the data to be compressed and the preset header data information. The preset header data information is represented by 9-bit bytes, specifically composed of 1 UInt8 integer sequence and 2 UInt32 integer sequences to form 9-bit bytes. The data to be compressed is the time series data to be managed that is currently to be compressed.

[0120] Through compression processing, the storage size of data in data management files is reduced, the resource storage consumption of the server is reduced, the compression ratio of time series data is improved, the use of disk space is reduced, the writing rate of time series data is increased, and time series data is managed scientifically.

[0121] In this embodiment, the step of writing all the time series data to be managed into the corresponding data management file in the form of a data stream specifically includes: writing all the time series data to be managed into the column data file in the basic management file according to the column field information in the table corresponding to all the time series data to be managed.

[0122] In this embodiment, the step of writing all the time series data to be managed into the corresponding data management file in the form of a data stream also includes: according to the storage position of all the time series data to be managed in the column data file, writing the compressed data blocks generated according to each time series data to be managed into the target position in the column data file, wherein there is a unique correspondence between the storage position corresponding to each time series data to be managed and the target position corresponding to its corresponding compressed data block.

[0123] Specifically, all time series data to be managed are written into the corresponding data management file in the form of data streams, following the preset writing rules, wherein the preset writing rules include: calculating the length of the written data in advance according to a fixed granularity, and setting a separator in the written data based on the preset interval length, thereby ensuring that there is an interval length when the data is written to the file, realizing hierarchical and standardized writing, avoiding the overall writing of data without intervals, and scientifically managing the time series data.

[0124] The present application obtains the table name information and table structure information of the target storage table, wherein the target storage table is used to store the time series data to be managed; uses the table name information of the target storage table as the file directory name in the preset file management server to generate a management root directory; caches the time series data to be managed in real time into the target storage table according to the preset data real-time transmission method, and obtains the cache entry time, wherein the cache entry time is the time when each time series data to be managed is cached into the target storage table; passes the cache entry time as a calculation parameter into the preset table partition expression, and performs a partitioning operation on all the cached data in the target storage table. The time series data to be managed is partitioned and the partitioning processing results are obtained; all return values ​​output by the preset table partitioning expression after the partitioning processing are obtained; under the management root directory, all return values ​​are used as file directory names to generate management subdirectories corresponding to each partition; according to the partitioning processing results and the table structure information of the target storage table, data management files are generated under the management subdirectories corresponding to each partition; all time series data to be managed cached in the target storage table are obtained, and all time series data to be managed are written into the corresponding data management files in the form of data streams, completing the file management of the time series data to be managed. This method is convenient for improving the compression ratio of time series data, reducing disk space usage, increasing the write rate of time series data, and scientifically managing time series data.

[0125] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0126] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0127] In the embodiments of the present application, it involves big data processing technology in artificial intelligence. Time series data is easily excessive in amount due to its high generation frequency, dependence on acquisition time, large number of measurement points and large amount of information, which makes management difficult. The present application facilitates improving the compression ratio of time series data, reducing disk space usage, increasing the writing rate of time series data, and scientifically managing time series data through partitioning processing, compression processing, and file management. Accordingly, the execution program corresponding to the time series data file management method described in the present application can be encapsulated as a processing plug-in, and new time series data can be processed through the plug-in, reducing manual workload and being more automated and intelligent.

[0128] Further references Figure 5 , as a response to the above Figure 2 The present application provides an embodiment of a time series data file management device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0129] like Figure 5 As shown, the time series data file management device 500 described in this embodiment includes: a table information acquisition module 501, a management root directory generation module 502, a time series data cache storage module 503, a table data partition processing module 504, a partition processing return value acquisition module 505, a management subdirectory generation module 506, a data management file generation module 507 and a time series data writing module 508.

[0130] A table information acquisition module 501 is used to acquire table name information and table structure information of a target storage table, wherein the target storage table is used to store time series data to be managed;

[0131] A management root directory generating module 502 is configured to generate a management root directory in a preset file management server using the table name information of the target storage table as the file directory name;

[0132] The time series data caching module 503 is used to cache the time series data to be managed in real time into the target storage table according to a preset real-time data transmission mode, and obtain the cache storage time, where the cache storage time is the time when each time series data to be managed is cached into the target storage table;

[0133] The in-table data partition processing module 504 is used to pass the cache entry time as a calculation parameter into a preset table partition expression, perform partition processing on all time series data to be managed cached in the target storage table, and obtain partition processing results;

[0134] The partition processing return value acquisition module 505 is used to obtain all return values ​​output by the preset table partition expression after partition processing;

[0135] A management subdirectory generation module 506 is configured to generate a management subdirectory corresponding to each partition under the management root directory using all returned values ​​as file directory names;

[0136] A data management file generating module 507 is configured to generate a data management file in a management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table;

[0137] The time series data writing module 508 is used to obtain all the time series data to be managed cached in the target storage table, and write all the time series data to be managed into the corresponding data management file in the form of data stream to complete the file management of the time series data to be managed.

[0138] In some specific embodiments of the present application, the time series data file management device 500 further includes a pre-partitioning processing module, which is used to set the initial number of rows of the time series data to be managed in each partition to 0; when the cache entry time of the time series data to be managed is passed into the preset table partitioning expression as a calculation parameter, the passed-in calculation parameters are obtained in sequence, and the number of the passed-in calculation parameters is counted, wherein each time an passed-in calculation parameter is obtained, the initial number of rows information is accumulated by 1; until the accumulated result corresponding to the initial number of rows information is the same as the row number threshold corresponding to the target partition, the pre-partitioning processing of the target partition is completed; repeat the above steps to complete the pre-partitioning processing of each partition.

[0139] In some specific embodiments of the present application, the time series data filing management device 500 also includes a data compression module, which is used to compress each time series data to be managed based on the LZ4 compression algorithm to generate compressed data blocks with the same number as the time series data to be managed, wherein the compressed data blocks are compressed by the data to be compressed and the preset header data information. The preset header data information is represented by 9-bit bytes, specifically composed of 1 UInt8 integer sequence and 2 UInt32 integer sequences to form a 9-bit byte. The data to be compressed is the time series data to be managed that is currently to be compressed.

[0140] The present application obtains the table name information and table structure information of the target storage table, wherein the target storage table is used to store the time series data to be managed; uses the table name information of the target storage table as the file directory name in the preset file management server to generate a management root directory; caches the time series data to be managed in real time into the target storage table according to the preset data real-time transmission method, and obtains the cache entry time, wherein the cache entry time is the time when each time series data to be managed is cached into the target storage table; passes the cache entry time as a calculation parameter into the preset table partition expression, and performs a partitioning operation on all the cached data in the target storage table. The time series data to be managed is partitioned and the partitioning processing results are obtained; all return values ​​output by the preset table partitioning expression after the partitioning processing are obtained; under the management root directory, all return values ​​are used as file directory names to generate management subdirectories corresponding to each partition; according to the partitioning processing results and the table structure information of the target storage table, data management files are generated under the management subdirectories corresponding to each partition; all time series data to be managed cached in the target storage table are obtained, and all time series data to be managed are written into the corresponding data management files in the form of data streams, completing the file management of the time series data to be managed. This method is convenient for improving the compression ratio of time series data, reducing disk space usage, increasing the write rate of time series data, and scientifically managing time series data.

[0141] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0142] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0143] To solve the above technical problems, the present application also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0144] The computer device 6 includes a memory 6a, a processor 6b, and a network interface 6c that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 having components 6a-6c, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0145] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0146] The memory 6a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 6a can be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 6a can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 6a can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 6a is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the time series data file management method. In addition, the memory 6a can also be used to temporarily store various types of data that have been output or are to be output.

[0147] In some embodiments, the processor 6b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 6b is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 6b is used to execute computer-readable instructions stored in the memory 6a or process data, such as executing computer-readable instructions for the time series data file management method.

[0148] The network interface 6c may include a wireless network interface or a wired network interface. The network interface 6c is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0149] The computer device proposed in this embodiment belongs to the field of data management technology. This application obtains the table name information and table structure information of the target storage table, wherein the target storage table is used to store the time series data to be managed; uses the table name information of the target storage table as the file directory name in the preset file management server to generate a management root directory; according to the preset data real-time transmission method, caches the time series data to be managed in real time into the target storage table, and obtains the cache entry time, wherein the cache entry time is the time when each time series data to be managed is cached into the target storage table; passes the cache entry time as a calculation parameter into the preset table partition expression, and performs a calculation on all cached data in the target storage table. The time series data to be managed is partitioned and the partitioning processing results are obtained; all return values ​​output by the preset table partitioning expression after the partitioning processing are obtained; under the management root directory, all return values ​​are used as file directory names to generate management subdirectories corresponding to each partition; according to the partitioning processing results and the table structure information of the target storage table, data management files are generated under the management subdirectories corresponding to each partition; all time series data to be managed cached in the target storage table are obtained, and all time series data to be managed are written into the corresponding data management files in the form of data streams, completing the file management of the time series data to be managed. This method is convenient for improving the compression ratio of time series data, reducing disk space usage, increasing the write rate of time series data, and scientifically managing time series data.

[0150] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the time series data file management method as described above.

[0151] The computer-readable storage medium proposed in this embodiment belongs to the field of data management technology. This application obtains the table name information and table structure information of the target storage table, wherein the target storage table is used to store the time series data to be managed; uses the table name information of the target storage table as the file directory name in the preset file management server to generate a management root directory; according to the preset data real-time transmission method, caches the time series data to be managed in real time into the target storage table, and obtains the cache entry time, wherein the cache entry time is the time when each time series data to be managed is cached into the target storage table; passes the cache entry time as a calculation parameter into the preset table partition expression, and performs a calculation on all cached data in the target storage table. The time series data to be managed is partitioned and the partitioning processing results are obtained; all return values ​​output by the preset table partitioning expression after the partitioning processing are obtained; under the management root directory, all return values ​​are used as file directory names to generate management subdirectories corresponding to each partition; according to the partitioning processing results and the table structure information of the target storage table, data management files are generated under the management subdirectories corresponding to each partition; all time series data to be managed cached in the target storage table are obtained, and all time series data to be managed are written into the corresponding data management files in the form of data streams, completing the file management of the time series data to be managed. This method is convenient for improving the compression ratio of time series data, reducing disk space usage, increasing the write rate of time series data, and scientifically managing time series data.

[0152] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0153] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A method for documenting and managing time series data, characterized in that: The steps include: Obtaining table name information and table structure information of a target storage table, wherein the target storage table is used to store time series data to be managed; In a preset file management server, the table name information of the target storage table is used as the file directory name to generate a management root directory; According to a preset real-time data transmission mode, the time series data to be managed is cached in real time into the target storage table, and a cache entry time is obtained, wherein the cache entry time is the time when each time series data to be managed is cached into the target storage table; The cache entry time is passed as a calculation parameter into a preset table partitioning expression, and partition processing is performed on all time series data to be managed cached in the target storage table to obtain the partition processing result; Obtain all return values ​​output by the preset table partition expression after partition processing; Under the management root directory, all returned values ​​are used as file directory names to generate management subdirectories corresponding to each partition; Generate a data management file in a management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table; All the time series data to be managed cached in the target storage table are obtained, and all the time series data to be managed are written into the corresponding data management file in the form of data stream, thereby completing the file management of the time series data to be managed.

2. The time series data file management method according to claim 1, characterized in that: The step of caching the time series data to be managed into the target storage table in real time according to the preset real-time data transmission mode specifically includes: When each time series data to be managed is cached and stored, the cache storage time is obtained; Using the cache entry time as a sorting field, sort all time series data to be managed in the order of cache entry time to obtain a sorting result; All time series data to be managed are cached one by one into the target storage table according to the sorting results.

3. The time series data file management method according to claim 1, characterized in that: Before executing the step of passing the cache storage time as a calculation parameter into a preset table partition expression, the method further includes: Set the number of rows of time series data to be managed in each partition, and use the number of rows of time series data to be managed in each partition as the row number threshold; Setting the target data format of the return value, wherein the return value represents the corresponding output result after the table partition expression is executed; After executing the step of passing the cache storage time as a calculation parameter into a preset table partition expression, the method further includes: Step A: Set the initial number of rows of time series data to be managed in each partition to 0; Step B: When the cache entry time of the time series data to be managed is passed into the preset table partition expression as a calculation parameter, the passed-in calculation parameters are obtained in sequence and the number of the passed-in calculation parameters is counted. Each time an passed-in calculation parameter is obtained, the initial row number information is accumulated by 1. Step C: until the accumulated result corresponding to the initial number of rows is the same as the row number threshold corresponding to the target partition, the pre-partitioning process for the target partition is completed; Step D: Repeat steps B to C to complete the pre-partitioning process for each partition.

4. The method for managing time series data files according to claim 3, wherein: The step of partitioning all the time series data to be managed that is cached in the target storage table specifically includes: Determine the time series data to be managed corresponding to each calculation parameter in each partition based on the calculation parameters obtained by pre-partitioning each partition; The target storage table is partitioned based on the calculation parameters obtained by pre-partitioning the partitions and the time series data to be managed corresponding to each calculation parameter in each partition.

5. The time series data file management method according to claim 4, characterized in that: Before executing the step of obtaining all return values ​​output by the preset table partitioning expression after partitioning processing, the method further includes: Get the minimum number of time series data in each partition after partitioning; Get the maximum number of time series data in each partition after partitioning; Based on the preset partition processing log record file, identify the time series data with the most partition processing times in each partition, and the partition processing times corresponding to the time series data with the most partition processing times; The return value output by the current partition is composed of the partition calculation value corresponding to the current partition, the minimum number of time series data in the current partition, the maximum number of time series data in the current partition, and the partition processing count corresponding to the time series data with the most processing times in the current partition. The target data format of the return value is the concatenation of the partition calculation value corresponding to the current partition, the minimum number of time series data in the current partition, the maximum number of time series data in the current partition, and the partition processing count corresponding to the time series data with the most processing times in the current partition. The step of obtaining all return values ​​output by the preset table partition expression after partition processing specifically includes: Get the partition calculation value corresponding to each partition, the minimum number of time series data in each partition, the maximum number of time series data in each partition, and the partition processing times corresponding to the time series data with the most processing times in each partition; According to the preset splicing symbol, the partition calculation value, the minimum number of the time series data, the maximum number of the time series data, and the partition processing times corresponding to the time series data with the most processing times corresponding to the same partition are spliced ​​to obtain the splicing format data corresponding to each partition; The concatenated format data corresponding to each partition is used as the return value of each partition.

6. The method for managing time series data files according to any one of claims 1 to 5, characterized in that: The table structure information of the target data table includes column field information, index field information, and the total number of data rows in the table. The data management file includes a basic management file and a partition management file. The step of generating a data management file in a management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table specifically includes: Generating the partition management file according to the partition processing result, wherein the partition management file includes a partition expression file, a partition field maximum value file, and a partition field minimum value file; The basic management file is generated according to the table structure information of the target storage table, wherein the basic management file includes a check file, a column information file, a count file, an index file, a column data file, and a column field mark file.

7. The method for managing time series data files according to claim 6, wherein: Before executing the step of writing all the time series data to be managed into the corresponding data management file in the form of a data stream, the method further includes: Compress each time series data to be managed based on the LZ4 compression algorithm to generate compressed data blocks with the same number as the time series data to be managed. The compressed data blocks are composed of the data to be compressed and the preset header data information. The preset header data information is represented by 9-bit bytes, specifically a 9-bit byte composed of a UInt8 integer sequence and two UInt32 integer sequences. The data to be compressed is the time series data to be managed that is currently being compressed. The step of writing all time series data to be managed into the corresponding data management file in the form of a data stream specifically includes: According to the column field information in the table corresponding to all the time series data to be managed, write all the time series data to be managed into the column data file in the basic management file; The step of writing all time series data to be managed into the corresponding data management file in the form of a data stream also includes: According to the storage locations of all the time series data to be managed in the column data file, the compressed data blocks generated according to each time series data to be managed are written to the target location in the column data file, wherein there is a unique correspondence between the storage location corresponding to each time series data to be managed and the target location corresponding to its corresponding compressed data block.

8. A time series data file management device, characterized in that: include: a table information acquisition module, configured to acquire table name information and table structure information of a target storage table, wherein the target storage table is used to store time series data to be managed; A management root directory generation module is used to generate a management root directory in a preset file management server using the table name information of the target storage table as the file directory name; A time series data caching module is used to cache the time series data to be managed in real time into the target storage table according to a preset real-time data transmission mode, and obtain a cache storage time, wherein the cache storage time is the time when each time series data to be managed is cached into the target storage table; The data partition processing module in the table is used to pass the cache storage time as a calculation parameter into the preset table partition expression, perform partition processing on all the time series data to be managed cached in the target storage table, and obtain the partition processing result; A partition processing return value acquisition module is used to obtain all return values ​​output by the preset table partition expression after partition processing; A management subdirectory generation module is used to generate management subdirectories corresponding to each partition under the management root directory using all returned values ​​as file directory names; A data management file generation module, configured to generate a data management file in a management subdirectory corresponding to each partition according to the partition processing result and the table structure information of the target storage table; The time series data writing module is used to obtain all the time series data to be managed cached in the target storage table, and write all the time series data to be managed into the corresponding data management file in the form of data stream to complete the file management of the time series data to be managed.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the time series data file management method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the time series data file management method according to any one of claims 1 to 7.

Images