Self-adaptive space pre-distribution method suitable for time sequence storage
By collecting statistical information in the timing data storage, calculating the estimated write data amount and file size, and dynamically creating data files, the problem of wasted storage space in the existing technology is solved, and more efficient data storage and write performance is achieved.
Patent Information
- Application Number
- CN202510059991.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-15
- Publication Date
- 2025-05-13
AI Technical Summary
In the prior art, the pre-allocated storage space in time-series data storage is unreasonable, resulting in waste of disk space, affecting data storage capacity and write performance.
By collecting statistical information of time series data, calculate the total amount of data expected to be written in the current time partition, calculate the file size that needs to be pre-allocated based on the estimated amount of data written and field information, dynamically create data files of corresponding sizes to achieve adaptive space pre-allocation.
It effectively reduces the waste of disk space, reasonably occupies file size, avoids frequent expansion of data files, improves write performance, and can store more data.
Smart Images
Figure CN119987662A_ABST
Abstract
Description
Technical Field
[0001] The invention discloses an adaptive space pre-allocation method suitable for time series storage, and relates to the technical field of storage space management. Background Art
[0002] With the rapid development of sensor and data acquisition technology, time series data is increasingly used in various fields. Time series data usually records the state changes of a system or phenomenon at different time points. As the amount of data increases, the storage cost is also increasing. For example, a certain amount of space will be pre-allocated before writing data to solve the performance problems caused by frequent expansion during the writing process. However, this also brings another problem. If the pre-allocated space is unreasonable or not fully used, it is easy to waste disk space. Summary of the invention
[0003] In view of the problems of the prior art, the present invention provides an adaptive space pre-allocation method suitable for time series storage, which adaptively pre-allocates storage space, effectively reduces the waste of disk space, and tries to store more data.
[0004] The specific scheme proposed by the present invention is:
[0005] The present invention provides an adaptive space pre-allocation method suitable for time series storage, comprising:
[0006] Step 1: Collect statistics of timing data: Collect statistics of timing data based on the amount of data written to the timing table and the number of devices.
[0007] Step 2: Obtain the total amount of data written to the current time partition and the number of devices based on the statistical information, and calculate the total amount of data expected to be written to the current time partition.
[0008] Obtain field information based on the current time series table, calculate the file size that needs to be pre-allocated based on the expected total amount of written data, and create a data file of the corresponding size based on the pre-allocated file size.
[0009] Furthermore, in step 1 of the adaptive space pre-allocation method for time series storage, a timed scheduling task is created, and statistical information of all time series tables in the time series database is collected at a preset time every day through the timed scheduling task.
[0010] Furthermore, in step 2 of the adaptive space pre-allocation method applicable to time series storage, calculating the total amount of data expected to be written to the current time partition includes:
[0011] Get the number of devices saved in the current time partition and calculate the total amount of data expected to be written to the current time partition in one day. The formula is as follows:
[0012] The estimated amount of data written in the current time partition in one day = the amount of data written from the preset time point of the previous day to the preset time point of today * the number of devices in the current time partition / the total number of devices,
[0013] Using the estimated amount of data written in the current time partition in one day, calculate the estimated total amount of data written in the current time partition. The formula is as follows:
[0014] The total amount of data expected to be written to the current time partition = the amount of data expected to be written to the current time partition in one day * the partition time interval, where the partition time interval is the estimated number of days.
[0015] Furthermore, in step 2 of the adaptive space pre-allocation method applicable to time series storage, the file size to be pre-allocated is calculated according to the total amount of data expected to be written, including:
[0016] Get the field information of the current time series table and calculate the number of bytes occupied by a row of data using the following formula:
[0017] Pre-allocated file size = the total amount of data expected to be written to the current time partition * the number of bytes occupied by a row of data,
[0018] Calculate the file size that needs to be preallocated.
[0019] The present invention also provides an adaptive space pre-allocation device suitable for time series storage, comprising a time series data statistical information collection module and a time series data file management module.
[0020] The timing data statistics collection module collects timing data statistics: collects timing data statistics based on the amount of data written to the timing table and the number of devices.
[0021] The time series data file management module obtains the total amount of data written to the current time partition and the number of devices based on statistical information, and calculates the total amount of data expected to be written to the current time partition.
[0022] Obtain field information based on the current time series table, calculate the file size that needs to be pre-allocated based on the expected total amount of written data, and create a data file of the corresponding size based on the pre-allocated file size.
[0023] Furthermore, the time series data statistical information collection module of the adaptive space pre-allocation device suitable for time series storage creates a timed scheduling task, and collects statistical information of all time series tables in the time series database at a preset time every day through the timed scheduling task.
[0024] Furthermore, the time series data file management module of the adaptive space pre-allocation device for time series storage calculates the total amount of data expected to be written into the current time partition, including:
[0025] Get the number of devices saved in the current time partition and calculate the total amount of data expected to be written to the current time partition in one day. The formula is as follows:
[0026] The estimated amount of data written in the current time partition in one day = the amount of data written from the preset time point of the previous day to the preset time point of today * the number of devices in the current time partition / the total number of devices,
[0027] Using the estimated amount of data written in the current time partition in one day, calculate the estimated total amount of data written in the current time partition. The formula is as follows:
[0028] The total amount of data expected to be written to the current time partition = the amount of data expected to be written to the current time partition in one day * the partition time interval, where the partition time interval is the estimated number of days.
[0029] Furthermore, the time series data file management module of the adaptive space pre-allocation device for time series storage calculates the file size to be pre-allocated according to the total amount of data expected to be written, including:
[0030] Get the field information of the current time series table and calculate the number of bytes occupied by a row of data using the following formula:
[0031] Pre-allocated file size = the total amount of data expected to be written to the current time partition * the number of bytes occupied by a row of data,
[0032] Calculate the file size that needs to be preallocated.
[0033] The benefits of the present invention are:
[0034] Adaptively pre-allocate storage space, reasonably occupy file size, effectively reduce disk space waste, avoid frequent expansion of data files, try to store more data, and improve write performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 It is a schematic diagram of the flow of collecting statistical information by the method of the present invention.
[0036] Figure 2 The present invention is a schematic diagram of the process flow of creating a pre-allocated file size by the method of the present invention. DETAILED DESCRIPTION
[0037] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.
[0038] Example 1
[0039] The present invention provides an adaptive space pre-allocation method suitable for time series storage, comprising:
[0040] Step 1: Collect statistical information of time series data: Collect statistical information of time series data based on the amount of data written to the time series table and the number of devices.
[0041] A timed scheduling task can be further created, which will call the statistical information collection interface for all time series tables in the database at 23:00 every day.
[0042] Call the time series table statistics collection interface to obtain the amount of data written between 23:00 the previous day and 23:00 today and the total number of devices saved in the current time series table.
[0043] Step 2: Obtain the total amount of data written to the current time partition and the number of devices based on the statistical information, and calculate the total amount of data expected to be written to the current time partition.
[0044] Obtain field information based on the current time series table, calculate the file size that needs to be pre-allocated based on the expected total amount of written data, and create a data file of the corresponding size based on the pre-allocated file size.
[0045] The calculation of the total amount of data expected to be written to the current time partition in step 2 may further include:
[0046] Get the number of devices saved in the current time partition and calculate the total amount of data expected to be written to the current time partition in one day. The formula is as follows:
[0047] The estimated amount of data written in the current time partition in one day = the amount of data written from 23:00 the previous day to 23:00 today * the number of devices in the current time partition / the total number of devices,
[0048] Using the estimated amount of data written in the current time partition in one day, calculate the estimated total amount of data written in the current time partition. The formula is as follows:
[0049] The total amount of data expected to be written to the current time partition = the amount of data expected to be written to the current time partition in one day * the partition time interval, where the partition time interval is the estimated number of days.
[0050] Calculating the file size that needs to be pre-allocated based on the total amount of data expected to be written may further include:
[0051] Get the field information of the current time series table and calculate the number of bytes occupied by a row of data using the following formula:
[0052] Pre-allocated file size = the total amount of data expected to be written to the current time partition * the number of bytes occupied by a row of data,
[0053] Calculate the file size that needs to be preallocated.
[0054] Example 2
[0055] The present invention also provides an adaptive space pre-allocation device suitable for time series storage, comprising a time series data statistical information collection module and a time series data file management module.
[0056] The timing data statistics collection module collects timing data statistics: collects timing data statistics based on the amount of data written to the timing table and the number of devices.
[0057] The time series data file management module obtains the total amount of data written to the current time partition and the number of devices based on statistical information, and calculates the total amount of data expected to be written to the current time partition.
[0058] Obtain field information based on the current time series table, calculate the file size that needs to be pre-allocated based on the expected total amount of written data, and create a data file of the corresponding size based on the pre-allocated file size.
[0059] As the information interaction and execution process between the modules in the above-mentioned device are based on the same concept as the embodiment of the method of the present invention, the specific contents can be found in the description of the embodiment of the method of the present invention and will not be repeated here.
[0060] Likewise, the benefits of the device of the present invention are:
[0061] Adaptively pre-allocate storage space, reasonably occupy file size, effectively reduce disk space waste, avoid frequent expansion of data files, try to store more data, and improve write performance.
[0062] It should be noted that not all steps and modules in the above-mentioned processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structure described in the above-mentioned embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities, or some components in multiple independent devices may be implemented together.
[0063] The above-described embodiments are only preferred embodiments for fully illustrating the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or changes made by those skilled in the art based on the present invention are within the protection scope of the present invention. The protection scope of the present invention shall be subject to the claims.
Claims
1. An adaptive space pre-allocation method suitable for time series storage, characterized in that include: Step 1: Collect statistics of timing data: Collect statistics of timing data based on the amount of data written to the timing table and the number of devices. Step 2: Obtain the total amount of data written to the current time partition and the number of devices based on the statistical information, and calculate the total amount of data expected to be written to the current time partition. Obtain field information based on the current time series table, calculate the file size that needs to be pre-allocated based on the expected total amount of written data, and create a data file of the corresponding size based on the pre-allocated file size.
2. The method for adaptive space pre-allocation for time series storage according to claim 1, characterized in that In step 1, a scheduled task is created to collect statistical information of all time series tables in the time series database at a preset time every day.
3. The method for adaptive space pre-allocation for time series storage according to claim 1, characterized in that In step 2, the total amount of data expected to be written to the current time partition is calculated, including: Get the number of devices saved in the current time partition and calculate the total amount of data expected to be written to the current time partition in one day. The formula is as follows: The estimated amount of data written in the current time partition in one day = the amount of data written from the preset time point of the previous day to the preset time point of today * the number of devices in the current time partition / the total number of devices, Using the estimated amount of data written in the current time partition in one day, calculate the estimated total amount of data written in the current time partition. The formula is as follows: The total amount of data expected to be written to the current time partition = the amount of data expected to be written to the current time partition in one day * the partition time interval, where the partition time interval is the estimated number of days.
4. The method for adaptive space pre-allocation for time series storage according to claim 1, characterized in that In step 2, the file size that needs to be pre-allocated is calculated based on the total amount of data expected to be written, including: Get the field information of the current time series table and calculate the number of bytes occupied by a row of data using the following formula: Pre-allocated file size = the total amount of data expected to be written to the current time partition * the number of bytes occupied by a row of data, Calculate the file size that needs to be preallocated.
5. An adaptive space pre-allocation device suitable for time series storage, characterized in that It includes a time series data statistical information collection module and a time series data file management module. The timing data statistics collection module collects timing data statistics: collects timing data statistics based on the amount of data written to the timing table and the number of devices. The time series data file management module obtains the total amount of data written to the current time partition and the number of devices based on statistical information, and calculates the total amount of data expected to be written to the current time partition. Obtain field information based on the current time series table, calculate the file size that needs to be pre-allocated based on the expected total amount of written data, and create a data file of the corresponding size based on the pre-allocated file size.
6. The adaptive space pre-allocation device for time series storage according to claim 5, characterized in that the time series The data statistics information collection module creates a scheduled task, which collects statistics for all time series tables in the time series database at a preset time every day.
7. The adaptive space pre-allocation device for time series storage according to claim 5, characterized in that The time series data file management module calculates the total amount of data expected to be written to the current time partition, including: Get the number of devices saved in the current time partition and calculate the total amount of data expected to be written to the current time partition in one day. The formula is as follows: The estimated amount of data written in the current time partition in one day = the amount of data written from the preset time point of the previous day to the preset time point of today * the number of devices in the current time partition / the total number of devices, Using the estimated amount of data written in the current time partition in one day, calculate the estimated total amount of data written in the current time partition. The formula is as follows: The total amount of data expected to be written to the current time partition = the amount of data expected to be written to the current time partition in one day * the partition time interval, where the partition time interval is the estimated number of days.
8. The adaptive space pre-allocation device for time series storage according to claim 5, characterized in that The time series data file management module calculates the file size that needs to be pre-allocated based on the total amount of data expected to be written, including: Get the field information of the current time series table and calculate the number of bytes occupied by a row of data using the following formula: Pre-allocated file size = the total amount of data expected to be written to the current time partition * the number of bytes occupied by a row of data, Calculate the file size that needs to be preallocated.