A data optimization storage method and device based on storage medium merging

Abstract

The application relates to the field of information technology processing, in particular to a data optimization storage method and equipment based on storage medium combination, which comprises the following steps: according to a time stamp and a key identification, time sequence data in a memory is divided into time buckets with different time granularities; differentiated combination trigger conditions and combination granularity strategies are configured for different time buckets; an optimized data storage structure is adopted; index blocks are sorted according to the combination granularity strategies, and the data blocks pointed by the index blocks are loaded in sequence; sorting and combination operations are performed on the data blocks in the memory, and the data blocks are immediately written into target storage media; and the iteration is performed until the data of all the target time buckets is completely stored. The equipment is used for executing the steps described in the method. The application solves the problems of excessively high system memory occupation, low resource utilization, poor system expansibility, improves system performance and stability, and enhances the system processing capacity of large data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology processing, in particular to a data optimized storage method and device based on storage medium merging. BACKGROUND

[0002] With the rapid development of Internet technology, network monitoring, business operation and network security fields are generating massive time series data every moment. In order to efficiently analyze these data, the industry generally adopts "time bucket" (DataBucket) aggregation technology, that is, the original data stream is merged according to a fixed time window (such as seconds, minutes, hours) to generate multi-granularity aggregated data such as second bucket, minute bucket and hour bucket.

[0003] The current time bucket-based data storage and query system faces serious technical challenges and performance bottlenecks in high concurrency and large data volume scenarios. In the data processing process, in order to ensure real-time performance, the system usually needs to maintain a large number of latest time bucket aggregation states in memory. With the accumulation of the number of buckets, these intermediate states will continue to occupy valuable memory space, and if not released in time, it will easily lead to memory exhaustion, causing system performance problems, and the amount of data that the system can handle is limited by the memory capacity.

[0004] The existing solutions either increase hardware resources for expansion, which is costly, or periodically clean up data to release space, sacrificing data integrity. Therefore, how to design a method that can intelligently and efficiently manage the life cycle of time bucket data under limited memory and storage resources has become a key technical problem. SUMMARY

[0005] The purpose of the present application is to overcome the problems of high memory occupation, low resource utilization and system instability caused by long-term residence of large bucket data in memory before reaching the merging opportunity in the prior art, and to provide a data optimized storage method and device based on storage medium merging.

[0006] In one aspect, the present application provides a data optimized storage method based on storage medium merging, comprising the following steps:

[0007] S1. According to the timestamp and key identification, the time series data in the memory is divided into different time granularity time buckets; the different time granularity time buckets are configured with differentiated merging trigger conditions and merging granularity strategies.

[0008] Preferably, the different granularity time buckets in step S1 represent that each of the time buckets stores time series data corresponding to the time granularity, including 1 second bucket, 10 second bucket, 1 minute bucket, 10 minute bucket, 1 hour bucket and 1 day bucket.

[0009] Preferably, the merging granularity strategy in step S1 is that real-time memory merging strategy is adopted for 1-second bucket and 10-second bucket data, and asynchronous storage medium merging strategy is adopted for 1-minute bucket and above time buckets.

[0010] S2. The storage structure of the time bucket in the target storage medium comprises data blocks and corresponding index blocks; the data of the same time bucket is composed of at least one data block and a corresponding number of index blocks.

[0011] Preferably, the data block in step S2 has key-value as the minimum storage granularity, and all the key-values in the same data block are sequentially stored according to the size of the key.

[0012] Preferably, the index block in step S2 is metadata of the data block, used to record the size of the data block, the data interval of the key-value, and the offset address of the data block in the storage medium.

[0013] Preferably, the endpoint value of the data interval of the key-value in step S2 is the data corresponding to the minimum key and the data corresponding to the maximum key stored in the corresponding data block in the index block.

[0014] S3. Read all the index blocks corresponding to the target time bucket from the target storage medium to the memory; sort the index blocks according to the merging granularity strategy, and load the data blocks pointed by the index blocks in order; perform sorting and merging operations on the data blocks in the memory to generate new aggregated data blocks and aggregated index blocks, and immediately write them to the target storage medium; iteratively perform index loading, data sorting, and merging writing processes until the data of all the target time buckets are completed.

[0015] Preferably, the sorting and merging operations in step S3 are as follows:

[0016] S31. Read N index blocks to be merged into the memory, where N≥2;

[0017] S32. Based on the information of the N index blocks, sort all the data blocks pointed by the index blocks in a global merging flow for planning a plurality of sequentially executed merging rounds and determining the data blocks to be involved in each merging round;

[0018] S33. According to the order of the merging rounds planned in step S32, sequentially execute each round of merging operation: load the data blocks corresponding to the current merging round from the target storage medium to the memory for multi-way merge sorting to generate ordered merging data of this round;

[0019] S34. Data persistence storage:

[0020] The ordered merged data generated in each merging round is written into the target storage medium, and the memory space occupied by the data block loaded in this round is immediately released; multiple rounds of iteration are performed until the data merging of all the target time buckets is completed.

[0021] In another aspect, the application also provides an apparatus for data optimization storage based on storage medium merging, comprising a power supply and a processor, a memory, an input / output interface connected to each other, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to realize the steps of the data optimization storage method based on storage medium merging.

[0022] Compared with the prior art, the application has the following beneficial effects:

[0023] 1. The application can significantly reduce memory occupation through the storage medium merging strategy.

[0024] 2. The application provides an on-demand loading and timely disk writing mechanism, which only loads the required data in the memory, and writes the data in the memory that completes the target task into the storage medium as soon as possible, thereby avoiding system unhealthiness caused by long-term high memory occupation and improving system stability.

[0025] 3. The application can realize enhanced data processing capacity through the optimized data storage structure and index mechanism, and the data volume that the system can process is no longer limited by the memory capacity. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 The merging strategy provided by the application is shown in the figure;

[0027] Figure 2 The logical diagram of merging 1-second bucket data into 10-second bucket data provided by the application is shown in the figure;

[0028] Figure 3 The time bucket storage structure model under the prior art is shown in the figure;

[0029] Figure 4 The optimized time bucket storage structure model provided by the application is shown in the figure;

[0030] Figure 5 The sorting and merging process provided by the application is shown in the figure;

[0031] Figure 6 The device structure of embodiment 2 of the application is shown in the figure. DETAILED DESCRIPTION

[0032] The application will be further described in conjunction with test examples and specific embodiments. However, it should not be understood that the scope of the above-mentioned subject matter of the application is limited to the following examples. Any technology achieved based on the content of the application falls within the scope of the application.

[0033] In the description of the specific embodiments of the application, the orientation or position relationship terms such as "upper", "lower", "left", "right", "center", "inner", "outer", and the like are expressed based on the orientation or position relationship shown in the drawings, or the orientation or position relationship when the product / device / apparatus of the application is usually used. These orientation or position relationship terms are only for the convenience of describing the application scheme or simplifying the description in the specific embodiments, and for the convenience of the technical personnel to quickly understand the scheme, and do not indicate or imply that a specific device / component / element must have a specific orientation or be constructed and operated in a specific position relationship, and therefore cannot be understood as a limitation on the application.

[0034] In addition, if the terms "horizontal", "vertical", "suspension", "parallel", and the like appear, it does not mean that the corresponding device / component / element must be absolutely horizontal or vertical or suspended or parallel, but can be slightly inclined or deviated. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined. Alternatively, it can be simplified to understand that the corresponding device / component / element is arranged in the "horizontal", "vertical", "suspension", "parallel" direction, and can have an error / deviation of ±10% relative to the corresponding direction, more preferably an error / deviation of ±8% or less, more preferably an error / deviation of ±6% or less, more preferably an error / deviation of ±5% or less, and more preferably an error / deviation of ±4% or less. As long as the corresponding device / component / element is within the error / deviation range, it can still achieve its role in the scheme of the application.

[0035] In addition, the terms "first", "second", "third", and the like appearing in the terms are only used to distinguish the description of the same or similar components, and should not be understood as emphasizing or implying the relative importance of the specific components.

[0036] In addition, in the description of the embodiments of the application, "several", "a plurality of", and "several" represent at least 2. It can be 2, 3, 4, 5, 6, 7, 8, 9, etc. in any case, and can even be more than 9.

[0037] Furthermore, in the description of the technical solutions of the present application, unless otherwise explicitly specified / limited / limited, the terms "set", "install", "connect", "connect", "provide", "lay", "arrange" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrally connected, which can be welding, riveting, bolting, screwing and other commonly used connection means in the art. The connection can be mechanical connection, electrical connection or communication connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication between two elements.

[0038] Embodiment 1

[0039] A data optimization storage method based on storage medium merging, comprising the following steps:

[0040] S1. Configure hierarchical merging strategy:

[0041] According to the timestamp and the key identification, the time sequence data in the memory is divided into different time granularity time buckets; the different time granularity time buckets are configured with differentiated merging trigger conditions and merging granularity strategies.

[0042] The different granularity time buckets include 1 second buckets, 10 second buckets, 1 minute buckets, 10 minute buckets, 1 hour buckets and 1 day buckets; the merging granularity strategy adopts a real-time memory merging strategy for 1 second buckets and 10 second buckets. These two types of buckets occupy less memory and have a shorter memory occupation time; for 1 minute level buckets and larger time buckets, due to the large amount of data, the occupation of memory resources is significantly increased, and the occupation state lasts for a long time. An asynchronous target storage medium merging strategy is adopted, and the merging strategy logic diagram is as follows Figure 1 .

[0043] Taking the time sequence data from 2025 / 05 / 11 16:53:53 to 2025 / 05 / 11 18:21:02 as an example, the data in the memory is divided into different time granularity time buckets.

[0044] First, the data in this time period is divided into 1 second time granularity, and there are 5229 seconds in this time period, generating 5229 1 second buckets.

[0045] Second, divide by 10 seconds as a time granularity, extract the scale from the entire time period, and there are 522 whole 10 second scales in the time period, generating 522 10 second buckets.

[0046] Third, divide by 1 minute as a time granularity, and there are 88 whole 1 minute scales in the time period, generating 88 1 minute buckets.

[0047] Finally, the time granularity is divided by hour, and there are 2 whole points in this time period, generating 2 1-hour buckets.

[0048] In this way, the original time series data is effectively organized into buckets of different time granularities for the next step of data processing and analysis.

[0049] Take the 1-second bucket data as an example to explain the data merging strategy process.

[0050] The system processes data in seconds, and each timestamp corresponds to a 1-second time bucket that stores all data records within the corresponding timestamp. Once the system completes processing of data for a specific timestamp in memory, the data will be stored through the following two steps: first, store the data in a persistent storage medium such as a disk; at the same time, keep the data in memory for subsequent merging operations of larger time buckets.

[0051] When the accumulation reaches 10 consecutive timestamp data, the time bucket merging operation is performed to merge these data into a larger 10-second time bucket. This merging process follows consistent logic to ensure data integrity and consistency.

[0052] As shown in Figure 2 , take timestamp 1764838960 as an example. When the data corresponding to this timestamp is processed and 10 consecutive timestamp data is collected, the system will perform a merging operation. Through this operation, 10 1-second buckets of data are merged into a single 10-second time bucket to optimize storage efficiency and subsequent data processing flow.

[0053] The principle of optimizing memory performance by time bucket hierarchical merging strategy: In a data system using key-value (key-value-data) structure storage, each time bucket is a collection of data containing multiple key-value data records. Within any single time bucket, the same key corresponds to only one data record to ensure data uniqueness and consistency.

[0054] For example, querying data within a specific time range, such as 2025 / 05 / 11 16:53:53 - 2025 / 05 / 11 18:21:02, a total of 5249 seconds. If the time bucket hierarchical merging strategy is not used, the data will only be aggregated by 1 second, and 5249 1-second buckets will be accessed during querying, resulting in 5249 input / output (I / O) operations.

[0055] After adopting the time bucket aggregation strategy, data is aggregated at different time granularities to form time buckets of different sizes:

[0056] 1 hour time bucket: for example, 17:00, corresponding to 1 data block;

[0057] 10 minutes time bucket: for example, 18:00 and 18:10, corresponding to 2 data blocks;

[0058] 1 minute time bucket: for example, 16:54 to 16:59 and 18:20, corresponding to 7 data blocks;

[0059] 1 second time bucket: for example, 16:53:53 to 16:53:59 and 18:21:00 to 18:21:01, corresponding to 9 data blocks.

[0060] In summary, the above-mentioned time bucket, when querying, only needs to access 19 time buckets, and initiates 19 I / O operations. This method significantly reduces the number of disk operations, thereby speeding up the query efficiency.

[0061] In summary, the time bucket hierarchical merging strategy can significantly improve the query efficiency, reduce I / O operations, release system resources, and optimize memory processing performance when processing large-scale time queries.

[0062] S2. Construct an optimized storage structure:

[0063] The storage structure of the time bucket in the target storage medium includes data blocks and their corresponding index blocks; the data of the same time bucket is composed of at least one data block and a corresponding number of index blocks; the data block takes key-value as the minimum storage granularity, and all key-values in the same data block are stored in order according to the size of the key; the index block is the metadata of the data block, used to record the size of the data block, the data interval of the key-value, and the offset address of the data block in the storage medium; the endpoint value of the data interval of the key-value is the data corresponding to the smallest key stored in the corresponding data block in the index block, and the data corresponding to the largest key.

[0064] The target storage medium can include one or more types of hard disk, storage array, and cloud storage.

[0065] The current prior art storage structure model is shown in Figure 3 The data in the time bucket is stored in a linear manner, and there is a lack of optimization for data access and retrieval performance, resulting in low efficiency when performing complex queries or large-scale data operations. The optimized time bucket storage structure model is shown in Figure 4 The main structure modules and functions are:

[0066] Data Block, used for batch storage of key-value data, with internal data ordered by key to optimize range queries and data retrieval.

[0067] Index Block, used to record metadata information of the Data Block to support fast data positioning and access. The metadata information includes:

[0068] Offset, used to indicate the starting offset position of the Data Block in the file for quick positioning.

[0069] Length, used to record the length of the Data Block, which helps determine the end position of the Data Block.

[0070] Min_data, used to store the data corresponding to the first key (minimum value) in the Data Block, to quickly determine the starting range of the Data Block.

[0071] Max_data, used to store the data corresponding to the last key (maximum value) in the Data Block, to quickly determine the end range of the Data Block.

[0072] By introducing the structure of Data Block and Index Block, the optimized storage structure significantly improves the data read-write efficiency and retrieval performance. This structure model is designed to improve read-write efficiency and data retrieval performance, and is particularly suitable for handling large-scale data sets, as it allows the system to quickly locate and access specific ranges of data without the need to scan the entire data set. In addition, the ordered data arrangement also makes range queries more efficient. In summary, the optimized storage structure not only improves data processing performance, but also provides greater flexibility and scalability for data management and maintenance.

[0073] S3. Perform multi-round merge write operation with on-demand sorting:

[0074] Read all the Index Blocks corresponding to the target time bucket from the target storage medium to the memory; sort the Index Blocks according to the merge granularity strategy, and load the Data Blocks pointed to in order; perform sorting and merging operations on the Data Blocks in the memory to generate new aggregated Data Blocks and aggregated Index Blocks, and immediately write them to the target storage medium; iterate the index loading, data sorting, and merge writing process until all the data of the target time bucket is stored.

[0075] The specific steps of sorting and merging operations are:

[0076] S31. Read N Index Blocks to be merged into the memory, where N ≥ 2;

[0077] S32. Based on the information of the N index blocks, all the data blocks pointed by the index blocks are globally sorted in a merging process for planning a plurality of sequentially executed merging rounds and determining the data blocks participating in the merging in each merging round;

[0078] S33. In the order of the merging rounds planned in step S32, each round of merging operation is executed in sequence: the data blocks corresponding to the current merging round are loaded from the target storage medium to the memory for multi-way merge sorting to generate ordered merging data of the round;

[0079] S34. Data persistence storage:

[0080] The ordered merging data generated in each merging round is written to the target storage medium, and the memory space occupied by the data blocks loaded in the round is immediately released; multiple rounds of iteration are performed until the data merging of all the target time buckets is completed.

[0081] To make the technical solutions and beneficial effects of sorting and merging clearer, a 1-hour bucket merging process is taken as an example to explain and describe in detail. In actual system processing, 24 1-hour bucket data are merged into 1 1-day bucket data. For ease of understanding, the 1-hour bucket data are taken as an example in the present application, and the sorting and merging process is described, as shown in the process schematic diagram Figure 5 .

[0082] (1) The index blocks of the 3 1-hour buckets are read from the storage medium to the system memory. The index blocks contain metadata of the data blocks, and the memory space occupied is small, so they can be completely loaded into the memory at one time for subsequent processing.

[0083] (2) The merging operation process is sorted according to the information in the index blocks. Since the data blocks are stored in order according to the key, and the index blocks record the maximum value (max_data) and the minimum value (min_data) of the data stored in the corresponding data blocks, these information can be used to effectively sort the merging process. After sorting by index, the number of rounds required for the merging process can be determined, which is 10 rounds in this example. In each round of merging process, only the data blocks required for merging in the round need to be loaded into the memory.

[0084] (3) According to the sorting result of the index blocks, the corresponding data blocks are loaded from different time buckets to the memory, and the merging operation is performed. For example, in the first round of merging, only data block 1 of time bucket 1 and data block 1 of time bucket 3 need to be loaded into the memory for merging. After the merging is completed, the sorted data blocks are directly released from the memory to optimize the memory usage.

[0085] (4) The data block after the merging processing is directly written into the non-volatile storage medium to realize the persistent storage of data and release the occupied memory space in time.

[0086] The optimization result shows that:

[0087] Taking the data of 1-hour bucket merging 1-day bucket as an example, assuming that the data size of each key-value format storage is 200 bytes, and the data amount processed per hour is 5 million key-value data. Before the optimization of the merging logic, in the process of merging 24 1-hour buckets into 1-day bucket, all key-value data needs to be loaded into the memory, and the required memory calculation method is as follows:

[0088]

[0089] Through the optimization of the merging logic, assuming that each data block contains 10,000 key-value data, the index block quantity for 5 million key-value data is

[0090]

[0091] Each index block includes the metadata of its corresponding data block, and the size of the offset and length data has negligible effect on the entire index block data size. Here, only the size of each index block containing two key-value data stored by the minimum data interval (min_data) and the maximum data interval (max_data) is calculated, which is:

[0092]

[0093] In the merging process, all indexes need to be loaded into the memory, so the memory occupied by the indexes is:

[0094]

[0095] In each round of merging process, the data block of each hour bucket needs to be loaded into the memory, and the size of each data block is:

[0096]

[0097] Therefore, the total memory requirement in the merging process is:

[0098]

[0099] By implementing the above optimization measures, the memory usage of the merging process is significantly reduced. Specifically, before optimization, the memory resources required for merging 24 1-hour data buckets into 1-day buckets were 22 GB. After optimization, the memory requirement of this process was reduced to 49 MB. This improvement represents a significant increase in memory usage efficiency, and the specific memory occupancy ratio is calculated as follows:

[0100]

[0101] This indicates that the memory usage after optimization is only 0.21% of that before optimization. Accordingly, the memory resource saving rate is calculated as follows:

[0102]

[0103] Therefore, it can be concluded that the optimized merging logic achieves a saving of about 99.79% in memory resource utilization compared to before optimization. This significant reduction in memory usage not only improves the efficiency of data processing, but also provides the possibility of processing larger-scale data sets, while reducing the hardware requirements of the system.

[0104] Embodiment 2

[0105] The embodiment provides a data optimization storage device based on storage medium merging, i.e. an electronic device, a structural schematic diagram as shown in Figure 6 The electronic device includes a power supply, at least one processor, a memory and an input / output interface in communication connection with the at least one processor. The memory stores instructions executable by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to execute the method disclosed in any of the preceding embodiments. The power supply is used to provide power for the electronic device.

[0106] When the integrated unit of the present application is realized in the form of a software functional unit and sold or used as an independent product, it can also be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes a plurality of instructions for causing a computer device, which can be a server, a personal computer, a network device, to execute all or part of the methods described in the various embodiments of the present application. The aforementioned storage medium includes mobile storage devices, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks, and various media that can store program codes.

[0107] The above merely describes preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A data optimization storage method based on storage medium merging, characterized in that, Includes the following steps: S1. Divide the time-series data in memory into time buckets with different time granularities based on the timestamp and key identifier; Configure differentiated merging trigger conditions and merging granularity strategies for the time buckets with different time granularities; S2. The storage structure of the time bucket in the target storage medium includes data blocks and their corresponding index blocks; the data in the same time bucket consists of at least one data block and a corresponding number of index blocks. S3. Read all the index blocks corresponding to the target time bucket from the target storage medium into the memory; sort the index blocks according to the merging granularity strategy, and load the data blocks they point to in sequence; The data blocks are sorted and merged in the memory to generate new aggregated data blocks and aggregated index blocks, which are then immediately written to the target storage medium. The index loading, data sorting, and merge writing processes are iteratively executed until all data in the target time bucket is stored. The specific steps for the sorting and merging operations are as follows: S31. Read the N index blocks to be merged into the memory, where N≥2; S32. Based on the information of the N index blocks, perform a global merging process sort on all the data blocks they point to, in order to plan multiple sequential merging rounds and determine the data blocks that need to be merged in each merging round; S33. Following the order of the merging rounds planned in step S32, perform the merging operation for each round in sequence: load the data block corresponding to the current merging round from the target storage medium into the memory for multi-way merge sorting to generate the ordered merged data for that round; S34. Persistent Data Storage: Write the ordered merged data generated in each merging round to the target storage medium, and immediately release the memory space occupied by the data block loaded in that round; Multiple iterations are performed until the data from all the target time buckets is merged.

2. The data optimization storage method based on storage medium merging according to claim 1, characterized in that, The time buckets with different time granularities mentioned in step S1 refer to the time series data corresponding to the time granularity stored in each time bucket, including 1-second bucket, 10-second bucket, 1-minute bucket, 10-minute bucket, 1-hour bucket, and 1-day bucket.

3. The data optimization storage method based on storage medium merging according to claim 2, characterized in that, The merging granularity strategy in step S1 is as follows: a real-time memory merging strategy is used for 1-second and 10-second bucket data, and an asynchronous storage medium merging strategy is used for 1-minute and longer time buckets.

4. The data optimization storage method based on storage medium merging according to claim 1, characterized in that, In step S2, the data block is stored with key-value as the smallest storage granularity, and all key-value pairs within the same data block are stored sequentially according to the key size.

5. The data optimization storage method based on storage medium merging according to claim 1, characterized in that, The index block mentioned in step S2 is the metadata of the data block, used to record the size of the data block, the data range of the key-value pair, and the offset address of the data block in the storage medium.

6. The data optimization storage method based on storage medium merging according to claim 5, characterized in that, The endpoint values ​​of the key-value data range are: the data corresponding to the smallest key stored in the corresponding data block in the index block, and the data corresponding to the largest key.

7. A device for data optimization storage based on storage medium merging, characterized in that, The device includes a power supply and interconnected processor, memory, and input / output interfaces, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement the steps of the data optimization storage method based on storage medium merging as described in any one of claims 1 to 6.

Images