LSM tree data writing method based on data priority
By performing feature extraction and weight calculation on LSM tree data, data priority sorting and classified storage are realized, and the traditional LSM tree data writing priority and classification problems are solved, writing efficiency and storage performance are improved, and important data are processed in a timely manner. and reasonable storage.
Patent Information
- Application Number
- CN202411030134.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-19
AI Technical Summary
Traditional LSM tree data does not consider the write priority and classification of data when writing to disk, resulting in unreasonable resource allocation, delayed processing of key data or not timely deletion of garbage data, and unreasonable disk space utilization.
By extracting and weighting the data features, prioritizing writing, and classifying and storing them according to the data feature interval when writing to disk, data of different block types are merged in time.
It significantly improves data writing efficiency and storage performance, ensures that important data is written first and partitioned reasonably, avoiding resource waste and data loss.
Smart Images

Figure CN118981279A_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the technical field of signal processing and relates to a method for writing LSM tree data based on data priority. Background Art
[0002] In modern database systems, LSM tree is an ordered off-site update data structure. As an efficient data structure, it is widely used to manage a large number of write-intensive workloads. Its working principle is to write data to the write buffer (MemTable) in memory. When the write buffer reaches a certain threshold, the data will become a read-only block (ImmutableMemtable). The data is sequentially written to the first-level ordered string table (SSTable) on the disk through the flush operation, and the SSTable level that reaches the preset threshold on the disk is merged and sorted with the SSTable of the next level through regular merging (Compaction) operations, thereby reducing the disk usage space and optimizing data storage and query efficiency. However, traditional LSM tree data does not consider the data write priority and data classification when writing data to disk. The sequential writing of data will lead to unreasonable resource allocation, delayed key data processing, or untimely deletion of junk data when facing data of different importance and urgency, resulting in unreasonable disk space utilization.
[0003] Therefore, it is urgent to develop a strategy that can determine data priority and write it in the corresponding order to improve the writing quality and efficiency of LSM tree data. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide an LSM tree data writing method based on data priority, which prioritizes and writes data by extracting data features and solving related write weights. At the same time, when writing data to disk, the data is classified and stored according to the coefficient interval solved for the data, and data of different block types are merged in time, thereby significantly improving data writing efficiency and storage performance.
[0005] In order to achieve the above object, the present invention provides the following technical solutions:
[0006] A method for writing data to an LSM tree based on data priority specifically comprises the following steps:
[0007] S1: Write the data key-value pair into the write buffer (Memtable) in the memory, and then determine whether the write buffer (Memtable) reaches the threshold. If so, enter it into the data sorting block;
[0008] S2: Data sorting block, used to solve the data feature values of the data in the write buffer according to the data features and assign corresponding weights to the importance of different data features to obtain data weight values, and sort the data according to the weight values;
[0009] S3: The sorted blocks are converted into read-only blocks (Immutable Memtable) and then input into the partition storage judgment block of the disk;
[0010] S4: The partition storage judgment block will divide the disk layer 0 into blocks according to the data characteristics, and solve the relevant data weights of the written data according to the data characteristics, and write it to the corresponding divided blocks.
[0011] Further, step S1 specifically includes the following steps:
[0012] S11: Extract data key-value pairs from the WAL of the disk (DISK) and input them into the write buffer (Memtable) section of the memory in order;
[0013] S12: Set the space threshold size of the write buffer (Memtable) section, and determine whether the write buffer (Memtable) section reaches the set space threshold, obtain the space information of the write buffer (Memtable), and if it exceeds the set threshold size, input the write buffer (Memtable) section data into the data sorting block.
[0014] Further, step S2 specifically includes the following steps:
[0015] S21: Collect characteristic information of data key-value pairs, select representative data features in the data, and establish relevant algorithms as a method for solving data characteristic values;
[0016] S22: Determine the impact of data features on data importance, and assign corresponding weight coefficients to the solved data features according to the impact degree;
[0017] S23: The data feature value is combined with the obtained weight coefficient to obtain the data weight value, and the data is sorted according to the bubble sort method.
[0018] Further, step S21 specifically includes: selecting data features in three aspects, namely, data anomaly detection situation, collection time, and data size, to calculate relevant data feature values;
[0019] The abnormal detection situation refers to a data point in the collected data that deviates from the expected situation, such as data missing, format error, etc. Since it cannot be converted into a numerical value by itself, it is abstracted and set to obtain the corresponding data characteristic value ω1, and the detected abnormal data situation needs to be written first, so the more abnormal situations appear in the data and the greater the impact, the larger the data characteristic value ω1;
[0020] The acquisition time refers to the time point when the data source obtains the data. It is usually transmitted together with the data as a data key. It can be converted into numerical values by itself, so there is no need for abstract settings. Since the timeliness requirements for data storage are relatively strict, its numerical conversion function can be set to an exponential form, and the data characteristic value ω2=e can be obtained. -λΔT , where λ is the decay rate and ΔT represents the time difference;
[0021] The data size refers to the value of the data itself and the size of the space occupied by the key. It can be converted into a numerical value. By judging the size of the write buffer occupied by the data, the data characteristic value can be set to obtain the data characteristic value. Where B is the write buffer threshold, x i is the size of the i-th data.
[0022] Further, step S22 specifically includes: by judging the importance of the three data features, the corresponding weight coefficients φ1, φ2, φ3 are given respectively, and according to the data feature values and the weight coefficients, the weight value is solved: f=φ1ω1+φ2ω2+φ3ω3.
[0023] Further, step S23 specifically includes: comparing the weight values of the data to obtain the writing priority of different data, re-arranging and combining the data, and placing the data with high priority at the front end of the data sorting block so that it can be written first.
[0024] Further, step S3 specifically includes the following steps:
[0025] S31: Store the data into a read-only block (Immutable Memtable) through a block operation;
[0026] S32: The read-only block (Immutable Memtable) data is sequentially input from the memory to the partition storage judgment block of the disk.
[0027] Further, step S4 specifically includes the following steps:
[0028] S41: Determine the influence of writing, reading and storing according to the data characteristics, solve the relevant block coefficients and perform corresponding block division;
[0029] S42: Perform numerical abstraction setting on the data features of the written data, perform weight division on different data features, obtain relevant data coefficients by solving, and write them into different blocks divided on the disk.
[0030] Further, step S41 specifically includes: selecting storage cost, access frequency, update frequency, and compressibility to calculate relevant data feature values according to the influence of data features on its writing, querying, and storage;
[0031] (1) The storage cost characteristic value is set to the number of layers n and the number of files in the nth layer F n The relevant equation, that is, the storage cost characteristic value is expressed as
[0032] (2) The access frequency characteristic value is set to be equal to the number of layers n and the reading frequency f c The relevant equation, that is, the access frequency eigenvalue is expressed as
[0033] (3) The update frequency characteristic value is set to the number of layers n and the write frequency f r The relevant equation, that is, the update frequency eigenvalue is expressed as
[0034] (4) The compressibility characteristic value is set to be equal to the data size D and the data compression size D i The relevant equation, that is, the compressibility eigenvalue is expressed as w4 = D i / D.
[0035] Further, step S42 specifically includes the following steps:
[0036] S421: After obtaining the characteristic values of different data characteristics, weight coefficients are set according to the degree of influence of the data characteristics on its writing, querying, and storage. Solve for the weight value of the data
[0037] S422: Divide the disk into disks according to weight value intervals at disk layer 0, and write weight values obtained from different data into corresponding disk intervals.
[0038] The beneficial effects of the present invention are:
[0039] 1) The present invention constructs a data sorting block in the memory area, performs write priority evaluation on data features, strengthens resource allocation for writing important data, and enhances the quality of writing.
[0040] 2) The present invention constructs a partition storage determination block in the disk area, divides the blocks according to the influence of data characteristics on writing, reading and storage, and stores different types of data in different blocks, which is beneficial to data reading.
[0041] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:
[0043] Figure 1 It is a data writing flow chart of the overall data writing method of the LSM tree data writing method based on data priority of the present invention;
[0044] Figure 2 Write sorting details for the data in the data sorting block example;
[0045] Figure 3 Detailed diagram of the data partition storage process. DETAILED DESCRIPTION
[0046] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.
[0047] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.
[0048] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0049] See also Figure 1 to Figure 3 The present invention proposes a method for writing data to an LSM tree based on data priority sorting, and the method specifically comprises the following steps:
[0050] S1: Write the data key-value pair into the write buffer (Memtable) in the memory, and then determine whether the write buffer (Memtable) reaches the threshold. If it reaches the threshold, enter it into the data sorting block; specifically, the following steps are included:
[0051] S101: When data is written into the memory, the memory reads the relevant data feature key-value pairs of the input data and places the data in the write buffer Memtable according to the sequential structure.
[0052] S102: According to the designated size of the memory write buffer Memtable, a threshold is set according to a certain ratio.
[0053] S103: Input the Memtable blocks that have been written to a data sorting block in excess of the threshold.
[0054] S2: Data sorting block, used to solve the data feature values of the data in the write buffer according to the data features and assign corresponding weights to the importance of different data features to obtain data weight values, and sort the data according to the weight values; specifically, it includes the following steps:
[0055] S201: before the write buffer reaches a threshold and becomes a read-only block, a data sorting block is set to extract and analyze data features of each item of data in the write buffer one by one.
[0056] S202: Select data features in three aspects: data anomaly detection situation, collection time, and data size to calculate relevant data feature values.
[0057] Anomaly detection refers to a situation where a data point in the collected data deviates from the expected situation, such as missing data, format error, etc. Since it cannot perform numerical conversion itself, it is abstracted and set to obtain the corresponding data characteristic value ω1, and the detected abnormal data situation needs to be written first. Therefore, the more abnormal situations appear in the data and the greater the impact, the larger the data characteristic value ω1.
[0058] The acquisition time refers to the time point when the data source obtains the data. It is usually transmitted along with the data as a data key. It can be converted into numerical values by itself, so there is no need for abstract settings. Since the timeliness requirements for data storage are relatively strict, its numerical conversion function can be set to an exponential form, where λ is the decay rate and ΔT represents the time difference. The data characteristic value ω2=e -λΔT .
[0059] Data size refers to the value of the data itself and the size of the space occupied by the key. It can be converted into a numerical value. By judging the size of the write buffer occupied by the data, its data characteristic value is set, where B is the write buffer threshold, x i is the size of the i-th data, and the data characteristic value can be obtained
[0060] S203: By judging the importance of these three data features, the corresponding weight coefficients φ1, φ2, and φ3 are given respectively. According to the data feature values and the weight coefficients, the weight value f=φ1ω1+φ2ω2+φ3ω3 is solved.
[0061] S204: Compare the weight values of the data to obtain the writing priorities of different data, rearrange and combine the data, and place the data with high priority at the front end of the data sorting block so that it can be written first.
[0062] S3: The sorted blocks are converted into read-only blocks (Immutable Memtable), and then input into the partition storage judgment block of the disk; specifically, the following steps are included:
[0063] S301: After the data is sorted in the sorting block, it will enter the read-only block of the memory area, and the data will be blocked and cannot be operated on by other means.
[0064] S302: The data of the read-only block will be written to the 0th layer of the disk.
[0065] S4: The partition storage judgment block will divide the disk layer 0 into blocks according to the data characteristics, and solve the relevant data weights of the written data according to the data characteristics, and write it to the corresponding divided blocks.
[0066] S401: To facilitate storage and search of different data structures, the disk layer 0 is partitioned and a partition storage determination block is set.
[0067] S402: According to the influence of data characteristics on its writing, querying and storage, four aspects including storage cost, access frequency, update frequency and compressibility are selected to calculate relevant data characteristic values.
[0068] (1) The storage cost characteristic value is set to the number of layers n and the number of files in the nth layer F n The relevant equation, whose data eigenvalue is expressed as
[0069] (2) The access frequency characteristic value is set to be equal to the number of layers n and the reading frequency f c The relevant equation, whose data eigenvalue is expressed as
[0070] (3) The update frequency characteristic value is set to the number of layers n and the write frequency f r The relevant equation, whose data eigenvalue is expressed as
[0071] (4) The compressibility characteristic value is set to be equal to the data size D and the data compression size D i The data characteristic value of the equation is expressed as w4=D i / D.
[0072] S403: After obtaining the characteristic values of different data characteristics, weight coefficients are set according to the degree of influence of the data characteristics on its writing, querying, and storage. Solve for the weight value of the data
[0073] S404: Divide the disk into disks according to weight value intervals at disk layer 0, and write weight values obtained from different data into corresponding disk intervals.
[0074] From the principle, it can be seen that the method of the present invention can effectively write important data and prevent the loss of key data. At the same time, partitioned and classified writing is conducive to the merging of data of the same type and data search.
[0075] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.
Claims
1. A method for writing data to an LSM tree based on data priority, characterized in that: The method specifically comprises the following steps: S1: Write the data key-value pair into the write buffer in the memory, and then determine whether the write buffer reaches the threshold. If so, enter it into the data sorting block; S2: Data sorting block, used to solve the data feature values of the data in the write buffer according to the data features and assign corresponding weights to the importance of different data features to obtain data weight values, and sort the data according to the weight values; S3: The sorted blocks are converted into read-only blocks and then input into the partition storage judgment block of the disk; S4: The partition storage judgment block will divide the disk layer 0 into blocks according to the data characteristics, and solve the relevant data weights of the written data according to the data characteristics, and write it to the corresponding divided blocks.
2. The LSM tree data writing method based on data priority according to claim 1 is characterized in that: Step S1 specifically includes the following steps: S11: Extract data key-value pairs from the WAL on disk and input them into the write buffer section of the memory in order; S12: Setting a space threshold size of the write buffer block, and determining whether the write buffer block reaches the set space threshold, obtaining space information of the write buffer, and if it exceeds the set threshold size, inputting the write buffer block data into the data sorting block.
3. The LSM tree data writing method based on data priority according to claim 1 is characterized in that: Step S2 specifically includes the following steps: S21: Collect characteristic information of data key-value pairs, select representative data features in the data, and establish relevant algorithms as a method for solving data characteristic values; S22: Determine the impact of data features on data importance, and assign corresponding weight coefficients to the solved data features according to the impact degree; S23: The data feature value is combined with the obtained weight coefficient to obtain the data weight value, and the data is sorted according to the bubble sort method.
4. The LSM tree data writing method based on data priority according to claim 3 is characterized in that: Step S21 specifically includes: selecting data features in three aspects, namely, data anomaly detection situation, collection time, and data size, to calculate relevant data feature values; The abnormal detection situation refers to a data point in the collected data that deviates from the expected situation. Since it cannot be converted into a numerical value, it is abstracted and set to obtain the corresponding data characteristic value ω1, and the detected abnormal data situation is written first; The acquisition time refers to the time point when the data source obtains the data. The numerical conversion function is set to an exponential form to obtain the data characteristic value ω2=e -λΔT , where λ is the decay rate and ΔT represents the time difference; The data size refers to the value of the data itself and the size of the space occupied by the key. The data characteristic value is set by judging the size of the write buffer occupied by the data. Where B is the write buffer threshold, x i is the size of the i-th data.
5. The LSM tree data writing method based on data priority according to claim 4 is characterized in that: Step S22 specifically includes: by judging the importance of the three data features, the corresponding weight coefficients φ1, φ2, φ3 are given respectively, and according to the data feature values and the weight coefficients, the weight value is solved: f=φ1ω1+φ2ω2+φ3ω3.
6. The LSM tree data writing method based on data priority according to claim 3 is characterized in that: Step S23 specifically includes: comparing the weight values of the data to obtain the writing priority of different data, rearranging and combining the data, and placing the data with high priority at the front end of the data sorting block so that it can be written first.
7. The LSM tree data writing method based on data priority according to claim 1 is characterized in that: Step S3 specifically includes the following steps: S31: Store the data into a read-only block through a block operation; S32: inputting the read-only block data from the memory into the partition storage judgment block of the disk in sequence.
8. The method for writing data to an LSM tree based on data priority according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41: Determine the impact of writing, querying and storage according to data characteristics, solve relevant block coefficients and perform corresponding block division; S42: Perform numerical abstraction setting on the data features of the written data, perform weight division on different data features, obtain relevant data coefficients by solving, and write them into different blocks divided on the disk.
9. The method for writing data to an LSM tree based on data priority according to claim 8, characterized in that: Step S41 specifically includes: selecting storage cost, access frequency, update frequency, and compressibility to calculate relevant data feature values according to the influence of data features on its writing, querying, and storage; (1) The storage cost characteristic value is set to the number of layers n and the number of files in the nth layer F n The relevant equation, that is, the storage cost characteristic value is expressed as (2) The access frequency characteristic value is set to be equal to the number of layers n and the reading frequency f c The relevant equation, that is, the access frequency eigenvalue is expressed as (3) The update frequency characteristic value is set to the number of layers n and the write frequency f r The relevant equation, that is, the update frequency eigenvalue is expressed as (4) The compressibility characteristic value is set to be equal to the data size D and the data compression size D i The relevant equation, that is, the compressibility eigenvalue is expressed as w4 = D i / D.
10. The LSM tree data writing method based on data priority according to claim 9 is characterized in that: Step S42 specifically includes the following steps: S421: After obtaining the characteristic values of different data characteristics, weight coefficients are set according to the degree of influence of the data characteristics on its writing, querying, and storage. Solve for the weight value of the data S422: Divide the disk into disks according to weight value intervals at disk layer 0, and write weight values obtained from different data into corresponding disk intervals.
