Data processing method, device, electronic device and storage medium
By encoding and verifying the data in the LSM tree, the read amplification and write amplification problems in the LSM tree structure are solved, achieving more efficient data storage and read/write performance, and reducing disk I/O pressure and storage costs.
Patent Information
- Application Number
- CN202210459910.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-28
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2042-04-28
AI Technical Summary
Existing LSM tree structures suffer from read amplification and write amplification issues in data storage systems, leading to wasted disk I/O and additional storage costs. Furthermore, the compaction mechanism introduces additional overhead.
The primary key of the data to be written is encoded to generate an encoded key and verification information, which are written to the LSM tree memory table. The global version number and value are then written in batches to the fragment in the target storage engine. The storage location of the value is located by the verification information, thereby reducing the data size of MemTable and the storage space of the target storage engine.
It reduces the storage space of the LSM tree, lowers write amplification, improves read and write performance, and saves disk I/O pressure and target storage engine storage space.
Smart Images

Figure CN114817263B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of data processing technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art
[0002] The LSM tree (Log Structured Merge Trees) structure is widely used in data storage systems. The LSM tree converts random writes into sequential writes, making full use of the fact that the sequential write performance of the disk is greater than the random write performance, thus improving the disk write performance. However, the cost of this approach is read amplification, and reading a record requires multiple disk IOs.
[0003] In related technologies, the compaction mechanism is used to alleviate the above-mentioned read amplification, but it also causes write amplification. For example, when the amount of data is large, the compaction mechanism can cause write amplification to reach 40 times, resulting in a large amount of IO waste. In addition, some improvement methods will bring additional storage costs and read and write overhead. Summary of the invention
[0004] In order to overcome the problems existing in the related art, the present disclosure provides a data processing method, device, electronic device and storage medium. The technical solution of the present disclosure is as follows.
[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:
[0006] Encode at least one primary key of the data to be written to obtain the encoded key, and generate verification information of the value corresponding to the encoded key; the encoded key includes a global version number corresponding to the primary key, and the verification information is used to locate the storage location of the value corresponding to the encoded key;
[0007] Writing the encoding key and the verification information into the memory table of the LSM tree;
[0008] The global version number and the value corresponding to the primary key key are written in batches into segments in the target storage engine, and the file identifier of the storage file corresponding to the segment corresponds to the global version number.
[0009] Optionally, encoding the primary key of the data to be written to obtain the encoded key includes:
[0010] Generate a global version number corresponding to the primary key key;
[0011] The primary key is encoded, and the encoded primary key is encapsulated with the global version number to obtain the encoded key.
[0012] Optionally, the segment includes a value file, and the value file includes an index area and a data area;
[0013] The step of writing the global version number and the value corresponding to the primary key key into a fragment in the target storage engine includes:
[0014] Write the value into the data area, and obtain the offset and data size of the value in the data area;
[0015] The global version number, the offset and the data size are written into the index area.
[0016] Optionally, also include:
[0017] Read the encoding key and the verification information from the memory table;
[0018] Check whether the encoding key has a corresponding value;
[0019] If the encoding key has a corresponding value, the corresponding value is read from the fragment based on the global version number and the verification information.
[0020] Optionally, identification information is stored at a position for storing value in the LSM tree, and the identification information includes a first identification and a second identification, the first identification indicates that a corresponding value is stored in the fragment, and the second identification indicates that a corresponding value is not stored in the fragment;
[0021] The detecting whether the encoding key has a corresponding value includes:
[0022] obtaining identification information from the location;
[0023] If the identification information is the first identification, the encoding key has a corresponding value; if the identification information is the second identification, the encoding key has no corresponding value.
[0024] Optionally, in a case where the fragment is recorded in a preset fragment list, reading a corresponding value from the fragment based on the global version number includes:
[0025] Determine the target segment in the segment list by using the global version number;
[0026] Obtain the target offset and target data size from the index area of the target segment, and use the verification information to locate and determine the location information of the stored value;
[0027] The target offset, the target data size and the location information are used to read the corresponding value from the data area.
[0028] Optionally, also include:
[0029] Detect whether there is a duplicate encoding key in the memory table;
[0030] If there are duplicate encoding keys, determine the target encoding key to be cleaned up and the target version number corresponding to the target encoding key, and record the target version number in the deletion file of the target segment;
[0031] Delete the corresponding value from the target storage engine according to the deletion file.
[0032] Optionally, the segment further includes a deletion file, and the deletion file is used to record the version number to be cleaned up in the segment;
[0033] The step of recording the target version number in a deletion file of the target segment includes:
[0034] Obtain the target version number corresponding to the target code key, and record the target version number in the deletion log;
[0035] When it is detected that the preset update condition is met, the target version number in the deletion log is recorded in the deletion file of the corresponding target segment.
[0036] Optionally, in a case where the segment is recorded in a preset segment list, deleting the corresponding value from the target storage engine according to the deletion file includes:
[0037] Determining whether the target segment needs to be cleaned up according to the deleted file;
[0038] If the target segment needs to be cleaned, then the target value in the target segment that does not need to be cleaned is determined;
[0039] Generate a new segment and write the target value into the new segment;
[0040] Deleting the target segment;
[0041] The new segment is recorded in a preset segment list, and the target segment is deleted from the segment list.
[0042] Optionally, determining the target value that does not need to be cleaned in the target segment includes:
[0043] Check whether the snapshot version number used to generate the snapshot exists in the LSM tree;
[0044] If a snapshot version number exists, the value corresponding to the snapshot version number and the value corresponding to the version number not recorded in the deletion file of the target segment are used as target values that do not need to be cleaned up.
[0045] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing device, including:
[0046] An encoding unit, configured to encode at least one primary key of the data to be written to obtain an encoding key;
[0047] a generating unit configured to generate verification information of a value corresponding to the encoding key; the encoding key includes a global version number corresponding to the primary key, and the verification information is used to locate a storage location of the value corresponding to the encoding key;
[0048] A writing unit, configured to write the encoding key and the verification information into a memory table of the LSM tree;
[0049] The storage unit is configured to write the global version number and the value corresponding to the primary key key in batches into segments in the target storage engine, and the file identifier of the storage file corresponding to the segment corresponds to the global version number.
[0050] Optionally, the encoding unit is specifically configured as follows:
[0051] Generate a global version number corresponding to the primary key key;
[0052] The primary key is encoded, and the encoded primary key is encapsulated with the global version number to obtain the encoded key.
[0053] Optionally, the segment includes a value file, and the value file includes an index area and a data area;
[0054] The writing unit is specifically configured as follows:
[0055] Write the value into the data area, and obtain the offset and data size of the value in the data area;
[0056] The global version number, the offset and the data size are written into the index area.
[0057] Optionally, also include:
[0058] A first reading unit is configured to read the encoding key and the verification information from the memory table;
[0059] A first detection unit is configured to detect whether the encoding key has a corresponding value;
[0060] The second reading unit is further configured to read the corresponding value from the fragment based on the global version number and the verification information if there is a corresponding value for the encoding key.
[0061] Optionally, identification information is stored at a position for storing value in the LSM tree, and the identification information includes a first identification and a second identification, the first identification indicates that a corresponding value is stored in the fragment, and the second identification indicates that a corresponding value is not stored in the fragment;
[0062] The detection unit is specifically configured as follows:
[0063] obtaining identification information from the location;
[0064] If the identification information is the first identification, the encoding key has a corresponding value; if the identification information is the second identification, the encoding key has no corresponding value.
[0065] Optionally, in a case where the segment is recorded in a preset segment list, the second reading unit is specifically configured to:
[0066] Determine the target segment in the segment list by using the global version number;
[0067] Obtain the target offset and target data size from the index area of the target segment, and use the verification information to locate and determine the location information of the stored value;
[0068] The target offset, the target data size and the location information are used to read the corresponding value from the data area.
[0069] Optionally, also include:
[0070] A second detection unit is configured to detect whether there is a duplicate encoding key in the memory table;
[0071] a determination unit configured to determine a target encoding key to be cleaned up and a target version number corresponding to the target encoding key if there is a duplicate encoding key;
[0072] a recording unit configured to record the target version number in a deletion file of the target segment;
[0073] The deleting unit is configured to delete the corresponding value from the target storage engine according to the deletion file.
[0074] Optionally, the segment further includes a deletion file, and the deletion file is used to record the version number to be cleaned up in the segment;
[0075] The recording unit is specifically configured as follows:
[0076] Obtain the target version number corresponding to the target code key, and record the target version number in the deletion log;
[0077] When it is detected that the preset update condition is met, the target version number in the deletion log is recorded in the deletion file of the corresponding target segment.
[0078] Optionally, in the case where the segment is recorded in a preset segment list, the deleting unit includes:
[0079] A first determining subunit is configured to determine whether the target segment needs to be cleaned up according to the deleted file;
[0080] A second determining subunit is configured to determine a target value in the target segment that does not need to be cleaned if the target segment needs to be cleaned;
[0081] A generating subunit generates a new segment and writes the target value into the new segment;
[0082] a deletion subunit configured to delete the target fragment;
[0083] The recording subunit is configured to record the new segment into a preset segment list and delete the target segment from the segment list.
[0084] Optionally, the second determining subunit is specifically configured to:
[0085] Check whether the snapshot version number used to generate the snapshot exists in the LSM tree;
[0086] If a snapshot version number exists, the value corresponding to the snapshot version number and the value corresponding to the version number not recorded in the deletion file of the target segment are used as target values that do not need to be cleaned up.
[0087] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:
[0088] processor;
[0089] a memory for storing processor-executable instructions;
[0090] Wherein, the processor is configured to execute the instructions to implement the data processing method as described in the first aspect.
[0091] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the storage medium are executed by a processor of a mobile terminal, the electronic device can execute the data processing method of the first aspect.
[0092] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes a computer program or computer instructions, and when the computer program or computer instructions are executed by a processor, the data processing method of the first aspect is implemented.
[0093] The technical solution provided by the embodiment of the present disclosure may include the following beneficial effects: encoding at least one primary key of the data to be written, obtaining the encoding key, and generating the verification information of the value corresponding to the encoding key; the encoding key includes the global version number corresponding to the primary key, and the verification information is used to locate the storage position of the value corresponding to the encoding key; writing the encoding key and the verification information into the memory table of the LSM tree; writing the global version number and the value corresponding to the primary key into the fragment in the target storage engine in batches, and the file identifier of the storage file corresponding to the fragment corresponds to the global version number. In this way, although the memory table MemTable of the LSM tree is provided with the key position for storing the key and the value position for storing the value, the value is not stored in the value position, but the verification information is stored in the value position. Since the data size of the verification information is much smaller than the data size of the value, the data size of the MemTable can be greatly reduced, thereby saving the storage space of the LSM tree. In addition, since the value is not stored in the LSM tree, when reading and writing the LSM tree, the write amplification of the LSM tree is greatly improved, the disk IO pressure is reduced, and the read and write performance is improved. In addition, since the target storage engine does not need to store the key corresponding to the value, the storage space of the target storage engine is greatly saved.
[0094] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0095] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0096] Figure 1 A data processing method according to an exemplary embodiment is shown in FIG. Figure 1 .
[0097] Figure 2 It is a schematic diagram of a server framework according to an exemplary embodiment.
[0098] Figure 3 A data processing method according to an exemplary embodiment is shown in FIG. Figure 2 .
[0099] Figure 4 A data processing method according to an exemplary embodiment is shown in FIG. Figure 3 .
[0100] Figure 5 It is a block diagram of a data processing device according to an exemplary embodiment.
[0101] Figure 6 It is a block diagram of a device according to an exemplary embodiment. DETAILED DESCRIPTION
[0102] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention as detailed in the appended claims.
[0103] Figure 1 is a flow chart of a data processing method according to an exemplary embodiment. Figure 1 As shown, the data processing method is used in a server, and the server can interact with the terminal for data. When the user performs an operation in the terminal, the server can process the database according to the operation. The processing of the database includes but is not limited to: writing data, reading data, and deleting data.
[0104] The server is equipped with an LSM tree and a target storage engine, such as Figure 2 As shown, the LSM tree is used to store the primary key in the key-value pair, and the target storage engine is used to store the value in the key-value pair.
[0105] The LSM tree is provided with a memory table MemTable, which is used to record the encoded key, the verification information meta and identification information of the corresponding value. Among them, the verification information is used to locate the storage location of the value corresponding to the encoded key; the identification information includes a first identification and a second identification, the first identification indicates that the value corresponding to the encoding is stored in the segment, and the second identification indicates that the value corresponding to the encoded key is not stored in the segment. In addition, since the MemTable is provided with a key position for storing the primary key key and a value position for storing the value value, but in this embodiment, the value value is not stored in the value position, so the meta and identification information are stored in the value position. Since the data size of the meta and identification information is much smaller than the data size of the value value, the data size of the MemTable can be greatly reduced, thereby saving the storage space of the LSM tree. At the same time, since the value value is not stored in the LSM tree, when reading and writing the LSM tree, the write amplification of the LSM tree is greatly improved, the disk IO pressure is reduced, and the read and write performance is improved.
[0106] It should be noted that MemTable may not set identification information. In this case, the value can be directly read from the storage, and the corresponding reading results include successful reading and failed reading. The difference from MemTable setting identification information is that when reading the value, you can first determine whether there is a value through the identification information. If there is a value, read it, and if there is no value, do not read it. In practical applications, whether the identification information is needed can be set according to actual needs. For ease of understanding, this embodiment is explained with the identification information set.
[0107] The target storage engine is configured with multiple segments, each of which includes two files: a value file and a delete file.
[0108] The value file consists of two parts: the file header and the file tail. The file header is the index area (index), which indicates the offset (offset) and data size (size) of the value corresponding to the global version number (version). Since version, offset, and size are all numbers, they can be stored in a fixed byte space, and there is no need to store the primary key corresponding to the value, which greatly saves storage space. At the same time, since the version itself is incremental, it can be recorded in the form of base + increment, which can further save storage space. The file tail is the data area (value data), which is used to store the value.
[0109] The delete file records the versions that need to be deleted during the LSM tree compression process (compaction). It only records the versions, so its data size will be very small. In addition, a delete journal can also be set in the target storage engine. The delete journal is used to record the versions that need to be cleaned (Garbage Collection) of all segments in the target storage engine. Since compaction is performed in the LSM tree, this file can be modified at any time and then asynchronously batch updated to the delete file corresponding to each segment. In this way, during compaction, it is only necessary to frequently read and write a delete journal, and there is no need to frequently read and write the delete file of each segment, thereby greatly reducing the disk IO pressure and improving the read and write performance.
[0110] Furthermore, the target storage engine is also equipped with three access interfaces: SET: write a value; GET: read a value, and DELETE: delete a value. The key of the three interfaces must be the version in the encoding key, so that the corresponding value can be uniquely obtained in the target storage engine by using the global uniqueness of the version.
[0111] The data processing method includes the following steps.
[0112] In step S11, at least one primary key of the data to be written is encoded to obtain an encoded key, and verification information of a value corresponding to the encoded key is generated; the encoded key includes a global version number corresponding to the primary key.
[0113] The data to be written can be in the form of key-value pairs. When writing key-value pairs, the primary key can be encoded separately to obtain the encoded key containing the version, and then the meta corresponding to the encoded key is generated. Since the key and the value are also corresponding, the value and the meta are also corresponding. The verification information is used to locate the storage location of the value corresponding to the encoded key.
[0114] In this embodiment, encoding at least one primary key of the data to be written to obtain the encoded key includes:
[0115] Generate a global version number corresponding to the at least one primary key key;
[0116] The at least one primary key is encoded, and the encoded primary key is encapsulated with the global version number to obtain an encoded key.
[0117] Specifically, when writing a key-value pair, in order to facilitate the construction of a snapshot, a globally incremented version can be generated when encoding at least one primary key, and then at least one primary key in the key-value pair can be encoded, and the encoded primary key and the version can be encapsulated to obtain an encoded key containing the version.
[0118] It should be noted that version is a globally unique and increasing serial number. Even if the same primary key is written multiple times, a coding key containing different versions will be generated. In other words, one primary key can correspond to multiple coding keys. The primary key in each coding key is the same, but the version is different. LSM tree can easily create snapshots using version.
[0119] Furthermore, the value in the key-value pair will be written into the target storage engine, so a first identifier can also be generated as identification information.
[0120] In step S12, the encoding key and the verification information are written into the memory table of the LSM tree.
[0121] Specifically, after obtaining the encoding key, the encoding key is written into the key position in the memory table, and the meta and the first identifier are written into the value position in the memory table.
[0122] In step S13, the global version number and the value corresponding to the primary key key are written in batches into segments in the target storage engine, and the file identifiers of the storage files corresponding to the segments correspond to the global version number.
[0123] Since the primary key and value in the key-value are written separately, when writing the value, in order to ensure that the primary key and the value are one-to-one corresponding, the version and value corresponding to the primary key also need to be written in batches to the segment in the target storage engine. The file identifier of the storage file corresponding to the segment corresponds to the global version number. In addition, the segment can also be recorded in the segment table to facilitate locating the segment through the segment table when reading.
[0124] In this embodiment, writing the value corresponding to the global version number and the primary key key into the fragment in the target storage engine includes:
[0125] Write the value into the data area, and obtain the offset and data size of the value in the data area;
[0126] The global version number, the offset and the data size are written into the index area.
[0127] Specifically, the value is written to the value data in the segment. After writing is completed, the offset and size of the value in the data area are obtained, and then the veriosn, offset and size are written to the index in the same segment.
[0128] It should be noted that a segment can write multiple values, so when writing a value to a segment, just choose a segment with storage space.
[0129] Furthermore, the segment table may include the segment file name (that is, the file identifier of the storage file corresponding to the segment), the starting version, and the ending version. The segment file name may be named using the corresponding version range. For example, if the starting version of segment A is 1234 and the ending version is 3456, then the file name of segment A may be 1234-3456. In addition, other naming methods of the file name are also applicable to this embodiment. In actual applications, they may be set according to actual needs, and this embodiment does not limit this.
[0130] Moreover, low-frequency writing will generate many smaller segments. In this case, if each segment is recorded in the segment table, the space of the segment table will be wasted. For this case, multiple segments with adjacent versions can be virtualized into a larger segment (not merged), and then the virtual segments are recorded in the segment table. In other words, one record in the segment table can correspond to multiple segments.
[0131] In this embodiment, before writing the value corresponding to the global version number and the primary key key into the fragment in the target storage engine, the method further includes:
[0132] Detect whether the data size of the value exceeds the data size threshold;
[0133] If yes, then writing the global version number and the value corresponding to the primary key key into the fragment in the target storage engine;
[0134] If not, write the value into the LSM tree.
[0135] Specifically, before writing the value into the target storage engine, it is also possible to detect whether the data size of the value to be written exceeds the preset value data size threshold. If the data size of the value to be written does not exceed the preset value data size threshold, then the value can be directly written into the LSM tree, that is, the key-value pair is written into the LSM tree together.
[0136] After step S13, the following steps may also be included: Figure 3 shown.
[0137] In step S31, the encoding key and the verification information are read from the memory table.
[0138] Specifically, when reading data, the encoding key and corresponding verification information can be read from the memory table of the LSM tree, where the encoding key includes the primary key and the corresponding version.
[0139] In step S32, it is detected whether the encoding key has a corresponding value.
[0140] After reading the encoding key, you can check whether the value corresponding to the encoding key is stored in the target storage engine. If so, you can continue to read the value. If not, you can report an error.
[0141] In this embodiment, the step of detecting whether the encoding key has a corresponding value includes:
[0142] obtaining identification information from the location;
[0143] If the identification information is the first identification, the encoding key has a corresponding value; if the identification information is the second identification, the encoding key has no corresponding value.
[0144] Specifically, when writing data, the verification information and identification information are written to the value position in the LSM tree. Therefore, when reading, in addition to reading the encoding key and verification information, the identification information can also be read. If the identification information is the first identification, the value corresponding to the encoding key is stored in the target storage engine; if the identification information is the second identification, the value corresponding to the encoding key is not stored in the target storage engine.
[0145] In step S33, if the encoding key has a corresponding value, the corresponding value is read from the fragment based on the global version number and the verification information.
[0146] If the identification information is the first identification, then the value corresponding to the version in the encoding key can be read from the target storage engine.
[0147] In this embodiment, when the segment is recorded in a preset segment list, reading a corresponding value from the segment based on the global version number and the verification information includes:
[0148] Determine the target segment in the segment list by using the global version number;
[0149] Obtain the target offset and target data size from the index area of the target segment, and use the verification information to locate and determine the location information of the stored value;
[0150] The target offset, the target data size and the location information are used to read the corresponding value from the data area.
[0151] Specifically, when the segment is recorded in the preset segment table, the segment is obtained from the segment table, and the version is obtained from the encoding key. Then, the version is used to locate the target segment in the segment table, and the target offset and target size are obtained from the index of the target segment. The meta is used to locate the storage location of the value corresponding to the encoding key to obtain the location information of the stored value. In this way, the value corresponding to the encoding key can be read from the value data of the segment using the target offset, target size and location information.
[0152] In the above way, the target storage engine can index the value corresponding to the primary key through version, without the need to index through the primary key. The data size of the version is much smaller than the data size of the primary key, so a lot of calculations and indexes are avoided, greatly improving the indexing efficiency.
[0153] Moreover, because the data size of the version is very small, it can almost be cached in the memory. Therefore, when reading the value, you only need to read the version from the memory, which greatly improves the reading efficiency.
[0154] After step S13, the following steps may also be included: Figure 4 shown.
[0155] In step S41, it is detected whether there is a duplicate encoding key in the memory table.
[0156] As mentioned above, when writing data, even if the same primary key is written multiple times, a coded key containing different versions will be generated. Therefore, when a primary key corresponds to multiple version values, some version values are useless, which will cause a waste of storage space. Therefore, the target storage engine can be cleaned up for data. During the cleanup, it is possible to detect whether there are duplicate coded keys in the memory table.
[0157] In this embodiment, the detecting whether there is a duplicate encoding key in the memory table includes:
[0158] For any encoding key in the memory table, obtain all corresponding version numbers;
[0159] If the number of all version numbers exceeds the quantity threshold, there are duplicate encoding keys.
[0160] Specifically, since the primary key and version are one-to-one corresponding, even if there are multiple versions of the coding key for the same primary key, the multiple versions are recorded separately. For example, key A-version1, key A-version 2, key A-version 3. Therefore, when cleaning, for any coding key, all corresponding versions can be obtained. If the number of all versions exceeds the preset number threshold, there are duplicate coding keys; if the number of all versions does not exceed the preset number threshold, there are no duplicate coding keys. For example, the versions corresponding to coding key A are obtained as version 1, version 2, and version 3, and the preset number threshold is 1. Then, the number of versions exceeds the number threshold, so there are duplicate coding keys A. For another example, the version corresponding to coding key B is obtained as version 4, and the preset number threshold is 1. Then, the number of versions does not exceed the number threshold, so there are no duplicate coding keys B.
[0161] In step S42, if there are duplicate encoding keys, the target encoding key to be cleaned up and the target version number corresponding to the target encoding key are determined, and the target version number is recorded in the deletion file of the target segment.
[0162] If duplication of any encoding key is detected, the target encoding key that needs to be cleaned up can be determined, and the target version corresponding to the target encoding key can be determined, and the target version can be recorded in the delete file of the target segment corresponding to the target version.
[0163] In this embodiment, recording the target version number in the deletion file of the target segment includes:
[0164] Obtain the target version number corresponding to the target code key, and record the target version number in the deletion log;
[0165] When it is detected that the preset update condition is met, the target version number in the deletion log is recorded in the deletion file of the corresponding target segment.
[0166] Specifically, after determining the target encoding key, the target version corresponding to the target encoding key can be obtained. If there are multiple target encoding keys, then multiple target versions can be obtained accordingly, and then each target version is recorded in the delete journal. When it is detected that the preset update conditions are met, each target version in the delete journal is recorded in the delete file of the corresponding target segment.
[0167] For example, it is detected that the target versions corresponding to the target coding key A are version 1 and version 2, version 1 and the corresponding value are stored in segment 1, version 2 and the corresponding value are stored in segment 2, and the target versions corresponding to the target coding key B are version 3 and version 4, version 3 and the corresponding value are stored in segment 3, and version 4 and the corresponding value are stored in segment 4. Then, after version 1, version 2, version 3 and version 4 are recorded in the delete journal, when it is detected that the preset update conditions are met, version 1 is recorded in the delete file in segment 1, version 2 is recorded in the delete file in segment 2, version 3 is recorded in the delete file in segment 3, and version 4 is recorded in the delete file in segment 4.
[0168] Among them, the preset update conditions include but are not limited to: after a preset time interval (for example, using the delete journal to update the delete file every 1 hour), the number of records in the delete journal exceeds a preset threshold, the file size of the delete journal exceeds a size threshold, etc. In actual applications, the specific method of setting the update conditions can be based on actual needs, and this embodiment does not limit this.
[0169] In step S43, the corresponding value is deleted from the target storage engine according to the deletion file.
[0170] After the delete file of each segment is updated, the target storage engine can delete the value corresponding to the version that needs to be cleaned up according to the delete file.
[0171] In this embodiment, when the segment is recorded in a preset segment list, deleting the corresponding value from the target storage engine according to the deletion file includes:
[0172] Determining whether the target segment needs to be cleaned up according to the deleted file;
[0173] If the target segment needs to be cleaned, then the target value in the target segment that does not need to be cleaned is determined;
[0174] Generate a new segment and write the target value into the new segment;
[0175] Deleting the target segment;
[0176] The new segment is recorded in a preset segment list, and the target segment is deleted from the segment list.
[0177] Specifically, after the delete file of the target segment is updated, the delete file is checked to determine whether the target segment needs to be cleaned. If the target segment needs to be cleaned, the segment is recorded in the preset segment table, and the segment is obtained from the segment table, and the target value that does not need to be cleaned in the target segment is determined. Then a new segment is generated, and the target value that does not need to be cleaned is written into the new segment. Then the target segment is deleted, and the new segment is recorded in the segment table, and the target segment is deleted from the segment table.
[0178] In this embodiment, determining whether the target segment needs to be cleaned up according to the deleted file includes:
[0179] Detect whether the number of value values that need to be cleaned up in the target segment exceeds a number ratio threshold. If it exceeds the number ratio threshold, the target segment needs to be cleaned up;
[0180] and / or,
[0181] Check whether the total data size of the value that needs to be cleaned in the target segment exceeds the size ratio threshold. If it exceeds the size ratio threshold, the target segment needs to be cleaned.
[0182] Specifically, when detecting the delete file, it can be detected whether the number of value values that need to be cleaned in the target segment exceeds the number ratio threshold. If the number of value values that need to be cleaned exceeds the number ratio threshold, the target segment needs to be cleaned. For example, if the number of value values that need to be cleaned in the target segment accounts for 62.3%, which exceeds the number ratio threshold of 60%, then the target segment needs to be cleaned.
[0183] You can also detect whether the total data size of the value values that need to be cleaned in the target segment exceeds the size ratio threshold. If the total data size of the value values that need to be cleaned exceeds the size ratio threshold, the target segment needs to be cleaned. For example, the total data size of all value values in the target segment is 1000M, and the total data size of the value values that need to be cleaned is 823M, which exceeds the size ratio threshold of 80%, then the target segment needs to be cleaned.
[0184] It should be noted that the target segment can be cleaned if either of the above two methods is met. If neither of the two methods is met, the target segment does not need to be cleaned. Moreover, in addition to the above two methods, other methods for cleaning the target segment are also applicable to this embodiment. In practical applications, they can be set according to actual needs, and this embodiment does not limit this.
[0185] In this embodiment, determining the target value that does not need to be cleaned in the target segment includes:
[0186] Check whether the snapshot version number used to generate the snapshot exists in the LSM tree;
[0187] If a snapshot version number exists, the value corresponding to the snapshot version number and the value corresponding to the version number not recorded in the deletion file of the target segment are used as target values that do not need to be cleaned up.
[0188] Specifically, when a snapshot is generated for the value of a certain version, if the snapshot is not released, the value of the version will be occupied and cannot be cleaned up. Therefore, when determining the target value, it is necessary to detect whether there is a version used to generate the snapshot (recorded as snapshot version) in the LSM tree. If so, the value corresponding to the snapshot version and the value corresponding to the version number not recorded in the deletion file of the target segment are used as target values that do not need to be cleaned up.
[0189] In this way, during the execution of Garbage Collection, it can be executed according to the delete journal without relying on the LSM tree. Moreover, after the execution of Garbage Collection is completed, the LSM tree does not need to be modified.
[0190] In this embodiment, at least one primary key of the data to be written is encoded to obtain the encoded key, and verification information of the value corresponding to the encoded key is generated; the encoded key includes a global version number corresponding to the primary key, and the verification information is used to locate the storage position of the value corresponding to the encoded key; the encoded key and the verification information are written into the memory table of the LSM tree; the global version number and the value corresponding to the primary key are batch-written into the fragments in the target storage engine, and the file identifier of the storage file corresponding to the fragment corresponds to the global version number. In this way, although the memory table MemTable of the LSM tree is provided with a key position for storing the key and a value position for storing the value, the value is not stored in the value position but the verification information is stored in the value position. Since the data size of the verification information is much smaller than the data size of the value, the data size of the MemTable can be greatly reduced, thereby saving the storage space of the LSM tree. In addition, since the value is not stored in the LSM tree, when reading and writing the LSM tree, the write amplification of the LSM tree is greatly improved, the disk IO pressure is reduced, and the read and write performance is improved. In addition, since the target storage engine does not need to store the key corresponding to the value, the storage space of the target storage engine is greatly saved.
[0191] Figure 5 is a block diagram of a data processing device according to an exemplary embodiment. Figure 5 The device includes an encoding unit 511, a generating unit 512, a writing unit 513 and a storage unit 514.
[0192] The encoding unit 511 is configured to encode at least one primary key key of the data to be written to obtain an encoded key;
[0193] The generating unit 512 is configured to generate verification information of the value corresponding to the encoding key; the encoding key includes a global version number corresponding to the primary key, and the verification information is used to locate the storage location of the value corresponding to the encoding key;
[0194] The writing unit 513 is configured to write the encoding key and the verification information into the memory table of the LSM tree;
[0195] The storage unit 514 is configured to write the global version number and the value corresponding to the primary key key in batches into segments in the target storage engine, and the file identifier of the storage file corresponding to the segment corresponds to the global version number.
[0196] Optionally, the encoding unit is specifically configured as follows:
[0197] Generate a global version number corresponding to the primary key key;
[0198] The primary key is encoded, and the encoded primary key is encapsulated with the global version number to obtain the encoded key.
[0199] Optionally, the segment includes a value file, and the value file includes an index area and a data area;
[0200] The writing unit is specifically configured as follows:
[0201] Write the value into the data area, and obtain the offset and data size of the value in the data area;
[0202] The global version number, the offset and the data size are written into the index area.
[0203] Optionally, also include:
[0204] A first reading unit is configured to read the encoding key and the verification information from the memory table;
[0205] A first detection unit is configured to detect whether the encoding key has a corresponding value;
[0206] The second reading unit is further configured to read the corresponding value from the fragment based on the global version number and the verification information if there is a corresponding value for the encoding key.
[0207] Optionally, identification information is stored at a position for storing value in the LSM tree, and the identification information includes a first identification and a second identification, the first identification indicates that a corresponding value is stored in the fragment, and the second identification indicates that a corresponding value is not stored in the fragment;
[0208] The detection unit is specifically configured as follows:
[0209] obtaining identification information from the location;
[0210] If the identification information is the first identification, the encoding key has a corresponding value; if the identification information is the second identification, the encoding key has no corresponding value.
[0211] Optionally, in a case where the segment is recorded in a preset segment list, the second reading unit is specifically configured to:
[0212] Determine the target segment in the segment list by using the global version number;
[0213] Obtain the target offset and target data size from the index area of the target segment, and use the verification information to locate and determine the location information of the stored value;
[0214] The target offset, the target data size and the location information are used to read the corresponding value from the data area.
[0215] Optionally, also include:
[0216] A second detection unit is configured to detect whether there is a duplicate encoding key in the memory table;
[0217] a determination unit configured to determine a target encoding key to be cleaned up and a target version number corresponding to the target encoding key if there is a duplicate encoding key;
[0218] a recording unit configured to record the target version number in a deletion file of the target segment;
[0219] The deleting unit is configured to delete the corresponding value from the target storage engine according to the deletion file.
[0220] Optionally, the segment further includes a deletion file, and the deletion file is used to record the version number to be cleaned up in the segment;
[0221] The recording unit is specifically configured as follows:
[0222] Obtain the target version number corresponding to the target code key, and record the target version number in the deletion log;
[0223] When it is detected that the preset update condition is met, the target version number in the deletion log is recorded in the deletion file of the corresponding target segment.
[0224] Optionally, in the case where the segment is recorded in a preset segment list, the deleting unit includes:
[0225] A first determining subunit is configured to determine whether the target segment needs to be cleaned up according to the deleted file;
[0226] A second determining subunit is configured to determine a target value in the target segment that does not need to be cleaned if the target segment needs to be cleaned;
[0227] A generating subunit generates a new segment and writes the target value into the new segment;
[0228] a deletion subunit configured to delete the target fragment;
[0229] The recording subunit is configured to record the new segment into a preset segment list and delete the target segment from the segment list.
[0230] Optionally, the second determining subunit is specifically configured to:
[0231] Check whether the snapshot version number used to generate the snapshot exists in the LSM tree;
[0232] If a snapshot version number exists, the value corresponding to the snapshot version number and the value corresponding to the version number not recorded in the deletion file of the target segment are used as target values that do not need to be cleaned up.
[0233] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.
[0234] Figure 6 6 is a block diagram of a device 600 for data processing according to an exemplary embodiment. For example, the device 600 may be provided as a server. Figure 6, the apparatus 600 includes a processing component 622, which further includes one or more processors, and a memory resource represented by a memory 632 for storing instructions, such as an application, that can be executed by the processing component 622. The application stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 622 is configured to execute instructions to perform the above-mentioned method data processing.
[0235] The device 600 may also include a power supply component 626 configured to perform power management of the device 600, a wired or wireless network interface 650 configured to connect the device 600 to a network, and an input / output (I / O) interface 658. The device 600 may operate based on an operating system stored in the memory 632, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.
[0236] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed in this disclosure. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0237] It should be understood that the present invention is not limited to the exact construction that has been described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A data processing method, characterized in that: include: Encode at least one primary key of the data to be written to obtain the encoded key, and generate verification information of the value corresponding to the encoded key; The encoding key includes a global version number corresponding to the primary key, and the verification information is used to locate the storage location of the value corresponding to the encoding key; Writing the encoding key and the verification information into the memory table of the LSM tree; The global version number and the value corresponding to the primary key key are written in batches into the segments in the target storage engine, the file identifiers of the storage files corresponding to the segments correspond to the global version number, and the segments are recorded in a preset segment list; The method further comprises: Detect whether there is a duplicate encoding key in the memory table; If there are duplicate encoding keys, determine the target encoding key to be cleaned up and the target version number corresponding to the target encoding key, and record the target version number in the deletion file of the target segment; Deleting the corresponding value from the target storage engine according to the deletion file; The segment further includes a deletion file, and the deletion file is used to record the version number to be cleaned up in the segment; The step of recording the target version number in a deletion file of the target segment includes: Obtain the target version number corresponding to the target code key, and record the target version number in the deletion log; When it is detected that the preset update condition is met, the target version number in the deletion log is recorded in the deletion file of the corresponding target segment.
2. The data processing method according to claim 1, characterized in that: The step of encoding at least one primary key of the data to be written to obtain the encoded key includes: Generate a global version number corresponding to the at least one primary key key; The at least one primary key is encoded, and the encoded primary key is encapsulated with the global version number to obtain an encoded key.
3. The data processing method according to claim 1, characterized in that: The fragment includes a value file, and the value file includes an index area and a data area; The step of writing the global version number and the value corresponding to the primary key key into a fragment in the target storage engine includes: Write the value into the data area, and obtain the offset and data size of the value in the data area; The global version number, the offset and the data size are written into the index area.
4. The data processing method according to claim 1, characterized in that: Also includes: Read the encoding key and the verification information from the memory table; Check whether the encoding key has a corresponding value; If the encoding key has a corresponding value, the corresponding value is read from the fragment based on the global version number and the verification information.
5. The data processing method according to claim 4, characterized in that: The position for storing the value in the LSM tree stores identification information, the identification information including a first identification and a second identification, the first identification indicating that the corresponding value is stored in the fragment, and the second identification indicating that the corresponding value is not stored in the fragment; The detecting whether the encoding key has a corresponding value includes: obtaining identification information from the location; If the identification information is the first identification, the encoding key has a corresponding value; if the identification information is the second identification, the encoding key has no corresponding value.
6. The data processing method according to claim 4, characterized in that: In a case where the fragment is recorded in a preset fragment list, reading a corresponding value from the fragment based on the global version number and the verification information includes: Determine the target segment in the segment list by using the global version number; Obtain the target offset and target data size from the index area of the target segment, and use the verification information to locate and determine the location information of the stored value; The target offset, the target data size and the location information are used to read the corresponding value from the data area.
7. The data processing method according to claim 1, characterized in that: In the case where the segment is recorded in a preset segment list, deleting the corresponding value from the target storage engine according to the deletion file includes: Determining whether the target segment needs to be cleaned up according to the deleted file; If the target segment needs to be cleaned, then the target value in the target segment that does not need to be cleaned is determined; Generate a new segment and write the target value into the new segment; Deleting the target segment; The new segment is recorded in a preset segment list, and the target segment is deleted from the segment list.
8. The data processing method according to claim 7, characterized in that: The determining of the target value that does not need to be cleaned in the target segment includes: Check whether the snapshot version number used to generate the snapshot exists in the LSM tree; If a snapshot version number exists, the value corresponding to the snapshot version number and the value corresponding to the version number not recorded in the deletion file of the target segment are used as target values that do not need to be cleaned up.
9. A data processing device, characterized in that: include: An encoding unit, configured to encode at least one primary key of the data to be written to obtain an encoding key; A generating unit, configured to generate verification information of a value corresponding to the encoding key; The encoding key includes a global version number corresponding to the primary key, and the verification information is used to locate the storage location of the value corresponding to the encoding key; A writing unit, configured to write the encoding key and the verification information into a memory table of the LSM tree; A storage unit is configured to write the global version number and the value corresponding to the primary key key into segments in a target storage engine in batches, wherein the file identifier of the storage file corresponding to the segment corresponds to the global version number, and the segment is recorded in a preset segment list; The fragment also includes a deletion file, and the deletion file is used to record the version number to be cleaned up in the fragment; the device also includes: A second detection unit is configured to detect whether there is a duplicate encoding key in the memory table; a determination unit configured to determine a target encoding key to be cleaned up and a target version number corresponding to the target encoding key if there is a duplicate encoding key; a recording unit configured to record the target version number in a deletion file of the target segment; a deleting unit configured to delete the corresponding value from the target storage engine according to the deletion file; The segment further includes a deletion file, and the deletion file is used to record the version number to be cleaned up in the segment; The recording unit is specifically configured as follows: Obtain the target version number corresponding to the target code key, and record the target version number in the deletion log; When it is detected that the preset update condition is met, the target version number in the deletion log is recorded in the deletion file of the corresponding target segment.
10. The data processing device according to claim 9, characterized in that: The encoding unit is specifically configured as follows: Generate a global version number corresponding to the primary key key; The primary key is encoded, and the encoded primary key is encapsulated with the global version number to obtain the encoded key.
11. The data processing device according to claim 9, characterized in that: The fragment includes a value file, and the value file includes an index area and a data area; The writing unit is specifically configured as follows: Write the value into the data area, and obtain the offset and data size of the value in the data area; The global version number, the offset and the data size are written into the index area.
12. The data processing device according to claim 9, characterized in that: Also includes: A first reading unit is configured to read the encoding key and the verification information from the memory table; A first detection unit is configured to detect whether the encoding key has a corresponding value; The second reading unit is further configured to read the corresponding value from the fragment based on the global version number and the verification information if there is a corresponding value for the encoding key.
13. The data processing device according to claim 12, characterized in that: The position for storing the value in the LSM tree stores identification information, the identification information including a first identification and a second identification, the first identification indicating that the corresponding value is stored in the fragment, and the second identification indicating that the corresponding value is not stored in the fragment; The detection unit is specifically configured as follows: obtaining identification information from the location; If the identification information is the first identification, the encoding key has a corresponding value; if the identification information is the second identification, the encoding key has no corresponding value.
14. The data processing device according to claim 12, characterized in that: In the case where the segment is recorded in a preset segment list, the second reading unit is specifically configured to: Determine the target segment in the segment list by using the global version number; Obtain the target offset and target data size from the index area of the target segment, and use the verification information to locate and determine the location information of the stored value; The target offset, the target data size and the location information are used to read the corresponding value from the data area.
15. The data processing device according to claim 14, characterized in that: In the case where the segment is recorded in a preset segment list, the deleting unit includes: A first determining subunit is configured to determine whether the target segment needs to be cleaned up according to the deleted file; A second determining subunit is configured to determine a target value in the target segment that does not need to be cleaned if the target segment needs to be cleaned; A generating subunit generates a new segment and writes the target value into the new segment; a deletion subunit configured to delete the target fragment; The recording subunit is configured to record the new segment into a preset segment list and delete the target segment from the segment list.
16. The data processing device according to claim 15, characterized in that: The second determining subunit is specifically configured as follows: Check whether the snapshot version number used to generate the snapshot exists in the LSM tree; If a snapshot version number exists, the value corresponding to the snapshot version number and the value corresponding to the version number not recorded in the deletion file of the target segment are used as target values that do not need to be cleaned up.
17. An electronic device, characterized in that: include: A processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the data processing method according to any one of claims 1 to 8 is implemented.
18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.
19. A computer program product comprising a computer program or computer instructions, characterized in that When the computer program or computer instruction is executed by a processor, the data processing method according to any one of claims 1 to 8 is implemented.
