Method for partitioning memory, electronic device, and storage medium
By partitioning the memory and selecting a partition key to partition the memory file layer, the problem of adjusting the merging strategy in the log structure merging tree is solved, improving query efficiency and disk utilization, and increasing overall throughput.
Patent Information
- Application Number
- PCT/CN2024/092795
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-17
- Filing Date
- 2024-05-13
- Publication Date
- 2024-11-21
AI Technical Summary
Existing technologies make it difficult to adjust merging strategies for specific data in log structure merging trees, resulting in high IO overhead for query operations, low disk bandwidth utilization, and insufficient overall throughput.
By partitioning the memory and selecting a partition key to divide the memory file layer into virtual partitions, and then merging these partitions, the adjustment area of the merging strategy is reduced, thereby improving hard disk bandwidth utilization and query efficiency.
It effectively reduces the read and write overhead of query operations, improves the overall throughput of the log structure merge tree, and enhances the query efficiency and resource utilization of the hard disk.
Smart Images

Figure CN2024092795_21112024_PF_FP_ABST
Abstract
Description
Method, electronic device and storage medium for partitioning memory
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 17, 2023, with application number 202310561033.2 and invention name “Method, electronic device and storage medium for partitioning memory”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present disclosure relates generally to the field of computers, and more particularly to an operating method, an electronic device, and a computer storage medium for partitioning a memory. Background Art
[0003] The Log-Structured Merge Tree (LSM-Tree) is a write-optimized data structure widely used in database storage engines. It provides key-value pair write, delete, modify, and query capabilities. When writing data, the LSM-Tree first writes the data to memory. Once the data volume reaches a predetermined amount, it is persistently stored on the hard disk in the form of an ordered file. The LSM-Tree stores data on the hard disk in layers, with each merged data layer being written to the next layer. When querying data on the hard disk, a binary search is typically used to search until the desired data is found.
[0004] Summary of the Invention
[0005] According to example embodiments of the present disclosure, a method for partitioning a memory, an electronic device, and a computer storage medium are provided.
[0006] In a first aspect of the present disclosure, a method for partitioning memory is provided, comprising storing received data in the form of a file in a file layer of the memory. The method may further comprise determining whether a data volume of the data in the file layer exceeds a data volume threshold. The method may further comprise selecting at least one key from a plurality of key-value pairs corresponding to the file as at least one partition key, and partitioning the file layer of the memory based on the at least one partition key to obtain a plurality of virtual partitions.
[0007] In a second aspect of the present disclosure, an electronic device is provided, comprising: at least one processing unit; and at least one memory, the at least one memory being coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, which instructions, when executed by the at least one processing unit, enable the electronic device to perform the method described in accordance with the first aspect of the present disclosure.
[0008] In a third aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has machine-executable instructions stored thereon. When the machine-executable instructions are executed by a device, the device may perform the method described according to the first aspect of the present disclosure.
[0009] According to a fourth aspect of the present disclosure, a computer program product is provided, comprising computer-executable instructions, wherein the computer-executable instructions implement the method described in the first aspect of the present disclosure when executed by a processor.
[0010] The purpose of providing the summary of the invention section is to introduce a series of concepts in a simplified form, which will be further described in the detailed description below. The summary of the invention section is not intended to identify the key features or essential features of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:
[0012] FIG1 shows a schematic diagram of an example system in which embodiments of the present disclosure can be implemented;
[0013] FIG2 is a flowchart illustrating a method for partitioning a memory according to an embodiment of the present disclosure;
[0014] FIG3 shows a flow chart of an exemplary method for triggering physical partitioning of a hard disk according to an embodiment of the present disclosure;
[0015] FIG4 illustrates a schematic process for physically partitioning a hard disk according to an embodiment of the present disclosure;
[0016] 5A-5C are diagrams illustrating a comparison of a file sequence before a merge operation is performed and a file sequence after the merge operation is performed and physical partition division is completed according to an embodiment of the present disclosure;
[0017] FIG6 shows a schematic block diagram of an example apparatus according to some embodiments of the present disclosure; and
[0018] FIG7 illustrates a block diagram of an example device that may be used to implement embodiments of the present disclosure. DETAILED DESCRIPTION
[0019] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.
[0020] Data stored in a log-structured merge tree (LSM-Tree) is stored in layers in memory and on disk, and each merge results in the data being written to the next layer. For example, if data written to the disk is in layer L0, it will be written to layer L1 after being merged once, to layer L2 after being merged twice, and so on. When querying data stored in the LSM-Tree, computing device 160 queries sequentially in layer order, for example, first querying layer L0, then layer L1, and finally layer L2, until the desired target data is found.
[0021] A core feature of LSM-Trees is the compaction operation. When the amount of data in memory reaches a threshold, a compaction operation is used to write the data from memory to disk. Furthermore, files on disk must be consolidated and written to the next level. The compaction operation is highly correlated with query load. An appropriate compaction strategy can reduce read and write I / O overhead during queries. Therefore, selecting an appropriate compaction strategy for data stored in LSM-Trees is crucial. Typically, computing devices apply a single compaction strategy to global data on disk. However, due to the large volume of data on disks, the scope of compaction strategy adjustment is relatively large, making it difficult to tailor the compaction strategy to specific data (for example, data that is frequently queried). Consequently, the I / O overhead saved by compaction for queries is limited, and query I / O overhead remains high. Although some research has explored techniques for selecting and adjusting compaction strategies for individual levels on disk, these solutions remain too granular and do not significantly reduce or improve query I / O overhead.
[0022] In view of this, an embodiment of the present disclosure provides a method for partitioning memory, comprising: storing received data in the form of a file on a file layer of the memory; determining that the data volume of the data on the file layer exceeds a data volume threshold; selecting at least one key from a plurality of key-value pairs corresponding to the file as at least one partition key; and partitioning the file layer of the memory based on the at least one partition key to obtain a plurality of virtual partitions. By adopting the method according to an embodiment of the present disclosure, the area to which the adjustment merge strategy is applied can be effectively narrowed, thereby making it easier to adjust the merge strategy for specific data, greatly saving the read and write overhead of query operations, and improving the utilization rate of hard disk bandwidth, thereby improving the overall throughput of the log structure merge tree.
[0023] FIG1 illustrates a schematic diagram of an example system 100 in which embodiments of the present disclosure can be implemented. System 100 may include a computing device 160. Computing device 160 may include a memory 162 and a hard disk 164. Furthermore, although not shown, those skilled in the art will appreciate that computing device 160 may also include other devices such as a processor, input / output devices, and the like.
[0024] In some embodiments, computing device 160 may use a log-structured merge tree (LSM-tree) data structure to store data in memory 162 and hard disk 164. When using a log-structured merge tree, computing device 160 may first write and store data in memory 162. As shown in FIG1 , data is written and stored in memory 162 in the form of a file. When the amount of data reaches a threshold (e.g., a threshold set by the user), computing device 160 may sequentially persist the data to hard disk 164 in the form of an ordered sequence of files. Based on the log-structured merge tree, computing device 160 may store the data on hard disk 164 in layers, as shown in FIG1 as L0, L1, and L2. Each time data is merged, it is written to the next layer. For example, data just written to hard disk 164 may be stored in L0, then written to L1 after one merge, then written to L2 after two merges, and so on. In addition, files in memory 162 and hard disk 164 are stored in a key-value format. A value is indexed by a key (this value is referred to as the "index value" below). Furthermore, each key also has its own value or symbol (hereinafter referred to as the "key value"), which is used for sorting. During a query, by querying a key, you can find the index value of the key in the corresponding file through the key's index.
[0025] Although FIG1 shows three layers in hard disk 164, it is understood that the hard disk may include a greater or fewer number of layers, and the present disclosure does not limit the number of layers in the hard disk. In addition, FIG1 shows files as boxes, and the number of files in memory 162 and hard disk 164 can be any suitable number, and the present disclosure does not limit the number of files stored in computing device 160.
[0026] In some embodiments, the computing device 160 shown in Figure 1 may include, but is not limited to, a personal computer, a server computer, a handheld or laptop device, a mobile device (such as a mobile phone, a personal digital assistant (PDA), a media player, etc.), a vehicle-mounted device, a multi-processor system, consumer electronics, wearable electronic devices, smart home devices, a minicomputer, a mainframe computer, an edge computing device, a distributed computing environment including any one of the above systems or devices, etc.
[0027] In some embodiments, the computing device 160 may store the received data in the form of a file on the file layer of the memory 162. The computing device 160 may determine that the data volume of the data on the file layer exceeds a data volume threshold. The computing device 160 may also select at least one key from a plurality of key-value pairs corresponding to the file as at least one partition key, and partition the file layer of the memory 162 based on the at least one partition key to obtain a plurality of virtual partitions. By adopting the operating method according to the embodiment of the present disclosure, the area to which the adjustment of the merge strategy is applied can be effectively narrowed, so that the adjustment of the merge strategy for specific data can be more easily implemented, which greatly saves the read and write overhead of the query operation, improves the utilization rate of the hard disk bandwidth, and improves the overall throughput of the log structure merge tree.
[0028] A block diagram of an example system 100 in which embodiments of the present disclosure can be implemented is described above in conjunction with FIG1 . A flow chart of a method 200 for partitioning memory according to an embodiment of the present disclosure is described below in conjunction with FIG2 . The actions involved in method 200 are described below in conjunction with computing device 160 as shown in FIG1 . For example, in some embodiments, method 200 may be performed by computing device 160. It should be understood that method 200 may also include additional actions not shown and / or may omit actions shown, and the scope of the present disclosure is not limited in this respect.
[0029] In box 202, the computing device 160 can store the received data in the form of a file on the file layer of the memory. According to the method of the embodiment of the present disclosure, the data structure of the log structure merge tree (LSM-tree) is used to store the data in the form of ordered files in the memory 162 and the hard disk 164. The files in the memory 162 and the hard disk 164 are stored in a key-value manner. The value stored in the file (i.e., the index value) can be indexed via each key. During the query operation, by querying the key, the file corresponding to the key can be queried via the index of the key. During the process of merging the files, the computing device 160 will merge the files in the order of the size of the key values. For example, the computing device 160 can sort the files in the order of the size of the key values, and for files with the same key, during the merge operation, the computing device 160 can retain relatively new data and merge the retained data into the file sequence.
[0030] In block 204, computing device 160 may determine that the amount of data on the file layer exceeds a data amount threshold. In some embodiments, the data amount threshold may be set by the user according to actual application needs, and this disclosure does not limit the size of the data amount threshold.
[0031] In block 206, computing device 160 may select at least one key from the plurality of key-value pairs corresponding to the file as at least one partition key. As described above, files are stored in memory 162 and hard disk 164 in the form of key-value pairs. Each file corresponds to a corresponding key-value pair. Table 1 below illustrates the relationship between files and corresponding key-value pairs:
[0032] Table 1
[0033] As described in Table 1 above, file 1 may include two key-value pairs, namely Key1 and the corresponding index value Value1, and Key2 and the corresponding index value Value2. In some embodiments, the key (key) and the corresponding index value (value) can be characters of any byte size, which is not limited by the present disclosure. In addition, N can be any positive integer, representing the number of files in the corresponding storage area. It can be understood that Table 1 above is only schematic, and the data of the files according to the embodiments of the present disclosure can be any number, and the number of key-value pairs in each file can also be arbitrary, which is not limited by the present disclosure.
[0034] The computing device 160 may select at least one key from the plurality of key-value pairs as a partition key. The partition key may represent a reference key for logically partitioning the memory 162. An exemplary process for determining the partition key will be described in detail below.
[0035] Continuing with FIG. 2 , in block 208, computing device 160 may partition the file layer of memory based on at least one partition key to obtain multiple virtual partitions. Based on the partition key, computing device 160 may logically partition memory 162 to obtain multiple virtual partitions. For example, for a particular partition key, files with a key value greater than the partition key value and files with a key value less than the partition key value are assigned to different partitions. The specific implementation of logically partitioning memory 162 based on the partition key will be described in detail below in conjunction with the accompanying drawings.
[0036] Multiple virtual partitions divided based on partition keys can be mapped to the hard disk. When computing device 160 stores files on hard disk 164, the mapping process between the virtual partitions in memory and the physical partitions on the hard disk can be achieved. This mapping process will be described in detail below with reference to the accompanying drawings.
[0037] Advantageously, by adopting the method of partitioning memory according to the embodiment of the present disclosure, the area where the adjustment merge strategy is applied can be effectively narrowed, thereby making it easier to adjust the merge strategy for specific data, greatly saving the read and write overhead of query operations, and improving the utilization of hard disk bandwidth, thereby improving the overall throughput of the log structure merge tree.
[0038] The computing device 160 may determine at least one partition key in the method 200 for logically partitioning the memory 162 to obtain a plurality of virtual partitions. In some embodiments, the computing device 160 may sample key-value pairs corresponding to files in the memory to obtain a plurality of sampled key-value pairs. For example, the computing device 160 may randomly sample key-value pairs corresponding to files in the memory to obtain a plurality of sampled key-value pairs. The sampled key-value pairs may be represented as KV1, KV2, ..., KV P . Where P is a positive integer.
[0039] The computing device 160 may sort the multiple sampling key-value pairs obtained by sampling. In some embodiments, the computing device 160 sorts the keys in the multiple sampling key-value pairs obtained by sampling in lexicographic order. For example, the computing device 160 may sort the keys in the multiple sampling key-value pairs obtained by sampling according to the size of the key values to form a key-value pair sequence. For example, assuming that the value of key K1 of KV1 is less than the value of key K2 of KV2, then in the arranged key-value pair sequence, key K1 is arranged before key K2 (for example, the sequence is sorted in ascending order from left to right). Assume that an example of a key-value pair sequence Qk can be {KV1, KV2, KV3, ..., KV a}, where the element KV in the key-value pair sequence Qk i Press key Ki Sort the values by size, from left to right in the sequence, K i In addition, in some embodiments, when the key K i When expressing by letters, the computing device 160 can also press the key K i The data are arranged in alphabetical order, and the present disclosure does not limit the specific sorting method.
[0040] In some embodiments, computing device 160 may select at least one key from the sorted key-value pair sequence consisting of multiple sampled key-value pairs as a partition key. The number of partition keys depends on the desired number of virtual partitions. For example, if memory 162 is desired to be divided into M virtual partitions, M keys may be selected from the sorted key-value pair sequence as partition keys. Accordingly, computing device 160 selects M partition keys. When selecting partition keys, computing device 160 may select keys from the key-value pair sequence at predetermined intervals. For example, if four partition keys are desired to divide memory 162 into four virtual partitions, computing device 160 may select four partition keys from the key-value pair sequence Qk, with the four selected partition keys being equally or approximately spaced within the key-value pair sequence Qk. In other words, by selecting partition keys, the sorted key-value pair sequence can be evenly divided. A partition key corresponds to the key at the end of the key-value pair sequence corresponding to a virtual partition. Selecting partition keys in this manner can ensure a relatively balanced number of files within the divided virtual partitions.
[0041] After determining the partition key, computing device 160 can determine that virtual partitioning of memory 162 has been implemented. For example, a partition key corresponds to the key at the end of the key-value pair sequence corresponding to a virtual partition. In some embodiments, computing device 160 can also mark the file layer that implements virtual partitioning, for example, by setting a flag bit or marker to indicate that the file layer has implemented virtual partitioning. Furthermore, computing device 160 can also record the partition key for use when triggering physical partitioning.
[0042] In a similar manner, the computing device 160 may continue to perform similar virtual partitioning operations on other file layers on the memory 162. The specific operation method is similar to the process described above and will not be repeated here for the sake of brevity.
[0043] When receiving a merge operation request, the computing device 160 may trigger the division of the physical partitions. The specific implementation process of dividing the physical partitions will be described below with reference to the accompanying drawings.
[0044] FIG3 illustrates a flowchart of an exemplary method for triggering physical partitioning according to an embodiment of the present disclosure. The method in FIG3 may be performed by the computing device 160 shown in FIG1 . It should be understood that the method 300 may include additional actions not shown and / or may omit actions shown, and the scope of the present disclosure is not limited in this respect.
[0045] At box 302, the computing device 160 may receive a merge request to perform a merge operation. The merge request may be for a file on a certain layer in the memory 162 (hereinafter referred to as a "file to be merged"), and is used to merge the file and write it to the next layer. In response to the merge request, the computing device 160 may determine at 304 whether the operating layer performing the merge operation has a virtual partition. In some embodiments, the computing device 160 may determine whether the operating layer has been logically divided into multiple virtual partitions by reading a record for the operating layer stored in a memory, such as a memory. For example, when the computing device 160 reads an identifier or mark indicating that the operating layer has been divided into virtual partitions from the record, it can be determined that the operating layer already has a virtual partition; otherwise, it is determined that the operating layer has no virtual partition.
[0046] When computing device 160 determines at 304 that the operating layer does not have a virtual partition, it returns to block 302 and continues to receive merge requests. If computing device 160 determines at 304 that the operating layer has a virtual partition, it may proceed to block 306 to perform a merge operation on the to-be-merged files on the operating layer.
[0047] During the physical partitioning operation, the computing device 160 divides the physical partitions based on the partition key. In some embodiments, the computing device 160 obtains key-value pairs corresponding to the files to be merged on the operation layer. The computing device 160 can obtain all key-value pairs corresponding to the files to be merged. The computing device 160 can sort the files to be merged based on the size of the key values in the obtained key-value pairs to obtain a sorted file sequence. The computing device 160 sequentially reads the keys in the key-value pairs corresponding to the sorted file sequence. The computing device 160 compares the values of the read keys with the values of the partition key. When the value of the first key read reaches or equals the value of the first partition key in the partition key, the computing device 160 performs a first abort operation to abort the read operation. The computing device 160 stores the file corresponding to the key read before the first abort operation in the first physical partition. As a result, the first physical partition corresponds to the first virtual partition divided according to the first partition key.
[0048] The computing device 160 may continue to read the remaining keys in the multiple key-value pairs corresponding to the sorted file sequence. The computing device 160 compares the value of the read key with the value of the key in the partition key (for example, the partition key other than the first partition key). The computing device 160 compares the value of the read key with the value of the partition key. When the computing device 160 determines that the value of the read second key reaches or is equal to the value of the second partition key in the partition key, the computing device 160 performs a second abort operation to abort the read operation. The computing device 160 stores the files corresponding to the keys read between the first abort operation and the second abort operation in the second physical partition. Thus, the second physical partition corresponds to the second virtual partition divided according to the second partition key.
[0049] The computing device 160 may continue to perform the above-mentioned read operation and abort operation to write the remaining files to the hard disk 164 and simultaneously implement physical partitioning of the hard disk 164. The specific execution process is similar to the process described above and will not be repeated here for the sake of brevity.
[0050] The following will schematically describe the process of physically partitioning a hard disk according to the above process with reference to FIG4 . FIG4 illustrates a schematic process 400 for physically partitioning a hard disk according to an embodiment of the present disclosure. It will be understood that the number of files and the number of file layers in the hard disk shown in FIG4 are merely schematic, and those skilled in the art may use a greater or lesser number of file data and file layers, and the present disclosure does not limit this.
[0051] As shown in FIG4 , after computing device 160 receives a merge request for a merge operation on files in memory 162, computing device 160 may determine whether the file layer in memory 162 has been logically divided into virtual partitions. If it is determined that memory 160 has multiple virtual partitions at the file layer, computing device 160 may perform a merge operation on the files in memory 162 to write the sorted and / or merged files to hard disk 164 and simultaneously implement physical partitioning of the hard disk.
[0052] As shown in Figure 4, there are 9 files in the memory, file 1 to file 9. For simplicity, assume that each file has a pair of key-value pairs. For example, file 1 has the first key-value pair KV1, file 2 has the second key-value pair KV2, and so on. File 9 has the ninth key-value pair KV9. Assume that the key K in each key-value pair is i(1≤i≤9) has corresponding values, for example, K1=20, K2=21, K3=25, K4=22, K5=28, K6=23, K7=24, K8=27, and K9=26. Furthermore, computing device 160 can know from the record of virtual partitions that the key layer in memory 162 is divided into three virtual partitions, and the corresponding partition keys are 22, 25, and 28.
[0053] If the files to be merged are files 1 through 9, computing device 160 can obtain key-value pairs corresponding to files 1 through 9 to determine the keys associated with files 1 through 9, such as K1 through K9. Computing device 160 can sort the files to be merged based on the size of the obtained key values. For example, based on the exemplary key values described above, the ordered sequence of files, arranged from smallest to largest, can be: {file 1, file 2, file 4, file 6, file 7, file 3, file 9, file 8, file 5}.
[0054] Computing device 160 can read and sort the files sequentially according to the above description. Accordingly, computing device 160 can read the keys of the files sequentially. For example, computing device 160 reads the value (20) of key K1 of file 1 and the value (21) of key (K2) of file 2. Computing device 160 can compare the read key values with the values of the partition keys (values 22, 25, and 28 in this example). Computing device 160 can continue to read file 4. At the same time, computing device 160 reads the value (22) of key (K4) of file 4 and compares the read key (K4) value (22) with the values of the partition keys (values 22, 25, and 28 in this example). When computing device 160 determines that the read key (K4) value (22) is equal to the value (22) of the first partition key in the partition keys, a first abort operation is performed. The computing device 160 merges files 1, 2, and 4 corresponding to the read keys (K1, K2, K4) and writes them to the first physical partition R1 of the hard disk 164, as shown in Figure 4. It can be understood that the first physical partition R1 corresponds to the first virtual partition determined based on the first partition key (e.g., the value is 22).
[0055] The computing device 160 continues to merge the files in the file sequence in sequence. In this example, the computing device 160 continues to read file 6 and the corresponding value (23) of the key (K6). The computing device 160 compares the value (23) of the key (K6) with the values of the remaining partition keys (values 25 and 28 in this example). The value (23) of the key (K6) is less than the value of any partition key. The computing device 160 continues to read file 7 and the corresponding value (24) of the key (K7). The computing device 160 compares the value (24) of the key (K7) with the values of the remaining partition keys (values 25 and 28 in this example). The value (24) of the key (K7) is less than the value of any partition key. The computing device 160 continues to read file 3 and the corresponding value (25) of the key (K3). The computing device 160 compares the value (25) of the key (K3) with the values of the remaining partition keys (values 25 and 28 in this example). Computing device 160 may determine that the value (25) of key (K3) is equal to the value 25 of the second partition key.
[0056] In this case, computing device 160 performs a second abort operation, merging files 6, 7, and 3 corresponding to the read keys (K6, K7, K3) and writing them to the second physical partition R2 of hard disk 164, as shown in FIG4 . It can be understood that the second physical partition R2 corresponds to the second virtual partition determined based on the second partition key (value 25).
[0057] Similar to the above operation, computing device 160 merges files 9, 8, and 5 and writes them to the third physical partition R3 of hard disk 164, as shown in Figure 4. It can be understood that the third physical partition R3 corresponds to the second virtual partition determined based on the third partition key (value 28).
[0058] In addition, in some other embodiments, a partition key may also correspond to the key at the starting end of the key-value pair sequence corresponding to a virtual partition. In this case, the number of partition keys is equal to the difference between the number of virtual partitions desired and 1. For example, when it is desired to divide the memory 162 into M virtual partitions, (M-1) keys can be selected from the sorted key-value pair sequence as partition keys. By selecting the partition keys, the sorted key-value pair sequence can be divided equally. After determining the partition keys, the computing device 160 can determine that the virtual partitioning of the memory 162 has been achieved.
[0059] After determining the partition key, computing device 160 can perform physical partitioning on the hard disk in a similar manner to the process described above. For example, during the physical partitioning operation, computing device 160 can obtain key-value pairs corresponding to the files to be merged at the operation layer. Computing device 160 can obtain all key-value pairs corresponding to the files to be merged. Computing device 160 can sort the files to be merged based on the values of the keys in the obtained key-value pairs to obtain a sorted file sequence. Computing device 160 sequentially reads the keys in the key-value pairs corresponding to the sorted file sequence. Computing device 160 compares the read key values with the partition key value. When the value of the first key read reaches the value of the first partition key in the partition key, computing device 160 performs a first abort operation to abort the read operation. Computing device 160 stores the files corresponding to the keys read before the first abort operation, excluding the first key, in the first physical partition. Thus, the first physical partition corresponds to the first virtual partition divided according to the first partition key.
[0060] The computing device 160 may continue to read the remaining keys in the multiple key-value pairs corresponding to the sorted file sequence. The computing device 160 compares the value of the read key with the value of the key in the partition key (for example, the partition key other than the first partition key). When the computing device 160 compares the value of the read key with the value of the partition key. When the computing device 160 determines that the value of the read second key reaches or is equal to the value of the second partition key in the partition key, the computing device 160 performs a second abort operation to abort the read operation. The computing device 160 stores the files corresponding to the first key and the keys other than the second key read between the first abort operation and the second abort operation in the second physical partition. Thus, the second physical partition corresponds to the second virtual partition divided according to the second partition key.
[0061] The computing device 160 may continue to perform the above-mentioned read operation and abort operation to write the remaining files to the hard disk and simultaneously implement physical partitioning of the hard disk. The specific execution process is similar to the process described above and will not be repeated here for the sake of brevity.
[0062] Figures 5A-5C illustrate a comparison of the file sequence before performing the merge operation and the file sequence after performing the merge operation and completing the physical partition division according to an embodiment of the present disclosure. Figures 5A-5C show schematic comparisons of the merge operation results in three different scenarios. Throughout Figures 5A-5C, the value range of the virtual partition key (also referred to as the "value domain") is Pa[0,200], Pb[201,350], Pc[351,700]. The values of the partition keys can correspond to the values 200, 350, and 700. Exemplarily, the operation objects in files 5A-5C are files at the L1 layer.
[0063] FIG5A illustrates that the key value ranges of the marked ordered file sequences do not span virtual partitions. For example, in FIG5A , before performing the merge operation, the key value range of file 511 is [220, 310], the key value range of file 513 is [80, 170], and the key value range of file 515 is [40, 590]. When performing the merge operation according to the merge operation method described above and performing the physical partition division operation, the computing device 160 can determine that the key value ranges of the file sequences in files 511, 513, and 515 each fall within the current virtual partition. Therefore, the computing device 160 can divide files 511, 513, and 515 into corresponding physical partitions. As shown in FIG5A , file 513′ is divided into the first physical partition, file 511′ is divided into the second physical partition, and file 515′ is divided into the third physical partition. It can be understood that although the value range of the keys of files 511', 513' and 515' is the same as that of files 511, 513 and 515 respectively, the files in files 511', 513' and 515' are arranged in sequence. When the files in files 511, 513 and 515 are not arranged in sequence, the order of files in files 511', 513' and 515' is different from the order of files in files 511, 513 and 515.
[0064] Figure 5B illustrates that the key value ranges of the marked ordered file sequence all span virtual partitions. For example, in Figure 5B , before the merge operation is performed, the key value range of file 521 is [130, 610], the key value range of file 523 is [30, 580], and the key value range of file 525 is [100, 690]. When performing a merge and physical partition division operation according to the merge operation method described above, computing device 160 can determine that the key value ranges of the file sequences in files 521, 523, and 525 all span the current virtual partition. Therefore, computing device 160 sorts and merges files 521, 523, and 525 according to the partition key according to the merge operation described above, and generates sorted files 522, 524, and 526. The key range corresponding to file 522 is [30, 200], the key range corresponding to file 524 is [201, 350], and the key range corresponding to file 526 is [351, 690]. Computing device 160 divides files 522, 524, and 526 into corresponding physical partitions. As shown in Figure 5B, file 522 is divided into the first physical partition, file 524 is divided into the second physical partition, and file 526 is divided into the third physical partition. 5B further illustrates that file 522, file 524, and file 526 are further stored in the next layer due to being merged with files in the lower layer. The present disclosure does not specifically limit the operations in each physical partition.
[0065] FIG5C illustrates that a portion of the key value ranges of the marked ordered file sequence spans across virtual partitions. For example, in FIG5C , before performing the merge operation, the key value range of file 531 is [130, 310], the key value range of file 533 is [80, 330], and the key value range of file 535 is [400, 590]. When performing a merge operation according to the merge operation method described above to perform a physical partition splitting operation, computing device 160 may determine that the key value ranges of file 531 and file 533 span the key value range of the virtual partition. Therefore, computing device 160, in accordance with the merge operation described above, sorts and merges files 531, 533, and 535 according to the partition key, generating sorted files 532, 534, and 535'. The key range corresponding to file 532 is [80, 200], the key range corresponding to file 544 is [201, 330], and the key range corresponding to file 535' is [400, 690]. Although the key value ranges of files 535 and 535' are the same, the order of files within files 535 and 535' may be different. For example, file 535' is a file sorted in order. Computing device 160 partitions files 532, 534, and 535' into corresponding physical partitions. As shown in FIG5C , file 532 is partitioned into the first physical partition, file 534 is partitioned into the second physical partition, and file 535' is partitioned into the third physical partition. 5C further illustrates that the file 532 divided into the first physical partition and the file 534 divided into the second physical partition are further stored in the next layer due to being further merged with the files in the lower layer. The present disclosure does not specifically limit the operations in each physical partition.
[0066] In some embodiments, after completing the physical partitioning, the computing device 160 may further perform merge-related operations on each layer (hereinafter referred to as "operation layer") of each physical partition in the hard disk 160. The computing device 160 may determine a first predicted target value for performing a first operation in a first state associated with a file to be operated on the operation layer in the physical partition within a first time interval and a second predicted target value for performing a second operation in the first state, wherein the first operation and the second operation are related to the merge operation performed on the file in the physical partition. In some embodiments, the first operation may be a first operation based on a Tiering strategy (A tier ), and the second operation can be a second operation based on the Leveling strategy (A levelThe Tiering-based operation skips the merge operation, and accordingly, multiple ordered file sequences can be maintained at each layer. The Leveling-based operation performs the merge operation, and accordingly, only one ordered file sequence is maintained at each layer.
[0067] In some embodiments, the status associated with a file may include, but is not limited to, the number of files, the read penalty associated with the merge operation, and the amount of data in the file within the corresponding time interval. The read penalty indicates that under the same circumstances, the first operation (A) based on the Tiering strategy is used. tier ) Compared with the second operation based on the Leveling strategy (A level ) requires more data to be accessed. In some embodiments, the read penalty can be calculated in units of the access granularity of the data on the hard disk (e.g., storage block). The decision value can be identified as a Q value. The Q value can represent the expected value of the "reward" that can be obtained by performing a corresponding operation or action in a given state. The larger the Q value, the greater the "reward" that can be obtained by performing the corresponding action. Therefore, the Q value can be used as a decision value to determine the operation.
[0068] Furthermore, computing device 160 may determine a reference value for selecting an operation based on the first predicted target value and the second predicted target value, and computing device 160 may also select an operation from the first operation or the second operation based on the reference value for merging the files. Computing device 160 may utilize a neural network to determine the first predicted decision value and the second predicted decision value. In some embodiments, the neural network includes a target network.
[0069] The parameters of the target network can come from a trained, separate update network. The update network can also be a neural network. The update network can be trained on a training sample set. The sample set used to train the update network can include the states corresponding to the files and their Q values. In some embodiments, the computing device 160 can update the Q values in a predetermined Q table and use the updated Q values as samples for training the update network. The Q value update includes an update of the Q value for the first operation and an update of the Q value for the second operation.
[0070] In some embodiments, the computing device 160 may determine the first Q value and the second Q value, respectively, in the state S within the time interval T. For example, the computing device 160 may determine the first Q value and the second Q value by looking up a Q table.
[0071] Since the second operation corresponds to performing a merge operation on the files, performing the second operation can reduce the number of searches for the ordered file sequence. Thus, the computing device 160 can determine one or more revenue values associated with performing the second operation corresponding to one or more time intervals after the time interval T.
[0072] In addition, computing device 160 may further determine a missing benefit value between time interval T and the latest time interval Ti among one or more subsequent time intervals. In some embodiments, when there is no merge operation between time interval T and time interval Ti, the missing benefit value is related to the decision value of performing the second operation during time interval Ti. When there is a merge operation between time interval T and time interval Ti, the missing benefit value may be set to 0.
[0073] The computing device 160 may update the first Q value and the second Q value based on the at least one benefit value and the missing benefit value. After completing the Q value update, the computing device 160 may use the state and the updated first Q value and the updated second Q value corresponding to the state as training samples to implement training of the updated network.
[0074] In some embodiments, the parameters of the trained update network can be shared with the target network, so that the target network generates a first prediction decision value and a second prediction decision value for the file on each layer in each physical partition, and the computing device 160 determines the operation to be taken to perform an appropriate merge operation on the file.
[0075] FIG6 shows a schematic block diagram of an example apparatus 600 according to some embodiments of the present disclosure. Apparatus 600 may be implemented in software, hardware, or a combination of both. As shown in FIG6 , apparatus 600 includes a storage module 610, a determination module 620, a selection module 630, and a partitioning module 640.
[0076] In some embodiments, the storage module 610 is configured to store the received data in the form of a file on a file layer in memory. The determination module 620 is configured to determine whether the data volume of the data on the file layer exceeds a data volume threshold. The selection module 630 is configured to select at least one key from a plurality of key-value pairs corresponding to the file as at least one partition key. Furthermore, the partitioning module 640 is configured to partition the file layer in memory based on the at least one partition key to obtain a plurality of virtual partitions.
[0077] The apparatus 600 in FIG. 6 can be used to implement the above-mentioned process described in conjunction with FIG. 1 to FIG. 5 , and for the sake of brevity, details thereof will not be repeated here.
[0078] The division of modules or units in the embodiments of the present disclosure is illustrative and is merely a logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the disclosed embodiments may be integrated into a single unit, exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.
[0079] FIG7 illustrates a block diagram of an example device 700 that can be used to implement embodiments of the present disclosure. It should be understood that the device 700 shown in FIG7 is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described herein. For example, the device 700 can correspond to the computing device 160 described herein in conjunction with FIG1 and can be used to perform the processes described above in FIG1 through FIG5 .
[0080] As shown in Figure 7, device 700 is in the form of a general-purpose computing device. Components of computing device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. Processing unit 710 may be a real or virtual processor and is capable of performing various processes according to a program stored in memory 720. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to increase the parallel processing capabilities of computing device 700.
[0081] The computing device 700 typically includes a plurality of computer storage media. Such media can be any available media accessible to the computing device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 can be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 can be a removable or non-removable medium and can include a machine-readable medium, such as a flash drive, a disk, or any other medium that can be used to store information and / or data (e.g., training data for training) and can be accessed within the computing device 700.
[0082] The computing device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 7 , a disk drive for reading from or writing to a removable, non-volatile disk (e.g., a “floppy disk”) and an optical drive for reading from or writing to a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to a bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules configured to perform various methods or actions of various implementations of the present disclosure.
[0083] The communication unit 740 enables communication with other computing devices via a communication medium. Additionally, the functionality of the components of the computing device 700 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the computing device 700 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or other network nodes.
[0084] Input device 750 may be one or more input devices, such as a mouse, keyboard, or trackball. Output device 760 may be one or more output devices, such as a display, a speaker, or a printer. Computing device 700 may also communicate with one or more external devices (not shown) via communication unit 740, as needed. External devices such as storage devices, display devices, and the like may also be used to communicate with one or more devices that allow a user to interact with computing device 700, or with any device that allows computing device 700 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).
[0085] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the program is executed by a processor, the method described above is implemented.
[0086] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0087] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0088] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0089] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple implementations of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and a part for a module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.
[0090] While various implementations of the present disclosure have been described above, the foregoing description is intended to be illustrative, not exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is selected to best explain the principles of the implementations, their practical applications, or improvements to existing technologies, or to enable others skilled in the art to understand the various implementations disclosed herein.
Claims
1. A method for partitioning a memory, the method comprising: storing the received data in the form of a file on the file layer of the memory; Determining that the data volume of the data on the file layer exceeds a data volume threshold; Selecting at least one key from a plurality of key-value pairs corresponding to the file as at least one partition key; as well as Based on the at least one partition key, partition the file layer of the memory to obtain a plurality of virtual partitions.
2. The method according to claim 1, wherein selecting at least one key from a plurality of key-value pairs corresponding to the file as at least one partition key comprises: Sampling the plurality of key-value pairs corresponding to the file to obtain a plurality of sampled key-value pairs; Sorting the key-value pairs of the plurality of samples obtained by sampling; The at least one key is selected from the sorted plurality of sampled key-value pairs as the at least one partition key.
3. The method according to claim 1, further comprising: Marking the file layer, wherein the mark indicates that the file layer has the multiple virtual partitions; as well as The at least one partition key is recorded.
4. The method according to claim 1, further comprising: Receive merge requests for merge operations; Determining that the operation layer performing the merge operation already has the multiple virtual partitions; as well as The merging operation is performed on the files to be merged on the operation layer.
5. The method according to claim 4, wherein performing the merge operation on the file on the operation layer comprises: Acquire a plurality of key-value pairs corresponding to the files to be merged on the operation layer; According to the size of the key values in the multiple key-value pairs obtained, the and sorting the files to obtain a sorted file sequence; Sequentially read the keys in the plurality of key-value pairs corresponding to the sorted file sequence; When the value of the read first key is equal to the value of the first partition key in the at least one partition key, performing a first abort operation; as well as storing a file corresponding to the key read before the first abort operation in the first physical partition, The first physical partition corresponds to a first virtual partition obtained by partitioning based on the first partition key.
6. The method according to claim 5, wherein performing the merge operation on the file on the operation layer further comprises: Continue to read the remaining keys in the multiple key-value pairs corresponding to the sorted file sequence in sequence; as well as When the value of the read second key is equal to the value of the second partition key in the at least one partition key, performing a second abort operation; as well as storing a file corresponding to the key read between the first suspend operation and the second suspend operation in a second physical partition, The second physical partition corresponds to a second virtual partition obtained by partitioning based on the second partition key.
7. The method according to claim 5, further comprising: In response to receiving the request, a merge-related operation is performed on the file to be operated in the first physical partition.
8. The method according to claim 7, performing merge-related operations on the files to be operated in the first physical partition, comprising: determining a first predicted target value for performing a first operation in a first state associated with the file within a first time interval and a second predicted target value for performing a second operation in the first state, wherein the first operation and the second operation are related to a merge operation performed on the file in the first physical partition; Based on the first predicted target value and the second predicted target value, determine Select the reference value of the operation; as well as Based on the reference value, an operation is selected from the first operation or the second operation to merge the files.
9. The method according to claim 8, wherein the first predicted target value and the second predicted target value are determined by a target network, the method further comprising: Obtaining network parameters of the updated network as parameters of the target network, The update network is trained on a training sample set.
10. The method according to claim 9 further comprises: respectively determining a first decision value and a second decision value in the first state; determining at least one benefit value corresponding to at least one subsequent time interval after the first time interval; determining a missing benefit between the at least one time interval and a second time interval among the at least one subsequent time interval; Based on the at least one benefit value and the missing benefit value, the first decision value and the second decision value are updated to obtain an updated first decision value and an updated second decision value in the first state.
11. The method according to claim 10, wherein: When there is no merge operation between the first time interval and the second time interval, the missing benefit value is related to a decision value of performing the second operation at the second time interval.
12. The method according to claim 10, wherein: When there is a merge operation between the first time interval and the second time interval, the missing benefit value is set to 0.
13. The method according to claim 10, further comprising: The first state, the updated first decision value and the updated second decision value are used as training samples in a training sample set for training the updated network.
14. An electronic device comprising: at least one processing unit; at least one memory coupled to the at least one The at least one processing unit stores instructions for execution by the at least one processing unit, and when the instructions are executed by the at least one processing unit, the electronic device performs the method according to any one of claims 1 to 13.
15. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.
16. A computer program product having a computer program stored thereon, wherein when the program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.
Citation Information
Patent Citations
High-performance and easy-to-expand key value storage method utilizing differential index mechanism
CN110825748A
Partition merging method and database server
CN110825794A
L0 layer data processing method and device for LSM-tree storage
CN115563109A
Method for partitioning memory, electronic equipment and storage medium
CN116578239A
Staggered merging in log-structured merge forests
US20210216517A1
Cited By
Data processing method and device, data query method and device, equipment, medium and product
CN120386794A
