Data processing method, device, electronic device and storage medium

By generating multiple sorted string table files and controlling their grouping range, the problem of large amount of data during merging in LSM-Tree is solved, and the problems of bandwidth waste and disk life are reduced, and more efficient data storage and disk life are achieved. extend.

CN114691039BActive Publication Date: 2024-11-26BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202210283606.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-22
Publication Date
2024-11-26
Estimated Expiration
2042-03-22

AI Technical Summary

Technical Problem

In the key-value storage method of LSM-Tree, a large amount of data needs to be read and written during merging, resulting in waste of bandwidth and shortened disk life.

Method used

By generating sorted string table files corresponding to at least two sorted string table groups currently defined, the data to be stored is stored in the form of these files, and the key name range of each group is controlled during merging, reducing the next level The coverage range is reduced, thereby reducing the amount of data read and written.

Benefits of technology

Reduces bandwidth waste and disk writes during merge, and extends disk life.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114691039B_ABST
    Figure CN114691039B_ABST
Patent Text Reader

Abstract

This disclosure relates to a data processing method, apparatus, electronic device, and storage medium, and pertains to the field of network technology. In this disclosure, based on the data to be stored in the memory of an electronic device, a specified operation is performed on the data to be stored to complete the operation of writing the data to be stored to a specified level on the disk of the electronic device. The specified operation includes: generating at least two sorted string table files corresponding to at least two currently defined sorted string table groups based on the data to be stored, so as to store the data to be stored in the form of at least two sorted string table files; wherein the key name range corresponding to any sorted string table file does not exceed the group range of the sorted string table group corresponding to the sorted string table file; and dividing the sorted string table files into the corresponding sorted string table groups. This can save bandwidth to a certain extent, reduce the amount of writing to the disk, and improve disk lifespan.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of network technology, and in particular to a data processing method, device, electronic device and storage medium. Background Art

[0002] At present, in the case of data key-value storage, the key-value storage method based on Log Structured Merge Tree (LSM-Tree) is more and more widely used. Among them, LSM-Tree is a hierarchical, ordered, disk-oriented data structure, and its core idea is to make full use of the sequential writing of disk batches to ensure data writing performance.

[0003] In the key-value storage method of LSM-Tree, multiple levels are divided, and the data to be stored in the memory is stored in the form of sorted string tables (SST) in these multiple levels located on the disk. Among them, a sorted string table can include multiple stored key name-key value pairs (key-value). In the related art, when the sorted string tables in multiple levels are merged (Compaction), the sorted string tables with overlapping ranges of key names stored between two adjacent levels need to be involved in the merger. In this way, in order to achieve the merger, the amount of data required to be read / written will be large, which will cause bandwidth waste and affect the life of the disk. Summary of the invention

[0004] The present disclosure provides a data processing method, device, electronic device and storage medium to at least solve the problem in the related art that a large amount of data needs to be read / written during merging, thereby causing bandwidth waste and affecting the life of the disk. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a data processing method, including:

[0006] According to the data to be stored this time in the memory of the electronic device, performing a specified operation on the data to be stored, so as to complete the operation of writing the data to be stored into a specified level of the disk in the electronic device;

[0007] The designated operation includes: based on the data to be stored, generating at least two sorted string table files corresponding to at least two currently defined sorted string table groups, so as to store the data to be stored in the form of the at least two sorted string table files; wherein the key name range corresponding to any of the sorted string table files does not exceed the grouping range of the sorted string table group corresponding to the sorted string table file; and dividing the sorted string table files into the corresponding sorted string table groups.

[0008] Optionally, the method further includes:

[0009] According to the available value range of the key name in the data to be stored, setting the initial grouping range of the initial sorting string table grouping;

[0010] When the total data volume in the initial sorting string table group is not less than the first data volume threshold value corresponding to the initial sorting string table group, the initial sorting string table group is divided into the at least two sorting string table groups according to the initial grouping range; the total data volume in any of the sorting string table groups after the division is less than the first data volume threshold value, and the grouping range of the sorting string table group after the division belongs to the initial grouping range.

[0011] Optionally, the method further includes:

[0012] Taking any of the sorted string table groups currently defined as a target group, and detecting the total amount of data in the target group;

[0013] When the total data volume is not less than the second data volume threshold value corresponding to the target group, the target group is divided into at least two new sorting string table groups according to the grouping range of the target group, and the currently defined sorting string table groups are updated according to the new sorting string table groups; the total data volume in the new sorting string table groups is less than the second data volume threshold value, and the grouping range of the new sorting string table groups belongs to the grouping range of the target group.

[0014] Optionally, dividing the target group into at least two new sorted string table groups according to the grouping range of the target group includes:

[0015] Selecting a candidate key name from the key names covered by the grouping range of the target group, and determining, when the grouping range of the target group is divided by the candidate key name, the sum of the data amounts of the sorted string table files belonging to each grouping range obtained by the division is used as a reference data amount;

[0016] According to the difference between the reference data amounts corresponding to each of the candidate key names and the number of sorting string table files hit by each of the candidate key names, a split key name is selected from the candidate key names, and the grouping range of the target group is divided into two new grouping ranges according to the split key name; a new grouping range corresponds to a new sorting string table group.

[0017] Optionally, the method further includes:

[0018] When there is a sorted string table file to be split in the target group, the sorted string table file to be split is split into two target sorted string table files according to the split key name; different target sorted string table files belong to different new sorted string table groups, and the sorted string table file to be split is the sorted string table file hit by the split key name.

[0019] Optionally, the method further includes:

[0020] In the case that the sorted string table files to be split belong to a specified level, each of the target sorted string table files is merged into a next level.

[0021] Optionally, the method further includes:

[0022] In the case of receiving new data to be stored, at least two sorted string table files corresponding to the updated currently defined sorted string table grouping are generated according to the new data to be stored.

[0023] Optionally, the generating, based on the data to be stored, at least two sorted string table files corresponding to at least two currently defined sorted string table groups comprises:

[0024] Determine the key name-key value pair to be stored according to the data to be stored;

[0025] Divide the key name-key value pairs to be stored into M groups of key name-key value pairs; wherein M is the number of currently defined sorting string table groups, and a key name range corresponding to a group of key name-key value pairs belongs to a grouping range of a sorting string table group;

[0026] M sorted string table files are generated based on the M groups of key name-key value pairs.

[0027] Optionally, the method further includes:

[0028] In response to a data search operation, determining a group range to which the key name to be searched indicated by the data search operation belongs as the group range to be searched;

[0029] In the sorted string table file within the sorted string table group indicated by the to-be-searched group range, the key value corresponding to the to-be-searched key name is searched.

[0030] According to a second aspect of an embodiment of the present disclosure, there is provided a data processing apparatus method, comprising:

[0031] A first generating module is configured to execute a specified storage operation on the data to be stored in the memory of the electronic device, so as to complete an operation of writing the data to be stored into a specified level of a disk in the electronic device;

[0032] The designated storage operation includes: based on the data to be stored, generating at least two SST sorting string table files corresponding to at least two currently defined sorting string table SST groups, so as to store the data to be stored in the form of the at least two SST sorting string table files; wherein the key name range corresponding to any of the SST sorting string table files does not exceed the grouping range of the SST sorting string table group corresponding to the SST sorting string table file; and dividing the SST sorting string table file into the corresponding SST sorting string table groups.

[0033] Optionally, the device further comprises:

[0034] A setting module, configured to set an initial grouping range of the initial sorting string table grouping according to an available value range of the key name in the data to be stored;

[0035] The first division module is configured to execute, when the total data amount in the initial sorting string table group is not less than the first data amount threshold value corresponding to the initial sorting string table group, dividing the initial sorting string table group into the at least two sorting string table groups according to the initial grouping range; the total data amount in any of the sorting string table groups after division is less than the first data amount threshold value, and the grouping range of the sorting string table group after division belongs to the initial grouping range.

[0036] Optionally, the device further comprises:

[0037] A detection module is configured to execute any currently defined sorted string table group as a target group and detect the total amount of data in the target group;

[0038] The second division module is configured to execute, when the total data volume is not less than the second data volume threshold value corresponding to the target group, dividing the target group into at least two new sorting string table groups according to the grouping range of the target group, and updating the currently defined sorting string table group according to the new sorting string table group; the total data volume in the new sorting string table group is less than the second data volume threshold value, and the grouping range of the new sorting string table group belongs to the grouping range of the target group.

[0039] Optionally, the second division module is specifically configured to execute:

[0040] Selecting a candidate key name from the key names covered by the grouping range of the target group, and determining, when the grouping range of the target group is divided by the candidate key name, the sum of the data amounts of the sorted string table files belonging to each grouping range obtained by the division is used as a reference data amount;

[0041] According to the difference between the reference data amounts corresponding to each of the candidate key names and the number of sorting string table files hit by each of the candidate key names, a split key name is selected from the candidate key names, and the grouping range of the target group is divided into two new grouping ranges according to the split key name; a new grouping range corresponds to a new sorting string table group.

[0042] Optionally, the device further comprises:

[0043] The splitting module is configured to execute, when there is a sorted string table file to be split in the target group, splitting the sorted string table file to be split into two target sorted string table files according to the splitting key name; different target sorted string table files belong to different new sorted string table groups, and the sorted string table file to be split is the sorted string table file hit by the splitting key name.

[0044] Optionally, the device further comprises:

[0045] The merging module is configured to merge the target sorting string table files into a next level when the sorting string table files to be split belong to a specified level.

[0046] Optionally, the device further comprises:

[0047] The second generating module is configured to generate at least two sorting string table files corresponding to the updated currently defined sorting string table grouping according to the new data to be stored when new data to be stored is received.

[0048] Optionally, the first generating module is specifically configured to execute:

[0049] Determine the key name-key value pair to be stored according to the data to be stored;

[0050] Divide the key name-key value pairs to be stored into M groups of key name-key value pairs; wherein M is the number of currently defined sorting string table groups, and a key name range corresponding to a group of key name-key value pairs belongs to a grouping range of a sorting string table group;

[0051] M sorted string table files are generated based on the M groups of key name-key value pairs.

[0052] Optionally, the device further comprises:

[0053] A determination module is configured to execute, in response to a data search operation, determine a group range to which the key name to be searched indicated by the data search operation belongs, as the group range to be searched;

[0054] The search module is configured to search for a key value corresponding to the key name to be searched in the sorted string table file within the sorted string table group indicated by the group range to be searched.

[0055] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0056] A processor; a memory for storing instructions executable by the processor;

[0057] The processor is configured to execute the instructions to implement the method as described in any one of the first aspects.

[0058] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device executes a method as described in any one of the first aspects.

[0059] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, the computer program product comprising readable program instructions, and when the readable program instructions are executed by a processor of an electronic device, the electronic device executes a method as described in any one of the first aspects.

[0060] The technical solution provided by the embodiment of the present disclosure brings at least the following beneficial effects: In the embodiment of the present disclosure, according to the data to be stored in the memory of the electronic device, a specified operation is performed on the data to be stored to complete the operation of writing the data to be stored to the specified level of the disk in the electronic device. Wherein, the specified operation includes: based on the data to be stored, generating at least two sorting string table files corresponding to at least two sorting string table groups currently defined, so as to store the data to be stored in the form of at least two sorting string table files; wherein the key name range corresponding to any sorting string table file does not exceed the grouping range of the sorting string table group corresponding to the sorting string table file; and dividing the sorting string table file into the corresponding sorting string table group. In this way, by setting the sorting string table group, based on the grouping range of the sorting string table group, the key name range of the sorting string table file in the specified level is controlled, and the number of sorting string table files covered in the next level can be reduced to a certain extent, and then the amount of data that needs to be read / written each time during the subsequent merge can be reduced, thereby saving bandwidth to a certain extent, reducing the amount of writing to the disk, and improving the disk life.

[0061] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0063] Figure 1 is a flow chart of a data processing method according to an exemplary embodiment;

[0064] Figure 2 is a splitting schematic diagram shown according to an exemplary embodiment;

[0065] Figure 3 is a storage schematic diagram according to an exemplary embodiment;

[0066] Figure 4 is another storage schematic diagram shown according to an exemplary embodiment;

[0067] Figure 5 is a block diagram of a data processing device according to an exemplary embodiment;

[0068] Figure 6 is a block diagram of a device for data processing according to an exemplary embodiment;

[0069] Figure 7 It is a block diagram of another device for data processing according to an exemplary embodiment. DETAILED DESCRIPTION

[0070] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0071] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0072] Figure 1is a flow chart of a data processing method according to an exemplary embodiment. The method can be applied to electronic devices such as Figure 1 As shown, the method may include:

[0073] Step 101: Perform a specified operation on the data to be stored in the memory of the electronic device to complete the operation of writing the data to be stored into a specified level of the disk in the electronic device; the specified operation includes: generating at least two sorted string table files corresponding to at least two currently defined sorted string table sorted string table groups based on the data to be stored, so as to store the data to be stored in the form of the at least two sorted string table files; wherein the key name range corresponding to any of the sorted string table files does not exceed the grouping range of the sorted string table group corresponding to the sorted string table file; and dividing the sorted string table files into corresponding sorted string table groups.

[0074] In the disclosed embodiment, the electronic device may be a server, a computer, or other device. The grouping range (span) may correspond one-to-one to the sorting string table grouping. The currently defined grouping range is the grouping range corresponding to the currently defined sorting string table grouping. A grouping range may represent a key name (key) range. For example, a grouping range may be expressed as [start key, end key). The key name ranges represented by different grouping ranges may not overlap. Among them, the currently defined sorting string table grouping and the grouping range may be fixed or dynamically changing, and the disclosed embodiment does not limit this.

[0075] The data to be stored in the memory may be written into the memory according to actual storage requirements. For example, the data to be stored may be the user's account ID and account-related information, audio and video information, etc. corresponding to the account ID. The data to be stored in the memory may be referred to as a "MemTable". When the data to be stored in the memory meets the preset write requirements, for example, when the size of the data to be stored in the memory exceeds the preset threshold, these data to be stored may be used as the data to be stored this time, and trigger the operation of storing the data to be stored in the memory into the disk, that is, trigger the execution of the steps in the embodiments of the present disclosure. It should be noted that new data may continue to be written to the memory subsequently, and when the newly written data meets the preset write requirements, the execution of the steps in the embodiments of the present disclosure may be triggered again.

[0076] Specifically, the data to be stored in the memory can be stored to the disk in the form of a sorted string table file. The data to be stored may include a key name (key)-key value (value) pair, a sorted string table file is a sorted string table, and a sorted string table file may store multiple key name-key value pairs. The key name range corresponding to the sorted string table file may be a range consisting of the minimum key name stored in the sorted string table file and the maximum key name stored. Alternatively, a key name-key value pair may also be stored in the sorted string table file, and accordingly, the range consisting of the negative infinity to the key names stored in the sorted string table file, or the positive infinity to the key names stored in the sorted string table file may be used as the key name range corresponding to the sorted string table file, which is not limited in the embodiment of the present disclosure. Further, the operation of completing the writing of the data to be stored to the specified level in the embodiment of the present disclosure may refer to: storing the data to be stored in the form of at least two sorted string table files to the specified level. The specified operation may be performed after the data to be stored is written into the specified level, that is, in the embodiment of the present disclosure, the data to be stored in the memory may be written into the specified level first, and then the data to be stored may be split within the specified level. Alternatively, the specified operation may be performed before the data to be stored is written into the specified level, that is, the data to be stored may be split outside the specified level first, and at least two sorted string table files obtained by the splitting may be written into the specified level.

[0077] In an embodiment of the present invention, the data to be stored is divided into a plurality of sorted string table files according to the currently defined grouping range, and the key name range of a single sorted string table file is controlled not to exceed the grouping range. In this way, the key name range covered by a single sorted string table file can be controlled to a certain extent, and the coverage range of the sorted string table file stored in the disk this time can be reduced as much as possible, thereby reducing the number of sorted string table files covered in the next level, thereby reducing the amount of data that needs to be read / written each time during subsequent compaction, and saving bandwidth to a certain extent, reducing the amount of writing to the disk, and improving the disk life. For example, compared with the method of generating only one sorted string table file, assuming that the key name range covered by the data to be stored this time is [1, 64), and only one corresponding sorted string table file with a key name range of [1, 64) is generated, the embodiment of the present disclosure, in the flush phase of flushing MemTable to the disk, splits the data to be stored into a plurality of sorted string table files based on the grouping ranges of at least two defined sorted string table groups. Assuming that the currently defined grouping range includes grouping range 1: [0-40), grouping range 2: [40-80), then two sorting string table files can be split: a sorting string table file whose corresponding key name range belongs to grouping range 1 and a sorting string table file whose corresponding key name range belongs to grouping range 2. Among them, the size of the key name range corresponding to each sorting string table file will be smaller than the size of the key name range: [1, 64). The size of the range can be understood as the difference between the maximum value and the minimum value in the range. The embodiment of the present disclosure performs pre-splitting at the stage of storing the data to be stored from the memory to the disk, that is, splitting multiple sorting string table files, and can control the key name range covered by a single sorting string table file, that is, can control the merging radius of a single sorting string table file, thereby controlling the number of sorting string table files participating in the merging of the lower layer.

[0078] It should be noted that the generated sort string table file can correspond to the group range one by one, or multiple sort string table files can correspond to one group range, as long as the group range based on the defined sort string table grouping is ensured and the key name range of the sort string table file is controlled not to be too large.

[0079] In the disclosed embodiment, the designated level may be set according to actual conditions. For example, the designated level may be the highest level "Level-0" in the LSM-tree storage structure. The sort string table group indicated by the grouping range may be the sort string table group corresponding to the grouping range. The sort string table file is divided into the sort string table groups indicated by the corresponding grouping range, and the file identifier of the sort string table file may be stored in correspondence with the grouping range corresponding to the sort string table file.

[0080] In summary, the data processing method provided by the embodiment of the present disclosure performs a specified operation on the data to be stored according to the data to be stored in the memory of the electronic device, so as to complete the operation of writing the data to be stored into the specified level of the disk in the electronic device. Wherein, the specified operation includes: based on the data to be stored, generating at least two sorting string table files corresponding to at least two sorting string table groups currently defined, so as to store the data to be stored in the form of at least two sorting string table files; wherein the key name range corresponding to any sorting string table file does not exceed the grouping range of the sorting string table group corresponding to the sorting string table file; and dividing the sorting string table file into the corresponding sorting string table group. In this way, by setting the sorting string table group, based on the grouping range of the sorting string table group, controlling the key name range of the sorting string table file in the specified level, the number of sorting string table files covered in the next level can be reduced to a certain extent, and then the amount of data that needs to be read / written each time during the subsequent merge can be reduced, thereby saving bandwidth to a certain extent, reducing the amount of writing to the disk, and improving the disk life.

[0081] Optionally, in one implementation, at least two sorting string table groups may be defined based on the following method. Specifically, the method may include the following steps:

[0082] Step S21: setting an initial grouping range of the initial sorting string table grouping according to the available value range of the key name in the data to be stored.

[0083] In the disclosed embodiments, the available value range of the key name in the data to be stored may be pre-set according to actual conditions, and the available value range may be a sorting range of the key name. For example, the range consisting of the minimum key name and the maximum key name that may appear in the actual data storage scenario may be used as the available value range. When determining the initial grouping range, the available value range may be directly determined as the initial grouping range, or a range covering the available value range may be selected as the initial grouping range. For example, the number of initial grouping ranges may be 1, and an initial grouping with a range of (-infinite, +infinite) may be defined in the initial stage of the storage engine.

[0084] Step S22: When the total data volume in the initial sorting string table group is not less than the first data volume threshold value corresponding to the initial sorting string table group, the initial sorting string table group is divided into the at least two sorting string table groups according to the initial grouping range; the total data volume in any of the sorting string table groups after the division is less than the first data volume threshold value, and the grouping range of the sorting string table group after the division belongs to the initial grouping range.

[0085] When setting the initial grouping, data may be written to the disk first, that is, the sorted string table file may be added to the initial sorted string table grouping. It should be noted that the data written in this step may be the data previously written to the memory, and these data may be the data actually required to be stored, and of course, may also be the data used to trigger the grouping splitting, which is not limited in the embodiments of the present disclosure.

[0086] As data is continuously written, the total data volume of the sorted string table files belonging to the initial sorted string table group will become larger and larger. When the total data volume of the sorted string table files belonging to the initial sorted string table group is not less than the first data volume threshold corresponding to the initial sorted string table group, the initial group can be split into two sorted string table groups, and the initial group range is divided into the group ranges of the two sorted string table groups. For example, the total data volume of the sorted string table files of the initial sorted string table group is size, and max size represents the first data volume threshold. As data is continuously written, the size of the span increases continuously. When the system detects that the size of the current span exceeds the configured max size, a suitable middle position L can be found. For the initial span, the initial span can be split into two spans: span left: (-infinite, L), span right: [L, +infinite). Of course, it can also be split into more sorted string table groups, and the embodiments of the present disclosure are not limited to this. Among them, the total data volume in the initial sorting string table group refers to the sum of the data volumes of all sorting string table files in the initial sorting string table group. The first data volume threshold can be set according to actual needs. For example, the first data volume threshold can be a value selected from 500 megabytes (MB) to 1 gigabyte (GB).

[0087] In the disclosed embodiment, the initial grouping range of the initial sorting string table grouping is set according to the available value range of the key name in the data to be stored, and the initial grouping range is split into at least two grouping ranges to obtain at least two sorting string table groups. In this way, it can be ensured to a certain extent that the grouping ranges of all defined sorting string table groups can cover the available value range of the key name, thereby ensuring that subsequent data storage operations can proceed smoothly.

[0088] Optionally, the following steps may be further performed in the embodiment of the present disclosure:

[0089] Step S31: taking any of the currently defined sorted string table groups as a target group, and detecting the total amount of data in the target group.

[0090] In the disclosed embodiment, each currently defined sorting string table group can be monitored to determine the total amount of data in each sorting string table group. For example, for any sorting string table group, the amount of data of the sorting string table file added to the sorting string table group each time can be recorded to calculate the total amount of data in the current group.

[0091] Step S32: When the total data volume is not less than the second data volume threshold value corresponding to the target group, the target group is divided into at least two new sorting string table groups according to the grouping range of the target group, and the currently defined sorting string table groups are updated according to the new sorting string table groups; the total data volume in the new sorting string table groups is less than the second data volume threshold value, and the grouping range of the new sorting string table groups belongs to the grouping range of the target group.

[0092] The second data volume threshold may be set according to actual needs. For example, the second data volume threshold may be a value selected from 500MB to 1GB. The second data volume threshold may be the same as the first data volume threshold. In the disclosed embodiment, the maximum storage capacity of each sort string table group may be 500MB to 1GB. The maximum amount of data that each sort string table group can accommodate may be the same, or may be different. For example, assuming that the disk capacity is 3 terabytes (TB) and the data volume threshold is 1GB, as flushing continues, a maximum of 3072 sort string table files (i.e., spans) may be triggered. Compared with the number of sort string table groups directly stored in 3TB itself, this span number is smaller and easier to manage.

[0093] As the data to be stored is continuously written, the total amount of data belonging to the target group will become larger and larger. When the total amount of data belonging to the target group is not less than the second data amount threshold corresponding to the target group, the target group can be split into two new sorting string table groups, and the grouping range of the target group can be divided into the grouping ranges of the two new sorting string table groups. Of course, it can also be split into more new sorting string table groups, and the embodiment of the present disclosure is not limited to this. Further, after each split, after obtaining a new sorting string table group, the currently defined sorting string table group can be updated, and the currently defined grouping range can be updated. Specifically, the currently defined sorting string table group can be updated to the newly obtained sorting string table group obtained by splitting and other existing sorting string table groups, and the grouping range of the currently defined sorting string table group can be updated to the grouping range of the new sorting string table group obtained by splitting and the grouping range of other existing sorting string table groups that have not been split. For example, it is assumed that the currently defined sorting string table group includes: group A, group B and group C. Group A is split to obtain new sorting string table groups: group a1 and group a2. Then the currently defined sort string table groups can be updated to: group a1, group a2, group B and group C.

[0094] In the disclosed embodiment, a second data volume threshold is set for each currently defined sort string table grouping, and when the total number of data within a certain sort string table grouping reaches the set second data volume threshold, the sort string table grouping is split, thereby avoiding too many sort string table files in the currently defined sort string table grouping, and, by dynamically splitting the sort string table grouping, the grouping range of the defined sort string table grouping can be gradually narrowed, thereby controlling the key name range covered by the sort string table files stored in the specified level to a smaller extent.

[0095] Optionally, in an implementation manner, the step of dividing the target group into at least two new sorted string table groups according to the group range of the target group may specifically include:

[0096] Step S41, selecting a candidate key name from the key names covered by the grouping range of the target group, and determining, when the grouping range of the target group is divided by the candidate key name, the sum of the data amounts of the sorted string table files belonging to each grouping range obtained by the division, as a reference data amount.

[0097] In this step, you can select any key name from the key names stored in each sorted string table file in the target group as the candidate key name corresponding to the sorted string table file. Alternatively, first calculate the mean of the maximum key name and the minimum key name in the grouping range of the target group, and then select the mean and the value near the mean as the candidate key name to find a suitable middle position. For example, calculate the value of the mean + X*preset increment and the value of the mean-X*preset increment when X takes the value of 1-P to obtain the candidate key name. Among them, the specific value of P can be set according to actual needs, and the embodiment of the present disclosure does not limit this.

[0098] For each candidate key name, we can first determine the sum of the data amounts of the sorted string table files contained in the two new ranges obtained after dividing the grouping range of the target group by the candidate key name, and then obtain the two reference data amounts corresponding to the candidate key name.

[0099] Step S42: According to the difference between the reference data amounts corresponding to each of the candidate key names and the number of sorting string table files hit by each of the candidate key names, a split key name is selected from the candidate key names, and the grouping range of the target group is divided into two new grouping ranges according to the split key name; a new grouping range corresponds to a new sorting string table group.

[0100] The sort string table file hit by the candidate key name may refer to the sort string table file whose corresponding key name range includes the candidate key name, or may also be the sort string table file whose stored key name includes the candidate key name, and the disclosed embodiment does not limit this. The principle of ensuring that the data size (spansize) of the newly split groups is not different as much as possible and the split sort string table is as small as possible can be adopted to calculate the difference between the reference data amounts corresponding to each candidate key name, and select the candidate key name with the smallest corresponding difference and / or the smallest number of hit sort string table files as the split key name (split Key).

[0101] During the division, the minimum key name in the grouping range of the target group to the split key name can be divided into a new grouping range, and the maximum key name in the grouping range from the split key name to the target group can be divided into a new grouping range. By dividing the new grouping range, the division of the new sorting string table group is achieved. Specifically, a correspondence between each new grouping range and the file identifier of the sorting string table file belonging to the new grouping range can be established, so as to achieve the creation of a new sorting string table group. For example, metadata corresponding to the new grouping range is created, and the file identifier of the sorting string table file belonging to the new grouping range is recorded in the metadata.

[0102] It should be noted that, in the embodiment of the present disclosure, each sorted string table group includes sorted string table files in all levels that belong to the grouping range of the sorted string table group. The sorted string table files of each layer in each sorted string table group comply with the original rule attributes of the LSM-tree, that is, except for L0, the sorted string table file ranges in the same layer of other layers do not overlap. The key name ranges covered by sorted string table files of different layers can overlap. There is no overlap or gap between the grouping ranges of the sorted string table groupings, that is, the grouping ranges of all sorted string table groupings constitute the initial grouping range.

[0103] In the disclosed embodiment, multiple candidate key names are selected from the key names covered by the grouping range of the target group, and the sum of the data volumes of the sorted string table files belonging to each grouping range obtained by dividing the grouping range by the candidate key names is determined as the reference data volume. By considering the difference between the reference data volumes corresponding to each candidate key name and the number of sorted string table files hit, a split key name is selected, and dynamic splitting is performed based on the split key name, which can ensure to a certain extent that the new sorted string table grouping obtained after the split is more reasonable, thereby ensuring the splitting effect.

[0104] Optionally, in the embodiment of the present disclosure, after selecting the split key name, the following steps may be performed:

[0105] Step S51: When there is a sorted string table file to be split in the target group, split the sorted string table file to be split into two target sorted string table files according to the split key name; different target sorted string table files belong to different new sorted string table groups, and the sorted string table file to be split is the sorted string table file hit by the split key name.

[0106] Specifically, when the number of sorting string table files hit by the split key name in the target group is not 0, that is, when there is a sorting string table file hit by the split key name, the sorting string table file hit by the split key name is used as the sorting string table file to be split, and the sorting string table file to be split is split according to the split key name, and the sorting string table file to be split is split into two target sorting string table files belonging to the new sorting string table groupings, respectively, to ensure that any sorting string table file belongs to only one sorting string table grouping, and the sorting string table file does not span multiple sorting string table groups. Specifically, the sorting string table file to be split can be split into a part smaller than the split key name and a part not smaller than the split key name, thereby obtaining two target sorting string table files. The two target sorting string table files can belong to different new sorting string table groups respectively. Among them, the part smaller than the split key name can include a key name smaller than the split key name and a corresponding key value, and the part not smaller than the split key name can include a key name not smaller than the split key name and a corresponding key value. The target sort string table file composed of a portion smaller than the split key name can be used as a sort string table file belonging to the first new sort string table group and added to the first new sort string table group. The target sort string table file composed of a portion not smaller than the split key name can be used as a sort string table file belonging to the second new sort string table group and added to the second new sort string table group. The grouping range of the first new sort string table group can be [start key in the grouping range of the target group, split key name), and the grouping range of the second new sort string table group can be [split key name, end key in the grouping range of the target group).

[0107] In the embodiment of the present disclosure, when there is a sorted string table file to be split in the target group, the sorted string table file to be split is split into two target sorted string table files belonging to new sorted string table groups respectively according to the split key name, wherein the sorted string table file to be split is the sorted string table file hit by the split key name. In this way, it can be ensured that when the sorted string table group is dynamically split, each sorted string table file can be correctly divided into the corresponding sorted string table group, thereby ensuring the grouping effect.

[0108] Optionally, when the sorted string table file to be split belongs to a specified level, each of the target sorted string table files can also be merged into the next level. The specified level can be the highest level "Level 0". That is to say, if there is a sorted string table file to be split in the specified level, the merge operation can be directly triggered to split the sorted string table file to be split in the specified level into two sorted string table files and then merge them into the next level. In this way, the number of sorted string table files in the specified level and the level below it can be reduced to a certain extent, thereby reducing the disk space occupied and leaving more space for subsequent data, so that the specified level can store more sorted string table files. Specifically, the target sorted string table file split from the sorted string table file to be split and the sorted string table file stored before the sorted string table file to be split in the specified level can be merged into the next level. If there is a sort string table file to be split in the next level of the specified level, and the key name range of the sort string table file to be split in the next level overlaps with the sort string table file in the specified level, then the sort string table file to be split in the next level can be split during the merging process. The remaining sort string table files in the specified level do not need to be merged. For example, Figure 2 is a schematic diagram of splitting according to an exemplary embodiment. Figure 2 As shown, one SST represents one sorting string table file, and finally Span A can be split into two spans, Span AL and Span AR. There is a sorting string table file 01 to be split in L0. During the splitting process, the sorting string table file 01 to be split and the sorting string table file stored before the sorting string table file 01 to be split can be merged with the sorting string table file in the L1 layer. Among them, all the sorting string table files in the 02 box are involved in the merger.

[0109] For other levels other than the specified level, since the sorted string table files with overlapping key name ranges are not allowed in the other levels except the specified level in the LSM-tree, that is, at most one sorted string table file in other levels needs to be split, the implementation of the present disclosure can start from Level 1 and only split within the level without triggering downward merging, thereby reducing the number of times the merge is triggered as much as possible. It should be noted that in addition to the existence of the sorted string table file to be split in the specified level, which can trigger the merge operation, in the embodiment of the present invention, for any level, when the data size of the sorted string table file in the level reaches a specified threshold, it can be merged to the lower level. Specifically, when merging downward, the sorted string table file in the next level of the level that has an overlapping key name range with the sorted string table file of the level can be selected and merged with the sorted string table file in the level. Among them, the lowest level does not perform downward merging.

[0110] Optionally, the step of generating at least two sorted string table files corresponding to at least two currently defined sorted string table groups based on the data to be stored may specifically include:

[0111] Step S61: Determine the key name-key value pairs to be stored according to the data to be stored.

[0112] In the disclosed embodiment, the data to be stored may be data including multiple key name-key value pairs. Accordingly, the key name-key value pairs included therein may be directly used as the key name-key value pairs to be stored.

[0113] Step S62, dividing the key name-key value pairs to be stored into M groups of key name-key value pairs; wherein M is the number of currently defined sorting string table groups, and the key name range corresponding to a group of key name-key value pairs belongs to the grouping range of one sorting string table group.

[0114] Specifically, the key names may be sorted according to their size to keep the key values ​​in the sorted order for storage. According to the key name ranges limited by the currently defined grouping ranges, the sorted key name-key value pairs to be stored are divided into M key name-key value pairs, thereby obtaining M groups of key name-key value pairs. The key names in each group of key name-key value pairs maintain the sorted order, and accordingly, the key names in each key name-key value pair of the sorted string table file can maintain the sorted order.

[0115] Step S63: Generate M sorted string table files based on the M groups of key name-key value pairs.

[0116] In this step, a sorted string table file can be configured based on a set of key name-key value pairs, thereby obtaining M sorted string table files for storage. For example, the key name-key value pairs in a set of key name-key value pairs can be stored in a block, and then the index of the block can be stored at the end of the sorted string table to facilitate searching based on the key name, thereby obtaining a sorted string table file. For example, Figure 3 is a storage schematic diagram according to an exemplary embodiment. Figure 3 As shown, for the Memtable to be stored this time, the Memtable can be split into three sorting string table files corresponding to the three currently defined sorting string table grouping ranges: Span1, Span2 and Span3, so as to be stored in the three sorting string table groups respectively. Figure 4 is another storage schematic diagram according to an exemplary embodiment. Figure 4 As shown, during the flush process, for the Memtable to be stored this time, the Memtable can be split into three sorting string table files corresponding to the three sorting string table grouping ranges currently defined: SpanA, SpanB and SpanC, so that they meet the definition constraints of span, so that no giant sorting string table file covering a lot of lower-level sorting string table files will be generated in the L0 layer. Among them, overlapping is allowed between the sorting string table files in the L0 layer, and the sorting string table files generated by flushing the memory to the disk will be placed in L0 first.

[0117] In the disclosed embodiment, the key name-key value pairs to be stored are determined according to the data to be stored. The key name-key value pairs to be stored are divided into M groups of key name-key value pairs in a one-to-one correspondence with the grouping range of the currently defined sorting string table grouping, and M sorting string table files are generated based on the M groups of key name-key value pairs. In this way, it can be ensured to a certain extent that the number of sorting string table files in each sorting string table grouping is balanced, avoiding the sorting string table grouping with too many or too few sorting string table files, thereby ensuring the balance of each grouping.

[0118] Optionally, in the embodiment of the present disclosure, upon receiving new data to be stored, at least two sorting string table files corresponding to the updated currently defined sorting string table grouping can be generated according to the new data to be stored. The specific generation method can refer to the above description and will not be repeated here. In the embodiment of the present disclosure, based on the updated sorting string table grouping, the data required for subsequent storage is stored, which can ensure that the key name range covered by the sorting string table file subsequently written to the specified level is smaller, thereby further reducing the number of sorting string table files covered in the lower layer when merging downward. It should be noted that in the embodiment of the present disclosure, after the new grouping range is determined, the split can be directly performed according to the new grouping range during the splitting period, thereby ensuring to a greater extent that the key name range covered by the sorting string table file written to the specified level is as small as possible.

[0119] It should be noted that in the disclosed embodiment, there is no overlap between the grouping ranges of each sort string table grouping, thereby ensuring that the key name ranges of the sort string table files in different sort string table groups do not overlap. In this way, when merging, it can be ensured that the sort string table files in the upper layer only need to participate in the merge with the sort string table files in the same group in the lower layer, thereby avoiding the merging of sort string table files in different groups. In the disclosed embodiment, the defined sort string table grouping is dynamically changing. As the total amount of data stored in the sort string table grouping increases, the number of defined sort string table groups and the corresponding grouping ranges will be automatically split and adjusted. Since the range of each span is very small, and there is no merge between the sort string table files of each span in the disclosed embodiment, the write amplification can be reduced as much as possible to a certain extent.

[0120] Optionally, in the embodiment of the present disclosure, the following operation may also be performed: for any of the sorted string table groups, when the creation time of the sorted string table group reaches a preset time and the total data volume of the sorted string table files in the sorted string table group is less than a third data volume threshold, the sorted string table group is used as the sorted string table group to be merged, and the sorted string table group to be merged is merged with the adjacent sorted string table group. In this way, the problem of too many sorted string table groups and too small group range as the dynamic splitting continues can be avoided.

[0121] Among them, the adjacent sorted string table grouping can be a grouping whose grouping range is adjacent to the grouping range of the sorted string table grouping to be merged. Specifically, there can be two adjacent sorted string table groups, and the two adjacent sorted string table groups can be a grouping whose end key of the grouping range is equal to the start key of the grouping range of the sorted string table grouping to be merged, and can be a grouping whose start key of the grouping range is equal to the end key of the grouping range of the sorted string table grouping to be merged. Specifically, before merging, it can be detected whether the sum of the total data volume of the adjacent sorted string table groups and the total data volume of the sorted string table groups to be merged is less than the second data volume threshold. If so, the merge operation can be performed, otherwise, the merge operation is not performed. Thereby ensuring that the size of the group after the merge is less than the span max size. When merging, the adjacent sorted string table grouping with a smaller total data volume can be selected to participate in the merge.

[0122] In the disclosed embodiment, the sorting string table files in each sorting string table group are recorded in the metadata corresponding to each sorting string table group, and different sorting string table groups correspond to different metadata. Specifically, the metadata corresponding to each sorting string table group corresponds to the grouping range of each sorting string table group. When the sorting string table files are divided into the sorting string table groups, the file identifier of the sorting string table file can be specifically written into the metadata corresponding to the sorting string table group.

[0123] Accordingly, when merging groups, metadata corresponding to the sorted string table group to be merged and the adjacent sorted string table group participating in the merging can be deleted. The sorted string table group to be merged and the grouping range of the adjacent sorted string table group participating in the merging are merged, metadata corresponding to the grouping range obtained after the merging is created, and the file identifiers of the sorted string table files in the sorted string table group to be merged and the sorted string table files in the adjacent sorted string table group participating in the merging are recorded in the metadata, thereby achieving the merging. In this way, there is no need to move the sorted string table files and merge the sorted string table files. The sorted string table group merging can be achieved only by changing the metadata, thereby ensuring the efficiency of the group merging to a certain extent and avoiding high merging costs.

[0124] Optionally, the following operations may also be performed in the embodiment of the present disclosure:

[0125] Step S71: In response to a data search operation, determine a group range to which the key name to be searched indicated by the data search operation belongs, as the group range to be searched.

[0126] The data search operation may be performed by a user, and the key name to be searched may be a key name specified based on the data search operation. For example, the data search operation may be a preset instruction input by a user, such as a Get instruction, and the key name carried in the Get instruction is the key name to be searched. In this step, the group range to which the key name to be searched belongs may be located from the currently defined multiple group ranges, that is, the sorted string table group to which the sorted string table file to which the key name to be searched belongs may be located.

[0127] Step S72: Search for a key value corresponding to the key name to be searched in the sorted string table file within the sorted string table group indicated by the group range to be searched.

[0128] For example, the sorted string table files in the sorted string table group indicated by the group range to be searched can be read, and the key name to be searched can be located in these sorted string table files. When the key name to be searched is located, the key value corresponding to the key name to be searched can be obtained. Finally, the key value can be returned to the querying party. For example, when searching for L0, since the sorted string table file in L0 is also divided into spans, when Get a key name, first locate the span to which it belongs, and then only need to read the limited sorted string table file in this span to complete the reading of L0, so as to perform the search.

[0129] In the disclosed embodiment, since the sorted string table is divided into groups, by first locating the group range to which the key name to be searched belongs, it is only necessary to read the sorted string table files within the sorted string table group indicated by the group range, and search is performed in these sorted string table files. This can reduce the number of reads, improve the read performance, and improve the data search efficiency to a certain extent without sacrificing the read performance.

[0130] An application scenario designed by the embodiment of the present disclosure is described below. At present, in large-capacity storage scenarios, persistent key-value storage plays a key role in various modern data-intensive applications. For example, key-value storage is often used in the fields of web indexing, e-commerce, photo libraries, cloud data, social networks, online games, messages, software libraries and advertising. Key-value storage provides effective insertion, point search and range query, and is a basic component of growing important applications. For write-intensive workloads, key-value storage based on LSM-Tree has become the latest technology. LSM-Tree is a write-retaining sequential access mode. Its core idea is to make full use of the sequential write of disk batches. LSM-Tree is much higher than random write performance. Among them, the writing of LSM-Tree is performed in the form of Append. Various distributed and local storage engines based on LSM-Tree are widely deployed in large-scale production environments. In order to provide high write performance, LSM-Tree batches key name-key value pairs and writes them sequentially. It can then be searched efficiently, for example, a single key name and a range can be queried. The LSM-Tree engine continuously reads, sorts, and writes key-value pairs in the background, thus maintaining the keys and values ​​in sorted order. In this way, the same data may be read and written multiple times during the entire life cycle of the key name, so there will be the problem of I / O write amplification. On the one hand, I / O write amplification causes unnecessary bandwidth waste due to repeated IO, and on the other hand, repeated writing will seriously affect the life of the solid-state drive (SSD) disk.

[0131] In one optimization method, a fragment (F) log-structured merge tree is used to add a data organization unit for each level: guard. By weakening the global order, the number of merges is reduced overall, thereby optimizing IO write amplification. The attributes of guard may be different at each level. However, the design of FLSM-Tree is relatively complex. The introduction of a new organizational structure, guard, breaks the original clear hierarchical structure, making it difficult to implement well in engineering practice. In addition, there is a range overlap between the sorted string table files inside guard, so both Get and Scan require more IO, resulting in further reduction in read performance. Guard allows sorted string table files to overlap, which results in more disk usage and higher storage costs.

[0132] In the disclosed embodiment, a dynamic grouping compression method is adopted, that is, Span is set for all levels of LSM-tree, which will not destroy the global orderliness. The sorting string table files are divided based on Span to control the coverage radius during compaction. The smaller the grouping range of Span, the smaller the radius of compaction. Correspondingly, the less IO amplification is achieved, so as to achieve the purpose of reducing IO amplification. Specifically, when compaction is performed between different levels, because the ranges of different sorting string table files are different, it is necessary to let the overlapping sorting string table files between two adjacent levels participate in the compaction. Especially for the L0 layer, in the related art, because the sorting string table file of the L0 layer is directly flushed from the MemTable to the disk, its range is uncertain and it is very easy to have a sorting string table file with a very large key name range covered, which results in the need for almost all sorting string table files of the L1 layer to participate in the compaction during compaction. As a result, write amplification is serious. In the disclosed embodiment, by controlling the key name range covered by the sorting string table file in the L0 layer (i.e., the specified level), write amplification can be effectively alleviated. At the same time, since the key name range covered by the sorted string table file written into L0 is limited in the embodiment of the present disclosure and grouped, in this way, when L0 stores more sorted string table files, it can also ensure to a certain extent that read / write amplification will not increase, thereby further reducing the number of compaction times.

[0133] Optionally, in the disclosed embodiment, each sorted string table grouping may span all levels of LSM-tree defined in the disk, and the number of levels defined in the disclosed embodiment may be 2 to 3 levels. Since the sorted string table file data is divided into independent spans, even 2 to 3 levels are required to store TB-level data, and the smaller the level, the smaller the write amplification can be. For example, the more levels there are, the more compactions may need to be performed overall, and therefore, the more serious the write amplification problem will be. Compared to the method of using a 7-layer structure, in the disclosed embodiment, only 2 to 3 layers are required, which can further alleviate the write amplification problem.

[0134] Figure 5 is a block diagram of a data processing device according to an exemplary embodiment. The device 20 can be applied to electronic devices such as Figure 5 As shown, the device 20 may include:

[0135] The first generating module 201 is configured to execute a specified storage operation on the data to be stored in the memory of the electronic device, so as to complete the operation of writing the data to be stored into a specified level of the disk in the electronic device;

[0136] The designated storage operation includes: based on the data to be stored, generating at least two SST sorting string table files corresponding to at least two currently defined sorting string table SST groups, so as to store the data to be stored in the form of the at least two SST sorting string table files; wherein the key name range corresponding to any of the SST sorting string table files does not exceed the grouping range of the SST sorting string table group corresponding to the SST sorting string table file; and dividing the SST sorting string table file into the corresponding SST sorting string table groups.

[0137] In an optional embodiment, the device 20 further includes:

[0138] A setting module, configured to set an initial grouping range of the initial sorting string table grouping according to an available value range of the key name in the data to be stored;

[0139] The first division module is configured to execute, when the total data amount in the initial sorting string table group is not less than the first data amount threshold value corresponding to the initial sorting string table group, dividing the initial sorting string table group into the at least two sorting string table groups according to the initial grouping range; the total data amount in any of the sorting string table groups after division is less than the first data amount threshold value, and the grouping range of the sorting string table group after division belongs to the initial grouping range.

[0140] In an optional embodiment, the device 20 further includes:

[0141] A detection module is configured to execute any currently defined sorted string table group as a target group and detect the total amount of data in the target group;

[0142] The second division module is configured to execute, when the total data volume is not less than the second data volume threshold value corresponding to the target group, dividing the target group into at least two new sorting string table groups according to the grouping range of the target group, and updating the currently defined sorting string table group according to the new sorting string table group; the total data volume in the new sorting string table group is less than the second data volume threshold value, and the grouping range of the new sorting string table group belongs to the grouping range of the target group.

[0143] In an optional embodiment, the second division module is specifically configured to execute:

[0144] Selecting a candidate key name from the key names covered by the grouping range of the target group, and determining, when the grouping range of the target group is divided by the candidate key name, the sum of the data amounts of the sorted string table files belonging to each grouping range obtained by the division is used as a reference data amount;

[0145] According to the difference between the reference data amounts corresponding to each of the candidate key names and the number of sorting string table files hit by each of the candidate key names, a split key name is selected from the candidate key names, and the grouping range of the target group is divided into two new grouping ranges according to the split key name; a new grouping range corresponds to a new sorting string table group.

[0146] In an optional embodiment, the device 20 further includes:

[0147] The splitting module is configured to execute, when there is a sorted string table file to be split in the target group, splitting the sorted string table file to be split into two target sorted string table files according to the splitting key name; different target sorted string table files belong to different new sorted string table groups, and the sorted string table file to be split is the sorted string table file hit by the splitting key name.

[0148] In an optional embodiment, the device 20 further includes:

[0149] The merging module is configured to merge the target sorting string table files into a next level when the sorting string table files to be split belong to a specified level.

[0150] In an optional embodiment, the device 20 further includes:

[0151] The second generating module is configured to generate at least two sorting string table files corresponding to the updated currently defined sorting string table grouping according to the new data to be stored when new data to be stored is received.

[0152] In an optional embodiment, the first generating module 201 is specifically configured to execute:

[0153] Determine the key name-key value pair to be stored according to the data to be stored;

[0154] Divide the key name-key value pairs to be stored into M groups of key name-key value pairs; wherein M is the number of currently defined sorting string table groups, and a key name range corresponding to a group of key name-key value pairs belongs to a grouping range of a sorting string table group;

[0155] M sorted string table files are generated based on the M groups of key name-key value pairs.

[0156] In an optional embodiment, the device 20 further includes:

[0157] A determination module is configured to execute, in response to a data search operation, determine a group range to which the key name to be searched indicated by the data search operation belongs, as the group range to be searched;

[0158] The search module is configured to search for a key value corresponding to the key name to be searched in the sorted string table file within the sorted string table group indicated by the group range to be searched.

[0159] In summary, the data processing device provided by the embodiment of the present disclosure performs a specified operation on the data to be stored according to the data to be stored in the memory of the electronic device, so as to complete the operation of writing the data to be stored into the specified level of the disk in the electronic device. Wherein, the specified operation includes: based on the data to be stored, generating at least two sorting string table files corresponding to at least two sorting string table groups currently defined, so as to store the data to be stored in the form of at least two sorting string table files; wherein the key name range corresponding to any sorting string table file does not exceed the grouping range of the sorting string table group corresponding to the sorting string table file; and dividing the sorting string table file into the corresponding sorting string table group. In this way, by setting the sorting string table group, based on the grouping range of the sorting string table group, the key name range of the sorting string table file in the specified level is controlled, so that the number of sorting string table files covered in the next level can be reduced to a certain extent, and then the amount of data that needs to be read / written each time during the subsequent merge can be reduced, so that the bandwidth can be saved to a certain extent, the amount of writing to the disk can be reduced, and the disk life can be improved.

[0160] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0161] According to one embodiment of the present disclosure, an electronic device is provided, comprising: a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement the steps in the data processing method in any of the above embodiments when executing.

[0162] According to an embodiment of the present disclosure, a storage medium is further provided. When instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute the steps in the data processing method in any of the above embodiments.

[0163] According to one embodiment of the present disclosure, a computer program product is also provided. The computer program product includes readable program instructions. When the readable program instructions are executed by a processor of an electronic device, the electronic device can execute the steps in the data processing method in any of the above embodiments.

[0164] Figure 6 700 is a block diagram of a device for data processing according to an exemplary embodiment. Among them, the device 700 may include a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input / output interface 712, a sensor component 714, a communication component 716 and a processor 720. The processing component 702 may include one or more processors 720 to execute instructions to complete all or part of the steps of the above-mentioned data processing method. In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 704 including instructions, and the above-mentioned instructions can be executed by the processor 720 of the device 700 to complete the above-mentioned method. Optionally, the storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0165] Figure 7 It is a block diagram of another device for data processing according to an exemplary embodiment.

[0166] The device 800 may include a processing component 822, a memory 832, an input / output interface 858, a network interface 850, and a power supply component 826. The device 800 may be provided as a server. The application program stored in the memory 832 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 822 is configured to execute instructions to perform the above-mentioned data processing method.

[0167] The user information (including but not limited to the user's device information, user personal information, etc.) and related data involved in this disclosure are all information authorized by the user or by all parties.

[0168] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0169] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: Applied to electronic equipment, the method comprises: According to the data to be stored in the memory of the electronic device, a specified operation is performed on the data to be stored. To complete the operation of writing the data to be stored into a specified level of the disk in the electronic device; The designation operation includes: generating at least two sorting string table files corresponding to at least two currently defined sorting string table groups based on the data to be stored, so as to store the data to be stored in the form of the at least two sorting string table files; wherein the key name range corresponding to any of the sorting string table files does not exceed the grouping range of the sorting string table group corresponding to the sorting string table file; and dividing the sorting string table files into the corresponding sorting string table groups; The method further includes: taking any of the currently defined sorting string table groups as a target group, detecting the total data volume in the target group; selecting a candidate key name from the key names covered by the grouping range of the target group when the total data volume is not less than a second data volume threshold corresponding to the target group, and determining the sum of the data volumes of the sorting string table files belonging to each grouping range obtained by dividing the grouping range of the target group by the candidate key name, as a reference data volume; selecting a split key name from the candidate key name according to the difference between the reference data volumes corresponding to each of the candidate key names and the number of sorting string table files hit by each of the candidate key names, and dividing the grouping range of the target group into two new grouping ranges according to the split key name; a new grouping range corresponds to a new sorting string table group; and updating the currently defined sorting string table group according to the new sorting string table group; the total data volume in the new sorting string table group is less than the second data volume threshold, and the grouping range of the new sorting string table group belongs to the grouping range of the target group.

2. The method according to claim 1, characterized in that The method further comprises: According to the available value range of the key name in the data to be stored, setting the initial grouping range of the initial sorting string table grouping; When the total data volume in the initial sorting string table group is not less than the first data volume threshold value corresponding to the initial sorting string table group, the initial sorting string table group is divided into the at least two sorting string table groups according to the initial grouping range; the total data volume in any of the sorting string table groups after the division is less than the first data volume threshold value, and the grouping range of the sorting string table group after the division belongs to the initial grouping range.

3. The method according to claim 1, characterized in that The method further comprises: When there is a sorted string table file to be split in the target group, the sorted string table file to be split is split into two target sorted string table files according to the split key name; different target sorted string table files belong to different new sorted string table groups, and the sorted string table file to be split is the sorted string table file hit by the split key name.

4. The method according to claim 3, characterized in that The method further comprises: In the case that the sorted string table files to be split belong to a specified level, each of the target sorted string table files is merged into a next level.

5. The method according to claim 1, characterized in that The method further comprises: In the case of receiving new data to be stored, at least two sorted string table files corresponding to the updated currently defined sorted string table grouping are generated according to the new data to be stored.

6. The method according to claim 1, characterized in that The step of generating at least two sorting string table files corresponding to at least two sorting string table groups currently defined based on the data to be stored comprises: Determine the key name-key value pair to be stored according to the data to be stored; Divide the key name-key value pairs to be stored into M groups of key name-key value pairs; wherein M is the number of currently defined sorting string table groups, and a key name range corresponding to a group of key name-key value pairs belongs to a grouping range of a sorting string table group; M sorted string table files are generated based on the M groups of key name-key value pairs.

7. The method according to claim 1, characterized in that The method further comprises: In response to a data search operation, determining a group range to which the key name to be searched indicated by the data search operation belongs as the group range to be searched; In the sorted string table file within the sorted string table group indicated by the to-be-searched group range, the key value corresponding to the to-be-searched key name is searched.

8. A data processing device, characterized in that: Applied to electronic equipment, the device comprises: A first generating module is configured to execute a specified storage operation on the data to be stored in the memory of the electronic device, so as to complete an operation of writing the data to be stored into a specified level of a disk in the electronic device; The designated storage operation includes: based on the data to be stored, generating at least two SST sorting string table files corresponding to at least two currently defined SST sorting string table groups, so as to store the data to be stored in the form of the at least two SST sorting string table files; wherein the key name range corresponding to any of the SST sorting string table files does not exceed the grouping range of the SST sorting string table group corresponding to the SST sorting string table file; dividing the SST sorting string table file into the corresponding SST sorting string table groups; The device also includes: A detection module is configured to execute any currently defined sorted string table group as a target group and detect the total amount of data in the target group; The second partitioning module is configured to execute, when the total data volume is not less than the second data volume threshold corresponding to the target group, selecting a candidate key name from the key names covered by the grouping range of the target group, and determining, when the grouping range of the target group is partitioned by the candidate key name, the sum of the data volumes of the sorted string table files belonging to each grouping range obtained by the partitioning as a reference data volume; selecting a split key name from the candidate key names according to the difference between the reference data volumes corresponding to each of the candidate key names and the number of sorted string table files hit by each of the candidate key names, and dividing the grouping range of the target group into two new grouping ranges according to the split key name; a new grouping range corresponds to a new sorted string table grouping; and updating the currently defined sorted string table grouping according to the new sorted string table grouping; the total data volume in the new sorted string table grouping is less than the second data volume threshold, and the grouping range of the new sorted string table grouping belongs to the grouping range of the target group.

9. The device according to claim 8, characterized in that The device also includes: A setting module, configured to set an initial grouping range of the initial sorting string table grouping according to an available value range of the key name in the data to be stored; The first division module is configured to execute, when the total data amount in the initial sorting string table group is not less than the first data amount threshold value corresponding to the initial sorting string table group, dividing the initial sorting string table group into the at least two sorting string table groups according to the initial grouping range; the total data amount in any of the sorting string table groups after division is less than the first data amount threshold value, and the grouping range of the sorting string table group after division belongs to the initial grouping range.

10. The device according to claim 8, characterized in that The device also includes: The splitting module is configured to execute, when there is a sorted string table file to be split in the target group, splitting the sorted string table file to be split into two target sorted string table files according to the splitting key name; different target sorted string table files belong to different new sorted string table groups, and the sorted string table file to be split is the sorted string table file hit by the splitting key name.

11. The device according to claim 10, characterized in that The device also includes: The merging module is configured to merge the target sorting string table files into a next level when the sorting string table files to be split belong to a specified level.

12. The device according to claim 8, characterized in that The device also includes: The second generating module is configured to generate at least two sorting string table files corresponding to the updated currently defined sorting string table grouping according to the new data to be stored when new data to be stored is received.

13. The device according to claim 8, characterized in that The first generating module is specifically configured to execute: Determine the key name-key value pair to be stored according to the data to be stored; Divide the key name-key value pairs to be stored into M groups of key name-key value pairs; wherein M is the number of currently defined sorting string table groups, and a key name range corresponding to a group of key name-key value pairs belongs to a grouping range of a sorting string table group; M sorted string table files are generated based on the M groups of key name-key value pairs.

14. The device according to claim 8, characterized in that The device also includes: A determination module is configured to execute, in response to a data search operation, determine a group range to which the key name to be searched indicated by the data search operation belongs, as the group range to be searched; The search module is configured to search for a key value corresponding to the key name to be searched in the sorted string table file within the sorted string table group indicated by the group range to be searched.

15. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the method according to any one of claims 1 to 7.

16. A storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product comprises readable program instructions, and when the readable program instructions are executed by a processor of an electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • LSM (Log Structured Merge) data compact method and device

    CN105095287A

  • Data storage method and device

    CN108804019A

  • Key value storage system based on NVM and SSD hybrid storage structure

    CN110347336A

  • Data reading and writing method, device and equipment and storage medium

    CN110764705A

  • Data storage method, electronic device, and computer program product

    US20220035548A1