Data storage method and device, equipment, storage medium and program product
By generating standard mask compression key identifiers, the problem of the inability to compress non-prefix identical key identifiers in the existing technology is solved, thus improving the data compression rate.
Patent Information
- Application Number
- CN202511587787.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-31
- Publication Date
- 2026-02-24
Smart Images
Figure CN121560221A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data storage method, apparatus, device, storage medium, and program product. Background Technology
[0002] To ensure disk read / write performance, the Log-Structured Merge-Tree (LSM) was developed. Its core idea is to fully utilize the fact that sequential write performance is far superior to random write performance, transforming batches of random writes into a single sequential write. Since the key identifiers in the Sorted String Tables (SSTs) of an LSM tree are ordered, to save data storage space, the key identifiers in the SSTs are compressed. Generally, key identifiers with the same prefix in the SST are compressed. For example, the key identifiers {123456; 123457; 123458} can be stored as {123456; 7; 8}.
[0003] However, the above method only compresses key identifiers with the same prefix. It cannot compress key identifiers with different prefixes. For example, for the key identifiers {12345678; 12355678; 12365679}, it can only be stored as {12345678; 55678; 65679}, which will reduce the data compression rate of key identifiers. Summary of the Invention
[0004] Therefore, it is necessary to provide a data storage method, apparatus, device, storage medium, and program product that can improve data compression rate in response to the above-mentioned technical problems.
[0005] Firstly, this application provides a data storage method, including:
[0006] Obtain the data block to be stored; wherein, the data block to be stored contains key identifiers in sequential order;
[0007] Based on the consistency of the first key identifier and each second key identifier in character position, the initial mask corresponding to each second key identifier is determined; wherein, the first key identifier is the starting key identifier among all key identifiers; each second key identifier is the key identifier among all key identifiers excluding the first key identifier;
[0008] Determine the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier;
[0009] Using a standard mask, each second key identifier in the data block to be stored is compressed to obtain a compressed data block, which is then stored.
[0010] In this embodiment, by determining the character retention status of each second key identifier at each character position based on the consistency of each key identifier at each character position, key identifier compression can be performed. This enables compression of identical characters at other character positions except for the prefix, improving the flexibility and comprehensiveness of key identifier compression, and thus effectively improving the data compression rate of the data block to be stored.
[0011] In one embodiment, the initial mask is a binary value; the standard mask for the data block to be stored is determined based on the initial mask corresponding to each second key identifier, including:
[0012] Use the initial mask corresponding to the first key identifier in each second key identifier as the first reference mask;
[0013] The first reference mask is updated based on the initial mask corresponding to each third key identifier to obtain the backup mask; wherein, each third key identifier is the key identifier of each second key identifier except for the first key identifier;
[0014] Invert the backup mask to obtain the standard mask for the data block to be stored.
[0015] In this embodiment, on the one hand, by using the initial mask corresponding to the third key identifier to update the first reference mask to obtain a backup mask, the accuracy of the subsequent standard mask can be guaranteed; on the other hand, by inverting the backup mask to obtain the standard mask, the representation meaning of the standard mask can be adapted to the conventional representation meaning of binary characters, thereby improving the efficiency of subsequent data compression.
[0016] In one embodiment, the first reference mask is updated based on the initial mask corresponding to each third key identifier to obtain a backup mask, including:
[0017] Based on the order of the third key identifiers, each third key identifier is used as the target key identifier in turn;
[0018] For each target key identifier, the initial mask corresponding to the target key identifier is used to update the intermediate mask to obtain the updated mask, and the updated mask corresponding to the last target key identifier is used as the backup mask.
[0019] Specifically, when the target key identifier is the first key identifier among all the third key identifiers, the intermediate mask is the first reference mask; when the target key identifier is not the first key identifier among all the third key identifiers, the intermediate mask is the updated mask corresponding to the previous key identifier of the target key identifier.
[0020] In this embodiment, by updating the first reference mask sequentially according to the arrangement order of each third key identifier, a backup mask can be obtained, which can ensure the reliability of the backup mask determination.
[0021] In one embodiment, the target key identifier is used to identify the initial mask, and the intermediate mask is updated to obtain the updated mask, including:
[0022] Perform a binary AND operation on the initial mask corresponding to the intermediate mask and the target key identifier to obtain the updated mask.
[0023] In this application embodiment, a mask update method is provided that can meet the requirement of determining whether each key identifier is consistent at the same character position, thereby ensuring the reliability of subsequent backup mask determination.
[0024] In one embodiment, a standard mask is used to compress each second key identifier in the data block to be stored, resulting in a compressed data block, including:
[0025] Based on the preset character retention rules and the character information at each character position in the standard mask, determine the reserved character positions in each character position;
[0026] For each second key identifier, the compressed key identifier corresponding to the second key identifier is determined based on the character located in the reserved character position in the second key identifier;
[0027] Compressed data blocks are generated based on the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier.
[0028] In this embodiment of the application, by compressing the second key identifier and generating compressed data blocks based on the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier, the rationality of the determination of the compressed data blocks can be guaranteed.
[0029] In one embodiment, the initial mask is a binary value; the initial mask corresponding to each second key identifier is determined based on the consistency of the first key identifier and each second key identifier in character position, including:
[0030] For each second key identifier, based on the consistency between the second key identifier and the first key identifier at each character position, the mask at each character position is selected from the binary characters to obtain the initial mask corresponding to the second key identifier.
[0031] In this application embodiment, an optional method for determining the initial mask is provided, which can improve the reliability of the initial mask determination.
[0032] In one embodiment, after storing the compressed data blocks, the method further includes:
[0033] In response to a key query request, if the key identifier to be queried is inconsistent with the first key identifier, a query mask is generated based on the consistency of the key identifier to be queried and the first key identifier in each character position;
[0034] If the presence of a key identifier to be queried is determined in the compressed data block based on the query mask and the standard mask, the key identifier to be queried is processed using the standard mask to obtain the compressed key identifier corresponding to the key identifier to be queried.
[0035] Based on the binary search principle, the compressed key identifier corresponding to the key identifier to be queried is used to search in the compressed data block to obtain the storage location of the key identifier to be queried in the compressed data block.
[0036] In this embodiment of the application, by using the method of generating compressed key identifiers in compressed data blocks, the same processing is performed on the key identifier to be queried, so as to query the storage location of the key identifier to be queried in the compressed data block, which can ensure the rationality of key identifier query.
[0037] In one embodiment, the initial mask is a binary value; determining the presence of a query key identifier in the compressed data block based on the query mask and the standard mask includes:
[0038] Invert the standard mask to obtain the second reference mask;
[0039] Perform a binary AND operation between the query mask and the second reference mask to obtain the updated query mask;
[0040] If the updated query mask matches the second reference mask, it is determined that the key identifier to be queried exists in the compressed data block;
[0041] If the updated query mask is inconsistent with the second reference mask, it is determined that there is no query key identifier in the compressed data block.
[0042] In this embodiment, a second reference mask is obtained by inverting the standard mask, and an updated query mask is obtained by performing a binary AND operation between the query mask and the second reference mask. Based on the consistency between the updated query mask and the second reference mask, it is determined whether there is a key identifier to be queried in the compressed data block, which can ensure the reliability of key identifier query.
[0043] Secondly, this application also provides a data storage device, comprising:
[0044] The acquisition module is used to acquire the data block to be stored; wherein, the data block to be stored contains key identifiers in sequential order;
[0045] The first determining module is used to determine the initial mask corresponding to each second key identifier based on the consistency of the first key identifier and each second key identifier in the character position; wherein, the first key identifier is the starting key identifier among the key identifiers; and each second key identifier is the key identifier among the key identifiers excluding the first key identifier.
[0046] The second determining module is used to determine the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier;
[0047] The compression module is used to compress each second key identifier in the data block to be stored using a standard mask, to obtain a compressed data block, and then stores the compressed data block.
[0048] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:
[0049] Obtain the data block to be stored; wherein, the data block to be stored contains key identifiers in sequential order;
[0050] Based on the consistency of the first key identifier and each second key identifier in character position, the initial mask corresponding to each second key identifier is determined; wherein, the first key identifier is the starting key identifier among all key identifiers; each second key identifier is the key identifier among all key identifiers excluding the first key identifier;
[0051] Determine the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier;
[0052] Using a standard mask, each second key identifier in the data block to be stored is compressed to obtain a compressed data block, which is then stored.
[0053] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:
[0054] Obtain the data block to be stored; wherein, the data block to be stored contains key identifiers in sequential order;
[0055] Based on the consistency of the first key identifier and each second key identifier in character position, the initial mask corresponding to each second key identifier is determined; wherein, the first key identifier is the starting key identifier among all key identifiers; each second key identifier is the key identifier among all key identifiers excluding the first key identifier;
[0056] Determine the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier;
[0057] Using a standard mask, each second key identifier in the data block to be stored is compressed to obtain a compressed data block, which is then stored.
[0058] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:
[0059] Obtain the data block to be stored; wherein, the data block to be stored contains key identifiers in sequential order;
[0060] Based on the consistency of the first key identifier and each second key identifier in character position, the initial mask corresponding to each second key identifier is determined; wherein, the first key identifier is the starting key identifier among all key identifiers; each second key identifier is the key identifier among all key identifiers excluding the first key identifier;
[0061] Determine the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier;
[0062] Using a standard mask, each second key identifier in the data block to be stored is compressed to obtain a compressed data block, which is then stored.
[0063] The aforementioned data storage method, apparatus, device, storage medium, and program product determine an initial mask corresponding to each second key identifier based on the consistency of the first key identifier at the starting position of the data block to be stored with other second key identifiers at their character positions. Then, based on the initial masks corresponding to each second key identifier, a standard mask for the data block to be stored is determined. Subsequently, the standard mask is used to compress each second key identifier in the data block to be stored, resulting in a compressed data block, which is then stored. Compared to related technologies that only compress key identifiers with the same prefix, the above method, by determining the character retention status of each second key identifier at each character position based on the consistency of each key identifier at each character position for key identifier compression, can compress identical characters at other character positions besides the prefix. This improves the flexibility and comprehensiveness of key identifier compression, thereby effectively increasing the data compression rate of the data block to be stored. Attached Figure Description
[0064] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0065] Figure 1 This is a flowchart illustrating a data storage method in one embodiment;
[0066] Figure 2 This is a flowchart illustrating the process of determining a standard mask in one embodiment;
[0067] Figure 3 This is a flowchart illustrating the process of determining compressed data blocks in one embodiment;
[0068] Figure 4 This is a schematic diagram of the compressed data block structure in one embodiment;
[0069] Figure 5 This is a schematic diagram of the data query process in one embodiment;
[0070] Figure 6 This is a flowchart illustrating the process of storing and judging results in one embodiment;
[0071] Figure 7 This is a flowchart illustrating the data storage method in another embodiment;
[0072] Figure 8 This is a structural block diagram of a data storage device in one embodiment;
[0073] Figure 9 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0074] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0075] To ensure disk read / write performance, the LSM tree was developed. Its core idea is to fully utilize the fact that sequential write performance is far superior to random write performance, transforming batches of random writes into a single sequential write. Since the key identifiers in each SST of an LSM tree are ordered, to save data storage space, the key identifiers in the SST are compressed. Generally, key identifiers with the same prefix in the SST are compressed.
[0076] However, the above method only compresses key identifiers with the same prefix, and cannot compress key identifier data with different prefixes, which reduces the data compression rate of key identifiers.
[0077] Based on this, in an exemplary embodiment, a data storage method is provided, which will be described using an application of the method to a server as an example. Figure 1 As shown, the specific steps include:
[0078] S101, Obtain the data block to be stored.
[0079] Here, the so-called data block to be stored is a data block that has storage requirements; furthermore, the data block to be stored contains sequentially ordered key identifiers. For example, the data block to be stored can be a data block dataBlock in SST.
[0080] Optionally, upon detecting a data storage event, the data block to be stored can be directly retrieved. This data storage event can be initiated when the MemTable in the LSM is full, or when any ordered key-identified data block is full.
[0081] S102, determine the initial mask corresponding to each second key identifier based on the consistency of the first key identifier and each second key identifier in the character position.
[0082] In this context, the first key identifier is the starting key identifier among all key identifiers, and each second key identifier is a key identifier other than the first key identifier. For example, if the data block to be stored is a dataBlock, the first key identifier could be the restartPoint in the dataBlock. The initial mask is used to characterize the consistency between the first key identifier and the second key identifier at each character position.
[0083] In one alternative implementation, for each second key identifier, the second key identifier and the first key identifier can be simultaneously input into a trained first mask generation model, and the first mask generation model outputs an initial mask for the second key identifier based on the consistency of the second key identifier and the first key identifier at each character position.
[0084] In another alternative implementation, two mask values representing consistency and inconsistency can be preset. For each second key identifier, the consistency between the second key identifier and the first key identifier at each character position can be determined by selecting the corresponding mask value from the two preset mask values, which is then used as the mask for each character position, thereby obtaining the initial mask for the second key identifier.
[0085] For example, the character "T" can be used to represent that the key identifiers at the character positions are the same, and the character "F" can be used to represent that the key identifiers at the character positions are different. For the first key identifier 12345678 and the second key identifier 12355678, the initial mask of the second key identifier can be obtained as "TTTFTTTT".
[0086] When the initial mask is a binary value, the initial mask can be determined by selecting the mask for each character position from the binary characters based on the consistency between the second key identifier and the first key identifier at each character position, thus obtaining the initial mask corresponding to the second key identifier.
[0087] Specifically, for each second key identifier, based on the consistency between the second key identifier and the first key identifier at each character position, and the representational meaning of the binary characters, the mask at each character position can be selected from the binary characters to concatenate the initial mask corresponding to the second key identifier.
[0088] For example, the binary character "1" can be used to represent that the key identifiers at the character positions are the same, and the binary character "0" can be used to represent that the key identifiers at the character positions are different. For the first key identifier 12345678 and the second key identifier 12355678, the initial mask of the second key identifier can be obtained as "0b11101111". Here, "0b" represents binary.
[0089] S103, determine the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier.
[0090] The so-called standard mask is used to represent the consistency of the identifiers at each character position among the key identifiers.
[0091] In one alternative implementation, for each character position, a standard mask can be determined based on the mask consistency of the initial mask of each second key identifier at that character position; after concatenating the standard masks at each character position, the standard mask of the data block to be stored can be obtained.
[0092] In another alternative implementation, the initial mask corresponding to each second key identifier can be directly input into the trained second mask generation model, and the second mask generation model can output the standard mask of the data block to be stored based on the initial mask of each second key identifier.
[0093] S104 uses a standard mask to compress each second key identifier in the data block to be stored, obtaining a compressed data block, and then stores the compressed data block.
[0094] The so-called compressed data block is a data block that has undergone compression processing.
[0095] Understandably, since the standard mask can represent the consistency of identifiers at each character position among the key identifiers, the compressible character bits of the second key identifier in the data block to be stored can be determined by referring to the standard mask; that is, consistent characters are compressed, and inconsistent characters are retained. Furthermore, since the first key identifier needs to represent the starting key identifier of the data block to be stored, the first key identifier cannot be compressed.
[0096] For each second key identifier in the data block to be stored, the second key identifier can be compressed according to the character retention positions represented by the standard mask. For example, if the standard mask represents the character retention positions as the first and third positions, the compressed second key identifier "1437" will be "13". After replacing the second key identifiers in the data block to be stored with the compressed second key identifiers, the compressed data block can be obtained. Furthermore, the compressed data block can be stored in the corresponding location.
[0097] In the aforementioned data storage method, an initial mask corresponding to each second key identifier is determined based on the consistency of the first key identifier at the starting position in the data block to be stored with other second key identifiers at their character positions. Then, a standard mask for the data block to be stored is determined based on the initial masks corresponding to each second key identifier. Finally, the standard mask is used to compress each second key identifier in the data block to be stored, resulting in a compressed data block, which is then stored. Compared to related technologies that only compress key identifiers with the same prefix, the above method, by determining the character retention status of each second key identifier at each character position based on the consistency of each key identifier at each character position, enables compression of identical characters at positions other than the prefix. This improves the flexibility and comprehensiveness of key identifier compression, thereby effectively increasing the data compression rate of the data block to be stored.
[0098] Based on the above embodiments, in this embodiment, the initial mask is a binary value. Furthermore, an optional method for determining the standard mask is provided, such as... Figure 2 As shown, the specific steps include:
[0099] S201, take the initial mask corresponding to the first key identifier in each second key identifier as the first reference mask.
[0100] The so-called first reference mask is the reference information for determining the standard mask.
[0101] Optionally, after determining the initial mask corresponding to each second key identifier, the initial mask corresponding to the first key identifier among the second key identifiers can be used as the first reference mask.
[0102] S202, based on the initial mask corresponding to each third key identifier, update the first reference mask to obtain the backup mask.
[0103] In this context, each third key identifier is a key identifier other than the first key identifier among the second key identifiers. The so-called spare mask is the mask obtained after updating the first reference mask, which can directly reflect the overall consistency between the second key identifier and the first key identifier at each character position.
[0104] In one alternative implementation, the first reference mask can be updated sequentially using the initial mask corresponding to the third key identifier according to the order of the key identifiers, and the last updated mask can be used as the backup mask.
[0105] S203, invert the spare mask to obtain the standard mask of the data block to be stored.
[0106] Understandably, to improve data compression efficiency, after determining the standard mask, Single Instruction Multiple Data (SIMD) instructions can be used to uniformly compress each key identifier. However, in machine instructions, the binary character "1" generally represents "consistent" and "retained," while "0" generally represents "inconsistent" and "not retained." In this embodiment, it is necessary to retain inconsistent key identifiers. Therefore, to make the standard mask conform to the above character representations, the spare mask representing the consistency situation can be inverted to obtain the standard mask representing the retention situation. That is, the character "1" in the spare mask indicates that the key identifiers at that position are consistent and can be compressed; therefore, the character "0" obtained after inversion represents the compression meaning.
[0107] For example, by inverting the spare mask "0b11101110", we can obtain the standard mask "0b00010001". In this case, only the 4th and 8th bits of each second key identifier in the data block to be stored are retained. That is, after compressing the second key identifier "12355678", we get "58"; after compressing the second key identifier "12365679", we get "69".
[0108] In this embodiment, on the one hand, by using the initial mask corresponding to the third key identifier to update the first reference mask to obtain a backup mask, the accuracy of the subsequent standard mask can be guaranteed; on the other hand, by inverting the backup mask to obtain the standard mask, the representation meaning of the standard mask can be adapted to the conventional representation meaning of binary characters, thereby improving the efficiency of subsequent data compression.
[0109] Based on the above embodiments, in this embodiment, the initial mask is a binary value. Further, an optional method for determining the standard mask is provided, specifically, according to the arrangement order of the third key identifiers, each third key identifier is sequentially used as the target key identifier; for each target key identifier, the initial mask corresponding to the target key identifier is used to update the intermediate masks to obtain the updated mask, and the updated mask corresponding to the last target key identifier is used as the backup mask.
[0110] Specifically, when the target key identifier is the first key identifier among all third key identifiers, the intermediate mask is the first reference mask; when the target key identifier is not the first key identifier among all third key identifiers, the intermediate mask is the updated mask corresponding to the previous key identifier of the target key identifier. The so-called updated mask is the mask after the update process.
[0111] Optionally, the first reference mask can be updated using the first third key identifier to obtain the updated mask, and this updated mask with the first third key identifier can be used as the intermediate mask. Then, the second third key identifier can be used as the target key identifier, and the intermediate mask can be updated using the target key identifier to obtain the updated mask. At this point, the updated mask can be used as the new intermediate mask.
[0112] Furthermore, based on the order of the key identifiers, it can be determined whether there are any third key identifiers that have not yet been updated. If so, the subsequent third key identifier is used as the new target key identifier, and the process of updating the intermediate mask using the target key identifier is repeated to obtain the updated mask. This continues until there are no more third key identifiers that have not yet been updated. At this point, the last updated mask can be used as a backup mask.
[0113] For example, the mask update method can be to perform a binary AND operation on the intermediate mask and the initial mask corresponding to the target key identifier to obtain the updated mask.
[0114] Specifically, since the principle of binary AND operation is that "1" is obtained only when both characters are "1", which satisfies the requirement in this embodiment to determine whether each key identifier is consistent in the same character position, for each target key identifier, the binary intermediate mask and the initial mask corresponding to the binary target key identifier can be ANDed to obtain the updated mask.
[0115] For example, given the third key identifier {0b11101111; 0b11101110; 0b11101100}, the result of performing a AND operation on "0b11101111" and "0b11101110" is "0b11101110". Then, performing a AND operation on "0b11101100" and "0b11101110" again yields "0b11101100". At this point, the backup mask is "0b11101100".
[0116] In this embodiment, by updating the first reference mask sequentially according to the arrangement order of each third key identifier, a backup mask can be obtained, which can ensure the reliability of the backup mask determination.
[0117] Based on the above embodiments, this application provides an optional method for determining compressed data blocks, such as... Figure 3 As shown, the specific steps include:
[0118] S301, determine the reserved character positions in each character position according to the preset character retention rules and the character information at each character position in the standard mask.
[0119] The so-called preset character preservation rules are the preservation rules corresponding to the mask characters. For example, in a binary mask, "1" represents preservation, and "0" represents non-preservation. The so-called reserved character positions are the positions of the characters that need to be preserved.
[0120] Optionally, for each character position, the standard mask at that position can be used as an index to query within a preset character preservation rule to obtain the preservation result for that character position. After querying the standard mask at each character position sequentially, the preserved character positions for each character position can be obtained. For example, for the standard mask "0b00010001", the preserved character positions are the 4th and 8th positions.
[0121] S302, for each second key identifier, determine the compressed key identifier corresponding to the second key identifier based on the character located in the reserved character position in the second key identifier.
[0122] The so-called compressed key identifier is the key identifier after compression.
[0123] Optionally, for each second key identifier, the character located in the reserved character position of the second key identifier can be directly used as the compressed key identifier corresponding to the second key identifier.
[0124] S303 generates compressed data blocks based on the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier.
[0125] Understandably, to more comprehensively characterize the compression status of the compressed data block, in addition to the compressed key identifiers corresponding to each second key identifier, the first key identifier and the standard mask also need to be retained in the compressed data block. Therefore, a compressed data block can be generated based on the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier. For example, the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier can be concatenated to obtain the compressed data block.
[0126] For example, you can refer to Figure 4 The diagram showing the compressed data block structure uses the first key identifier, standard mask, and number of compressed characters as the first field, which is then concatenated with the compressed key identifiers corresponding to each second key identifier to obtain the compressed data block.
[0127] It is worth noting that, in order to ensure the completeness of information in the compressed data block, the metadata corresponding to each key identifier can also be stored sequentially at the end of the compressed data block.
[0128] In this embodiment of the application, by compressing the second key identifier and generating compressed data blocks based on the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier, the rationality of the determination of the compressed data blocks can be guaranteed.
[0129] Based on the above embodiments, this application provides an optional method for data querying, such as... Figure 5 As shown, the specific steps include:
[0130] S501, in response to a key query request, if the key identifier to be queried is inconsistent with the first key identifier, a query mask is generated based on the consistency of the key identifier to be queried and the first key identifier in each character position.
[0131] Here, a key query request refers to a query request for the key identifier in each compressed data block. The key identifier to be queried is the key identifier that needs to be queried. The query mask indicates the consistency between the key identifier to be queried and the first key identifier at each character position.
[0132] Optionally, after detecting a key query request for a key identifier in each compressed data block, for each compressed data block, in order to determine whether a key query request exists in the compressed data block, the first key identifier of the compressed data block can be compared with the key identifier to be queried.
[0133] If the first key identifier of the compressed data block matches the key identifier to be queried, the storage location of the key identifier to be queried within the compressed data block can be directly found. If the first key identifier of the compressed data block does not match the key identifier to be queried, a query mask can be generated by referring to the consistency of the first key identifier and the key identifier to be queried at each character position.
[0134] For example, when the mask is a binary character, the mask at each character position can be selected from the binary character based on the consistency between the first key identifier and the key identifier to be queried at each character position to obtain the query mask.
[0135] S502, if it is determined that there is a key identifier to be queried in the compressed data block based on the query mask and the standard mask, the key identifier to be queried is processed using the standard mask to obtain the compressed key identifier corresponding to the key identifier to be queried.
[0136] Understandably, since the standard mask can represent the character positions retained by the compressed key identifier, the query mask and the standard mask can be processed to determine whether the key identifier to be queried has similar character characteristics to the key identifier in the compressed data block. If so, it is determined that the key identifier to be queried exists in the compressed data block; if not, it is determined that the key identifier to be queried does not exist in the compressed data block. At this time, the next compressed data block can be queried.
[0137] Optionally, the process of compressing the compressed data block can be referenced, and a standard mask can be used to process the key identifier to be queried in the same way to obtain the compressed key identifier corresponding to the key identifier to be queried.
[0138] For example, the reserved character positions in the key identifier to be queried can be determined according to the preset character retention rules and the character information at each character position in the standard mask; then, the compressed key identifier can be determined according to the character located at the reserved character position in the key identifier to be queried.
[0139] S503, based on the binary search principle, uses the compressed key identifier corresponding to the key identifier to be queried to perform a query in the compressed data block to obtain the storage location of the key identifier to be queried in the compressed data block.
[0140] Optionally, the compressed key identifier corresponding to the key identifier to be queried can be compared with the compressed key identifier corresponding to each second key identifier in the compressed data block; then, based on the comparison results, the storage location of the key identifier to be queried in the compressed data block can be determined.
[0141] It is worth noting that since the compressed key identifiers corresponding to each second key identifier are ordered sequentially, a binary search method can be used to improve query efficiency. The compressed key identifier corresponding to the key identifier to be queried is compared with the compressed key identifiers corresponding to each second key identifier in the compressed data block to determine the storage location of the key identifier to be queried in the compressed data block.
[0142] In this embodiment of the application, by using the method of generating compressed key identifiers in compressed data blocks, the same processing is performed on the key identifier to be queried, so as to query the storage location of the key identifier to be queried in the compressed data block, which can ensure the rationality of key identifier query.
[0143] Based on the above embodiments, in this application embodiment, the initial mask is a binary value; furthermore, an optional method for storing the result judgment is provided, such as... Figure 6 As shown, the specific steps include:
[0144] S601, invert the standard mask to obtain the second reference mask.
[0145] The so-called second reference mask represents the character bits in the compressed data block whose second key identifiers are consistent with the first key identifiers.
[0146] It is worth noting that since the standard mask represents the character bits that are not consistent with the first key identifier, in order to determine the character bits in the compressed data block that are consistent with the first key identifier, the standard mask can be inverted to obtain the second reference mask.
[0147] S602, perform a binary AND operation on the query mask and the second reference mask to obtain the updated query mask.
[0148] Optionally, an updated query mask that can characterize the consistency between the query mask and the second reference mask in character position can be obtained by performing a binary AND operation on the query mask and the second reference mask.
[0149] S604, determine whether the updated query mask is consistent with the second reference mask. If yes, execute S604; otherwise, execute S605.
[0150] If the updated query mask matches the second reference mask, proceed to step S604; if the updated query mask does not match the second reference mask, proceed to step S605.
[0151] S604, It has been determined that a key identifier to be queried exists in the compressed data block.
[0152] Understandably, if the updated query mask matches the second reference mask, it proves that the reserved character bits determined when processing the key identifier to be queried and each second key identifier in the data block using the aforementioned key identifier compression logic are consistent. Therefore, it can be determined that the key identifier to be queried exists in the compressed data block. At this point, the standard mask can be used to process the key identifier to be queried to obtain the compressed key identifier corresponding to the key identifier to be queried. Then, based on the binary search principle, the compressed key identifier corresponding to the key identifier to be queried is used to search within the compressed data block to obtain the storage location of the key identifier to be queried in the compressed data block.
[0153] S605, Determined that the key identifier to be queried does not exist in the compressed data block.
[0154] Understandably, if the updated query mask is inconsistent with the second reference mask, it proves that the reserved character bits determined by the key identifier to be queried and each second key identifier in the data block are inconsistent when the above key identifier compression logic is applied. Therefore, it can be determined that the key identifier to be queried does not exist in the compressed data block. At this time, the above query logic can be applied to the next compressed data block.
[0155] In this embodiment, a second reference mask is obtained by inverting the standard mask, and an updated query mask is obtained by performing a binary AND operation between the query mask and the second reference mask. Based on the consistency between the updated query mask and the second reference mask, it is determined whether there is a key identifier to be queried in the compressed data block, which can ensure the reliability of key identifier query.
[0156] Figure 7 This is a flowchart illustrating a data storage method in another embodiment. Based on the above embodiments, this embodiment provides an optional example of a data storage method. (In conjunction with...) Figure 7 The specific implementation process is as follows:
[0157] S701, Obtain the data block to be stored, which contains the key identifiers in sequential order.
[0158] S702, for each second key identifier in the data block to be stored, based on the consistency between the second key identifier and the first key identifier in the data block to be stored at each character position, select the mask at each character position from the binary characters to obtain the initial mask corresponding to the second key identifier.
[0159] Among them, the first key identifier is the starting key identifier among all key identifiers; each second key identifier is the key identifier other than the first key identifier among all key identifiers.
[0160] S703, take the initial mask corresponding to the first key identifier in each second key identifier as the first reference mask, and update the first reference mask according to the initial mask corresponding to each third key identifier to obtain the backup mask.
[0161] Among them, each third key identifier is a key identifier other than the first key identifier among each second key identifier.
[0162] Optionally, based on the order of the third key identifiers, each third key identifier is used as the target key identifier in sequence; for each target key identifier, a binary AND operation is performed on the intermediate mask and the initial mask corresponding to the target key identifier to obtain the updated mask, and the updated mask corresponding to the last target key identifier is used as the backup mask; wherein, when the target key identifier is the first key identifier among the third key identifiers, the intermediate mask is the first reference mask; when the target key identifier is not the first key identifier among the third key identifiers, the intermediate mask is the updated mask corresponding to the previous key identifier of the target key identifier.
[0163] S704 inverts the spare mask to obtain the standard mask for the data block to be stored.
[0164] S705 determines the reserved character positions in each character position based on the preset character retention rules and the character information at each character position in the standard mask.
[0165] S706, for each second key identifier, determine the compressed key identifier corresponding to the second key identifier based on the character located in the reserved character position in the second key identifier.
[0166] S707 generates compressed data blocks based on the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier.
[0167] For example, after obtaining the data block to be stored {12345678; 12355678; 12365679}, the first key identifier is "12345678". At this time, the initial mask mask1 of the second key identifier "12355678" is "0b11101111", and the initial mask mask2 of the second key identifier "12365679" is "0b11101110".
[0168] Furthermore, performing a bitwise AND operation between mask2 and mask1 yields the backup mask "0b11101110". Inverting the backup mask "0b11101110" gives the standard mask diffMask "0b00010001". Then, compressing the second key identifier "12355678" using diffMask results in "58", and compressing the second key identifier "12365679" results in "69". That is, the compressed data block is {12345678; diffMask=0b00010001; cmprKeyLen=2; 58; 69}. Here, cmprKeyLen represents the number of characters retained after compression.
[0169] The specific processes of S701-S707 described above can be found in the description of the above method embodiments. Their implementation principles and technical effects are similar, and will not be repeated here.
[0170] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0171] Based on the same inventive concept, this application also provides a data storage device for implementing the data storage method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more data storage device embodiments provided below can be found in the limitations of the data storage method described above, and will not be repeated here.
[0172] In one exemplary embodiment, such as Figure 8 As shown, a data storage device 1 is provided, comprising: an acquisition module 10, a first determination module 20, a second determination module 30, and a compression module 40, wherein:
[0173] The acquisition module 10 is used to acquire the data block to be stored; wherein the data block to be stored contains key identifiers in sequential order;
[0174] The first determining module 20 is used to determine the initial mask corresponding to each second key identifier based on the consistency of the first key identifier and each second key identifier in the character position; wherein, the first key identifier is the starting key identifier among the key identifiers; and each second key identifier is the key identifier among the key identifiers other than the first key identifier.
[0175] The second determining module 30 is used to determine the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier;
[0176] Compression module 40 is used to compress each second key identifier in the data block to be stored using a standard mask to obtain a compressed data block, and then stores the compressed data block.
[0177] In an exemplary embodiment, the initial mask is a binary value; the second determining module 30 includes:
[0178] The first determining unit is used to take the initial mask corresponding to the first key identifier in each second key identifier as the first reference mask;
[0179] The second determining unit is used to update the first reference mask according to the initial mask corresponding to each third key identifier to obtain a backup mask; wherein, each third key identifier is a key identifier other than the first key identifier among each second key identifier;
[0180] The third determining unit is used to invert the backup mask to obtain the standard mask of the data block to be stored.
[0181] In one exemplary embodiment, the second determining unit is specifically used for:
[0182] Based on the order of the third key identifiers, each third key identifier is used as the target key identifier in sequence. For each target key identifier, the intermediate mask is updated using the initial mask corresponding to the target key identifier to obtain the updated mask, and the updated mask corresponding to the last target key identifier is used as the backup mask. Wherein, when the target key identifier is the first key identifier among the third key identifiers, the intermediate mask is the first reference mask; when the target key identifier is not the first key identifier among the third key identifiers, the intermediate mask is the updated mask corresponding to the previous key identifier of the target key identifier.
[0183] In one exemplary embodiment, the second determining unit is further configured to:
[0184] Perform a binary AND operation on the initial mask corresponding to the intermediate mask and the target key identifier to obtain the updated mask.
[0185] In one exemplary embodiment, the compression module 40 is specifically used for:
[0186] Based on the preset character retention rules and the character information at each character position in the standard mask, the reserved character positions in each character position are determined; for each second key identifier, the compressed key identifier corresponding to the second key identifier is determined based on the character located at the reserved character position in the second key identifier; based on the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier, a compressed data block is generated.
[0187] In an exemplary embodiment, the initial mask is a binary value; the first determining module 20 is specifically used for:
[0188] For each second key identifier, based on the consistency between the second key identifier and the first key identifier at each character position, the mask at each character position is selected from the binary characters to obtain the initial mask corresponding to the second key identifier.
[0189] In an exemplary embodiment, after storing the compressed data block, the data storage device 1 further includes a data query module, wherein the data query module is specifically used for:
[0190] In response to a key query request, if the key identifier to be queried is inconsistent with the first key identifier, a query mask is generated based on the consistency of the key identifier to be queried and the first key identifier at each character position. If the query mask and the standard mask determine that the key identifier to be queried exists in the compressed data block, the standard mask is used to process the key identifier to be queried to obtain the compressed key identifier corresponding to the key identifier to be queried. Based on the binary search principle, the compressed key identifier corresponding to the key identifier to be queried is used to perform a query in the compressed data block to obtain the storage location of the key identifier to be queried in the compressed data block.
[0191] In one exemplary embodiment, the data query module is further configured to:
[0192] Invert the standard mask to obtain the second reference mask; perform a binary AND operation between the query mask and the second reference mask to obtain the updated query mask; if the updated query mask matches the second reference mask, it is determined that the key identifier to be queried exists in the compressed data block; if the updated query mask does not match the second reference mask, it is determined that the key identifier to be queried does not exist in the compressed data block.
[0193] Each module in the aforementioned data storage device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.
[0194] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 9 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores key-value data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a data storage method.
[0195] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0196] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0197] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.
[0198] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0199] It should be noted that the data involved in this application (including but not limited to key-value data) is all data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0200] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0201] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0202] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A data storage method, characterized in that, The method includes: Obtain the data block to be stored; wherein the data block to be stored contains key identifiers in sequential order; Based on the consistency of the first key identifier and each second key identifier in character position, the initial mask corresponding to each second key identifier is determined; wherein, the first key identifier is the start key identifier among all key identifiers; each second key identifier is the key identifier among all key identifiers excluding the first key identifier; The standard mask of the data block to be stored is determined based on the initial mask corresponding to each second key identifier; Using the standard mask, each second key identifier in the data block to be stored is compressed to obtain a compressed data block, and the compressed data block is stored.
2. The method according to claim 1, characterized in that, The initial mask is a binary value; determining the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier includes: Use the initial mask corresponding to the first key identifier in each second key identifier as the first reference mask; The first reference mask is updated based on the initial mask corresponding to each third key identifier to obtain the backup mask; wherein, each third key identifier is a key identifier other than the first key identifier among each second key identifier; The backup mask is inverted to obtain the standard mask for the data block to be stored.
3. The method according to claim 2, characterized in that, The step of updating the first reference mask according to the initial mask corresponding to each third key identifier to obtain a backup mask includes: Based on the order of the third key identifiers, each third key identifier is used as the target key identifier in turn; For each target key identifier, the intermediate mask is updated using the initial mask corresponding to the target key identifier to obtain the updated mask, and the updated mask corresponding to the last target key identifier is used as the backup mask. Wherein, if the target key identifier is the first key identifier among all third key identifiers, the intermediate mask is the first reference mask; if the target key identifier is not the first key identifier among all third key identifiers, the intermediate mask is the updated mask corresponding to the previous key identifier of the target key identifier.
4. The method according to claim 3, characterized in that, The step of updating the intermediate mask using the initial mask corresponding to the target key identifier to obtain the updated mask includes: Perform a binary AND operation on the intermediate mask and the initial mask corresponding to the target key identifier to obtain the updated mask.
5. The method according to any one of claims 1-4, characterized in that, The step of using the standard mask to compress each second key identifier in the data block to be stored to obtain a compressed data block includes: Based on the preset character retention rules and the character information at each character position in the standard mask, the reserved character positions in each character position are determined; For each second key identifier, the compressed key identifier corresponding to the second key identifier is determined based on the character located in the reserved character position in the second key identifier; Based on the first key identifier, the standard mask, and the compressed key identifiers corresponding to each second key identifier, a compressed data block is generated.
6. The method according to any one of claims 1-4, characterized in that, The initial mask is a binary value; the step of determining the initial mask corresponding to each second key identifier based on the consistency of the first key identifier and each second key identifier in character position includes: For each second key identifier, based on the consistency between the second key identifier and the first key identifier at each character position, a mask is selected from the binary characters at each character position to obtain the initial mask corresponding to the second key identifier.
7. The method according to any one of claims 1-4, characterized in that, After storing the compressed data block, the method further includes: In response to a key query request, if the key identifier to be queried is inconsistent with the first key identifier, a query mask is generated based on the consistency between the key identifier to be queried and the first key identifier at each character position; If it is determined that the key identifier to be queried exists in the compressed data block based on the query mask and the standard mask, the key identifier to be queried is processed using the standard mask to obtain the compressed key identifier corresponding to the key identifier to be queried. Based on the binary search principle, the compressed key identifier corresponding to the key identifier to be queried is used to search in the compressed data block to obtain the storage location of the key identifier to be queried in the compressed data block.
8. The method according to claim 7, characterized in that, The initial mask is a binary value; determining the presence of the query key identifier in the compressed data block based on the query mask and the standard mask includes: Invert the standard mask to obtain the second reference mask; Perform a binary AND operation between the query mask and the second reference mask to obtain the updated query mask; If the updated query mask matches the second reference mask, it is determined that the key identifier to be queried exists in the compressed data block; If the updated query mask is inconsistent with the second reference mask, it is determined that the key identifier to be queried does not exist in the compressed data block.
9. A data storage device, characterized in that, The device includes: An acquisition module is used to acquire a data block to be stored; wherein the data block to be stored contains key identifiers in sequential order; The first determining module is used to determine the initial mask corresponding to each second key identifier based on the consistency of the first key identifier and each second key identifier in the character position; wherein, the first key identifier is the starting key identifier among the key identifiers; and each second key identifier is the key identifier among the key identifiers other than the first key identifier. The second determining module is used to determine the standard mask of the data block to be stored based on the initial mask corresponding to each second key identifier; The compression module is used to compress each second key identifier in the data block to be stored using the standard mask to obtain a compressed data block, and to store the compressed data block.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.
11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.
12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 8.