Data processing method, computer device and storage medium

Abstract

The application discloses a data processing method, a computer device and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: receiving a data storage request, wherein the data storage request carries a key-value pair to be stored, and the key values in the key-value pair are globally ordered; generating a new file containing one or more key-value pairs according to the order of the key values in the key-value pair, and storing the new file; and the file identifiers of all files containing the key-value pairs are globally ordered. A new leaf node is added after the last leaf node in a target tree, the target tree is updated according to the new leaf node, the new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file. When data storage is performed, indexing in the target tree is not needed, the new file can be directly generated for storage, and the target tree can be directly updated by adding the new leaf node, so that the data storage efficiency can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a data processing method, a computer device and a storage medium. BACKGROUND

[0002] At present, data storage is often based on tree structure. Specifically, after receiving a key-value pair to be stored, the key-value pair is first indexed from the root node to a leaf node in the tree according to the key value in the key-value pair. Then the key-value pair is added to the data of the leaf node to update the leaf node, and the updated leaf node is used to update the nodes layer by layer upwards until the root node is updated to complete the update of the tree. Then, the data of each node in the updated tree is stored to the disk to complete the persistence of the tree. However, the data storage efficiency of this way is low. SUMMARY

[0003] The present application provides a data processing method, a computer device and a storage medium, which can improve the data storage efficiency. The technical solution is as follows:

[0004] In a first aspect, a data processing method is provided, which includes:

[0005] receiving a data storage request, the data storage request carrying a key-value pair to be stored, the key value in the key-value pair being globally ordered;

[0006] generating a new file containing one or more key-value pairs according to the order of the key value in the key-value pair, and storing the new file; wherein the file identifiers of all files containing the key-value pair are globally ordered;

[0007] adding a new leaf node after the last leaf node in the target tree, updating the target tree according to the new leaf node, the new leaf node containing the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file

[0008] In the present application, when data storage is performed, indexing in the target tree is not required, and a new file can be directly generated for storage, and accordingly a new leaf node can be directly added in the target tree to update the target tree, so that the data storage efficiency can be improved.

[0009] Optionally, the target tree is an n-ary tree, and n is an integer greater than or equal to 2.

[0010] Optionally, the branch node in the target tree includes a child node list, and the child node list includes the minimum key value and the maximum key value in each child node of the branch node.

[0011] Optionally, the target tree is located in the memory, and the files are stored in the disk, and the method further comprises:

[0012] After rebooting, the target tree is reconstructed in the memory according to all the files stored in the disk.

[0013] Optionally, the key-value pair comprises a target key-value pair, the key value in the target key-value pair comprises a block number and a key value in a block identified by the block number, and the value value in the target key-value pair is a value value in the block.

[0014] Optionally, the generating a new file containing one or more key-value pairs according to the order of the key values in the key-value pairs and storing the new file comprises:

[0015] Upon receiving each data storage request, saving all the key-value pairs carried by the data storage request in the memory according to the order of the key values in the key-value pairs;

[0016] If the difference between the total data amount of one or more key-value pairs saved in the memory in order and the preset data amount is less than or equal to the preset difference, generating a new file containing the one or more key-value pairs;

[0017] Storing the new file into the disk.

[0018] Optionally, the updating the target tree according to the new leaf node comprises:

[0019] updating the nodes in the target tree layer by layer upwards according to the new leaf node until the root node is updated; wherein, in the process of updating the nodes layer by layer upwards, if the original node in the target tree needs to be updated, the original node is added with a write lock, and the write lock is released after the update is completed.

[0020] Optionally, the method further comprises:

[0021] receiving a first data query request, the first data query request carrying a target key value;

[0022] determining a first leaf node to which the target key value is indexed in the target tree;

[0023] obtaining a value value in a key-value pair to which the target key value belongs from a file identified by the file identifier in the first leaf node.

[0024] Optionally, the determining a first leaf node to which the target key value is indexed in the target tree comprises:

[0025] indexing nodes layer by layer from a root node downwards in the target tree according to the target key value until the first leaf node is indexed;

[0026] wherein, in the process of indexing nodes layer by layer downwards, a read lock is added to each indexed node, a next node is indexed according to the indexed node and the target key value, and the read lock of the indexed node is released after the next node is indexed; the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0027] Optionally, the target tree is an n-ary tree, the n is an integer greater than or equal to 2, and the method further comprises:

[0028] receiving a second data query request, the second data query request carrying a target file identifier;

[0029] determining index information in the target tree according to the target file identifier, the n, and a tree height of the target tree;

[0030] indexing a second leaf node in the target tree according to the index information, the second leaf node being a leaf node containing the target file identifier.

[0031] Optionally, the determining the index information in the target tree according to the target file identifier, the n, and the tree height of the target tree comprises:

[0032] adding a read lock to a root node of the target tree, the target tree having a tree height of m, the m being an integer greater than or equal to 2;

[0033] determining a target value corresponding to the target file identifier, the target value corresponding to the target file identifier being an ordering sequence number of a file identified by the target file identifier among all files;

[0034] letting i be the m, and letting a specified parameter be the target value corresponding to the target file identifier;

[0035] dividing the specified parameter by the n to obtain a quotient and a remainder;

[0036] taking the remainder as index information of the i-th layer, the index information of the i-th layer being used to indicate an ordering sequence number of a node of the i-th layer to be indexed among all child nodes of a parent node of the i-th layer;

[0037] if i is not 2, letting i be i-1, letting the specified parameter be the quotient, and re-executing the step of dividing the specified parameter by the n to obtain a quotient and a remainder and subsequent steps until i is 2;

[0038] If i is 2, the index refers to the root node, and a read lock of the root node is released after the index refers to the root node.

[0039] In a second aspect, the present application provides a data processing apparatus, the apparatus comprising:

[0040] A first receiving module is configured to receive a data storage request, the data storage request carrying a key-value pair to be stored, the key value in the key-value pair being globally ordered;

[0041] A generating module is configured to generate a new file containing one or more key-value pairs according to the order of the key value in the key-value pair, and store the new file; wherein the file identifiers of all files containing the key-value pair are globally ordered;

[0042] An updating module is configured to add a new leaf node after the last leaf node in the target tree, update the target tree according to the new leaf node, the new leaf node containing the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file.

[0043] In a third aspect, a computer device is provided, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being executed by the processor to implement the data processing method of the first aspect.

[0044] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the data processing method of the first aspect.

[0045] In a fifth aspect, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the steps of the data processing method of the first aspect.

[0046] It can be understood that the beneficial effects of the second aspect, the third aspect, the fourth aspect and the fifth aspect described above can be referred to the related description in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 is a schematic diagram of a target tree provided by an embodiment of the present application;

[0049] Figure 2 is a schematic diagram of a key-value pair provided by an embodiment of the present application;

[0050] Figure 3 is another schematic diagram of a key-value pair provided by an embodiment of the present application;

[0051] Figure 4 is a flowchart of a data processing method provided by an embodiment of the present application;

[0052] Figure 5 is a schematic diagram of index information provided by an embodiment of the present application;

[0053] Figure 6 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0054] Figure 7 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0056] It should be understood that the "multiple" mentioned in the present application refers to two or more than two. In the description of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in the present application only describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases: A exists alone, A and B exist together, and B exists alone. In addition, in order to clearly describe the technical solutions of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different.

[0057] The phrase "one embodiment" or "some embodiments" appearing in the present application are used as antecedents for the phrase "including a feature, structure, or characteristic" where no other feature, structure, or characteristic to which the phrase "one embodiment" or "some embodiments" refers is present. Use of the phrase "in one embodiment" or "in some embodiments" is not a limitation on the scope or on the meaning of the phrase "including a feature, structure, or characteristic" such that the feature, structure, or characteristic in question is present in one embodiment but is not present in all embodiments, unless specifically stated otherwise.

[0058] The application scenarios of the embodiments of the present application are described below.

[0059] With the continuous development of computer technology, data has become an important resource and asset. The importance of data is reflected in supporting decision-making, promoting business development, promoting innovation, improving customer experience, improving efficiency and effectiveness, managing risks, and promoting governance. With the continuous generation of data, how to efficiently store data is a problem that needs to be solved.

[0060] In the related art, data is usually stored based on a tree structure (such as a B+ tree). Specifically, after receiving a key-value pair to be stored, the key-value pair is first indexed to a leaf node in the tree according to the key value in the key-value pair, and then the key-value pair is added to the leaf node and the tree is updated. Finally, each node in the updated tree is stored to the disk. However, this way of storing data is inefficient.

[0061] Therefore, the embodiments of the present application propose a data processing method. When storing data, no indexing in the target tree is required, and a new file can be directly generated for storage. Accordingly, a new leaf node can be directly added to update the target tree, thereby improving the data storage efficiency.

[0062] The data processing method provided by the embodiments of the present application can be applied in a system that has a data processing requirement. In the embodiments of the present application, the system is referred to as a storage layer system. The storage layer system is described below.

[0063] The embodiments of the present application are applied to a scenario in which an external system stores data based on the storage layer system. For example, the embodiments of the present application can be applied to a scenario in which data is stored based on a target tree.

[0064] For example, after a blockchain system executes all transactions in a block, a series of ledger data is generated. The ledger data can include multiple key-value pairs. In this case, the data processing method provided by the embodiments of the present application can be used to store the ledger data based on the target tree.

[0065] The external system to which the storage layer system interfaces is a system that has a data storage requirement. For example, the external system to which the storage layer system interfaces can be a database system, a blockchain system, and the like, and the embodiments of the present application do not limit this.

[0066] The storage layer system is configured to receive, process, and reply to various requests sent by the external system. For example, the request sent by the external system can include a data storage request, a data query request, and the like, and the embodiments of the present application do not limit this.

[0067] In some embodiments, the storage layer system can interface with a database system. The database system can interface with other systems that have a data storage requirement, such as a blockchain system and the like. For example, when the blockchain system needs to store data, the data can be first sent to the database system, and then sent to the storage layer system by the database system, and the storage layer system can store the data based on the target tree.

[0068] Optionally, the storage layer system and the database system can be independent systems, in which case the storage layer system and the database system can run on independent physical machines. Alternatively, the database system can include the storage layer system, in which case the database system includes a specific system that interfaces with the blockchain system and runs on the same physical machine as the storage layer system.

[0069] Next, the structure of the target tree provided by the embodiments of the present application is described.

[0070] The target tree is a tree structure. The target tree includes branch nodes and leaf nodes at the lowest level of the target tree, wherein a node with a child node number of 0 is a leaf node, a node with a child node number greater than or equal to 1 is a branch node, and the branch nodes and the leaf nodes are logical nodes. The uppermost branch node in the target tree can be referred to as a root node, which is a branch node without a parent node in the target tree.

[0071] Optionally, the target tree can be an n-ary tree, that is, the number of child nodes of each branch node in the target tree is at most n, and n is an integer greater than or equal to 2. Any branch node in the n-ary tree can have at most n child nodes. Moreover, for branch nodes at the same level in the n-ary tree, a new branch node can be added after the last branch node at the same level only when the number of child nodes of the last branch node at the same level reaches n.

[0072] The leaf node stores a file identifier, a maximum key value, and a minimum key value.

[0073] The file identifier is used to identify a file stored in the storage layer system. For example, the file identifier can be a file identity (ID), a file name, and the like, and the embodiments of the present application do not limit this.

[0074] The minimum key value in the leaf node is the minimum key value in the file identified by the file identifier in the leaf node. The maximum key value in the leaf node is the maximum key value in the file identified by the file identifier in the leaf node.

[0075] In the embodiment of the present application, the leaf nodes in the target tree correspond to files one by one. That is, one leaf node corresponds to the file identifier of a file and the minimum key value and the maximum key value in the file.

[0076] In the embodiment of the present application, the storage layer system can store one or more files, and each file can contain one or more key-value pairs. The file identifiers of all the files stored in the storage layer system are globally ordered, that is, the file identifier of any file is ordered in all the files stored in the storage layer system. In addition, the key values in all the key-value pairs contained in these files are also globally ordered, that is, the key value in any key-value pair contained in any file is ordered in all the key-value pairs in all the files stored in the storage layer system.

[0077] Optionally, the global order of the key values in the key-value pairs can be in the order of increasing dictionary order.

[0078] In this case, the minimum key value in a file refers to the key value with the earliest order among the key values in all the key-value pairs in the file. The maximum key value in a file refers to the key value with the latest order among the key values in all the key-value pairs in the file.

[0079] It should be noted that in the computer field, dictionary order can be used to indicate the size relationship between any two strings. The size relationship between two strings depends on the size relationship of the American Standard Code for Information Interchange (ASCII) codes of the characters from left to right in the string.

[0080] Optionally, the global order of the file identifiers of the files can be in the order of increasing dictionary order.

[0081] For example, the file identifier of a file can start from 0 and increase by a preset value.

[0082] The preset value can be set in advance. For example, the preset value can be set to 1, 2, 3, etc., which is not limited in the embodiment of the present application.

[0083] In this case, the file identifiers of all the files are an arithmetic sequence. For example, assuming that the preset value is 1, the file identifiers of the first file, the second file, the third file,..., and the kth file generated can be 0, 1, 2,..., and k-1, respectively.

[0084] In some embodiments, a global continuously increasing counter can be maintained inside the storage layer system, and the counter can be used to generate the file identifier. In this case, the storage layer system can take the current value of the counter as the file identifier of a new file each time a new file is generated, so that the files stored in the storage layer system are a batch of files with continuously increasing file identifiers.

[0085] The branch node includes a child node list, and the child node list includes the minimum key value and the maximum key value in each child node of the branch node. For example, the range from the minimum key value in a child node to the maximum key value in the child node can be referred to as the key range of the child node.

[0086] The child nodes of a branch node can be branch nodes or leaf nodes of the next layer. The minimum key value in a branch node is the minimum key value in all the key values in the child node list of the branch node, that is, the key value with the earliest order in all the key values in the child node list; and the maximum key value in the branch node is the maximum key value in all the key values in the child node list of the branch node, that is, the key value with the latest order in all the key values in the child node list. The minimum key value and the maximum key value in a leaf node are the minimum key value and the maximum key value stored in the leaf node.

[0087] For example, Figure 1 is a schematic diagram of a target tree provided by an embodiment of the present application. Referring to Figure 1 , the target tree is a 2-ary tree, and the target tree includes a plurality of leaf nodes (for example, the leaf nodes n4, n5, and n6 in Figure 1 ) and a plurality of branch nodes (for example, the branch nodes n1, n2, and n3 in Figure 1 ). The child nodes of the node n1 include the nodes n2 and n3, the child nodes of the node n2 include the nodes n4 and n5, and the child nodes of the node n3 include the node n6. In the embodiment of the present application, the uppermost branch node in the target tree can be referred to as a root node, for example, Figure 1 the node n1 in

[0088] Figure 1The leaf node n4 in the tree holds the file identifier 0, the minimum key value 1, and the maximum key value 3. The leaf node n5 holds the file identifier 1, the minimum key value 4, and the maximum key value 6. The leaf node n6 holds the file identifier 2, the minimum key value 7, and the maximum key value 9.

[0089] Figure 1 The branch node n2 in the tree holds a list of child nodes, which includes the minimum key value 1 and the maximum key value 3 in the leaf node n4, and includes the minimum key value 4 and the maximum key value 6 in the leaf node n5.

[0090] Figure 1 The branch node n3 in the tree holds a list of child nodes, which includes the minimum key value 7 and the maximum key value 9 in the leaf node n6.

[0091] Figure 1 The branch node n1 in the tree holds a list of child nodes, which includes the minimum key value 1 and the maximum key value 6 in the branch node n2, and includes the minimum key value 7 and the maximum key value 9 in the branch node n3.

[0092] In an embodiment of the present application, the storage layer system can store the key-value pairs sent by the external system based on the target tree. For example, the target tree can also be referred to as a solid branch tree.

[0093] In some cases, the key values in the key-value pairs sent by the external system to the storage layer system are globally ordered, and the concept of global order has been explained above and will not be repeated here.

[0094] In some embodiments, the storage layer system can be interfaced with a database system, and the database system can be interfaced with a blockchain system. In this case, the blockchain system can send the key-value pairs to be stored to the database system, and the database system can send a data storage request carrying the key-value pairs to the storage layer system after receiving the key-value pairs.

[0095] As an example, the key-value pairs to be stored sent by the blockchain system to the database system can be block data. Since the block data is strictly organized according to the block number, i.e., the key values in the block data are block numbers, the key values in the block data are naturally globally ordered. In this case, the database system can directly send the block data to the storage layer system for storage in the data storage request.

[0096] As another example, the key-value pairs that a blockchain system sends to a database system for storage can be ledger data within a block. Ledger data within a block consists of multiple key-value pairs; the key values ​​in multiple key-value pairs within the same block are different, but the key values ​​in key-value pairs in different blocks may be the same.

[0097] In this scenario, the database system can first process the ledger data. Specifically, the database system can introduce block numbers into the ledger data to construct globally ordered key-value pairs. These key-value pairs are then sent to the storage layer system along with the data storage request for storage. This ensures that the keys in the key-value pairs sent by the database system to the storage layer system are globally ordered.

[0098] Optionally, in this embodiment of the application, such a key-value pair may be referred to as a target key-value pair. The key value in the target key-value pair includes the block number and the key value within the block identified by the block number (i.e., the key value in the key-value pair in the ledger data), and the value value in the target key-value pair is the value value within that block (i.e., the value value in the key-value pair in the ledger data).

[0099] In some embodiments, the key value in the target key-value pair can be obtained by concatenating the block number and the key value within the block. Of course, the key value in the target key-value pair can also be obtained in other ways, and this application embodiment does not limit this.

[0100] Since block numbers are naturally globally ordered, and the key values ​​within a block are different, the key values ​​in a target key-value pair containing the block number and the key values ​​within the block identified by that block number are also globally ordered.

[0101] For example, if a block with block number 00000001 contains multiple key-value pairs, for each of these key-value pairs, the block number 00000001 can be added before the key value in that key-value pair to obtain a new key value.

[0102] For example, such as Figure 2 As shown in Figure (a), block 1 with block number 00000001 contains key-value pairs bbbb:{value1...}, bggg:{value2...}, caaa:{value3...}, and xaab:{value4...}. Figure 2 As shown in Figure (b), block 2 with block number 00000002 contains key-value pairs aabb:{value5...}, bggg:{value6...}, and bcde:{value7...}.

[0103] Then, after the blockchain system sends the key-value pairs to the database system, the database system can introduce the block number 00000001 in the key-value pairs in block 1, specifically, concatenate 00000001 before the key value in each key-value pair in block 1, to obtain the target key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, 00000001xaab:{value4...}. The block number 00000002 is introduced in the key-value pairs in block 2, specifically, concatenate 00000002 before the key value in each key-value pair in block 2, to obtain the target key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, 00000002bcde:{value7...}. The key values in the target key-value pairs obtained in this way are globally ordered.

[0104] It should be noted that the storage layer system in the embodiments of the present application is not limited to storing the key-value pairs in the blockchain system proposed above, but can also store other key-value pairs, as long as the key values in the key-value pairs are globally ordered.

[0105] The process of constructing an initial target tree by the storage layer system will be described below, which can include the following steps (1)-(3):

[0106] (1) The storage layer system receives a data storage request, which carries a key-value pair to be stored, and the key value in the key-value pair is globally ordered.

[0107] The data storage request is used to instruct the storage layer system to store the key-value pair to be stored. In some embodiments, the data storage request can be sent by an external system. For example, the blockchain system or the database system.

[0108] In this case, the key values in all key-value pairs to be stored sent by the external system to the storage layer system are ordered. After the storage layer system stores these key-value pairs, the key values in all key-value pairs stored in the storage layer system are also ordered.

[0109] (2) The storage layer system generates a new file containing one or more key-value pairs according to the order of the key values in the key-value pairs, and stores the new file; wherein the file identifiers of all files containing key-value pairs are globally ordered.

[0110] The key values in the key-value pairs received by the storage layer system are in order. In this case, the storage layer system generates a new file according to the order of the key values in the key-value pairs, and the key values in the key-value pairs in the new file are also in order.

[0111] Optionally, when storing the new file, the storage layer system can store the new file in a disk.

[0112] For example, the storage layer system can include one or more disks, and the storage layer system can store all the files in the same disk or store all the files in different disks, and the embodiments of the present application do not limit this.

[0113] Optionally, if the storage layer system stores all the files in different disks, the storage layer system can also obtain the disk identifier of the disk where the new file is stored after storing the new file.

[0114] In some embodiments, the operation of step (2) can be that the storage layer system saves all the key-value pairs carried by the data storage request in the memory according to the order of the key values in the key-value pairs carried by the data storage request each time the storage layer system receives a data storage request. If the difference between the total data amount of one or more key-value pairs saved in order in the memory and the preset data amount is less than or equal to the preset difference, the storage layer system generates a new file containing the one or more key-value pairs. The storage layer system stores the new file in a disk.

[0115] The preset data amount can be set in advance. For example, the preset data amount can be set to 8 megabytes (MB), 9 MB, 10 MB, etc., and the embodiments of the present application do not limit this.

[0116] Optionally, the preset data amount can be set according to an integer multiple of the block size of the storage layer system. For example, if the block size is 4 MB, the preset data amount can be set to 4 MB, 8 MB, 12 MB, etc. The block size of the storage layer system refers to the smallest unit used by the storage layer system to organize and manage data storage.

[0117] The preset difference can be set in advance. For example, the preset difference can be set to 100 kilobytes (KB), 200 KB, 300 KB, etc., and the embodiments of the present application do not limit this.

[0118] It should be noted that the data amount of each key-value pair is generally close, and the data amount of a key-value pair is usually within a predetermined range. In the embodiments of the present application, the preset data amount can be much larger than the data amount of a key-value pair. The preset difference can be close to the data amount of a key-value pair, such as equal to or slightly larger than the data amount of a key-value pair.

[0119] The storage layer system can save all the key-value pairs carried by each received data storage request in order in the memory. In this case, if the total data amount of one or more key-value pairs currently saved in order in the memory is close to the preset data amount (i.e., the difference between the total data amount and the preset data amount is less than or equal to the preset difference), the one or more key-value pairs can be stored in the form of a new file in the disk. In this way, batch storage of key-value pairs can be implemented, thereby improving the data storage efficiency. Moreover, in this case, the file size of each file stored in the disk is basically consistent, and is close to the preset data amount, thereby facilitating file management.

[0120] It should be noted that if the total data amount of all the key-value pairs currently saved in the memory is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference, it indicates that the total data amount of the key-value pairs currently saved in the memory is greatly different from the preset data amount, and the condition for generating a new file is not met. In this case, the storage layer system can continue to receive data storage requests sent by the external system, and save the key-value pairs carried by the data storage requests in the memory, until the difference between the total data amount of the key-value pairs saved in order in the memory and the preset data amount is less than or equal to the preset difference, and then a new file is generated and stored.

[0121] The generated new file is stored in the disk, i.e., the persistence of the new file is implemented. Moreover, after the new file is stored in the disk, one or more key-value pairs in the new file are deleted from the memory.

[0122] It should be noted that the key values in the key-value pairs saved in the memory are in order. In this case, the new file is generated according to the key-value pairs saved in the memory, and the key values in all the key-value pairs in the new file are also in order.

[0123] It should be further noted that since one file can include one or more key-value pairs, the key-value pairs in one file can cover the data of one block or multiple blocks.

[0124] For example, as shown in FIG. 1, the storage layer system can receive a data storage request sent by the external system, and save the key-value pairs carried by the data storage request in the memory. In this case, if the total data amount of the key-value pairs currently saved in order in the memory is close to the preset data amount, the key-value pairs can be stored in the form of a new file in the disk. Figure 3As shown, the storage layer system receives a data storage request 1 carrying key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, 00000001xaab:{value4...}. The storage layer system saves the key-value pairs carried in the data storage request 1 in sequence in the memory, and the key-value pairs currently saved in the memory include 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, 00000001xaab:{value4...}. Since the total data amount of these key-value pairs is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference, the condition for generating a new file is not met.

[0125] Afterwards, the storage layer system receives a data storage request 2 carrying key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, 00000002bcde:{value7...}. The storage layer system saves the key-value pairs carried in the data storage request 2 in sequence in the memory. Among them, after saving the key-value pair 00000002bggg:{value6...} in the memory, the storage layer system finds that the difference between the total data amount of the key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} currently saved in the memory and the preset data amount is less than or equal to the preset difference, meeting the condition for generating a new file, and thus a new file containing the key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} is generated and stored to the disk. And the storage layer system saves the key-value pair 00000002bcde:{value7...} carried in the data storage request 2 in the memory.

[0126] Afterwards, the storage layer system receives a data storage request 3 carrying a key-value pair 00000003bcbb:{value1...}. The storage layer system saves the key-value pair carried in the data storage request 3 in sequence in the memory, and then the key-value pairs currently saved in the memory include 00000002bcde:{value7...} and 00000003bcbb:{value1...}. Since the total data amount of these key-value pairs is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference, the new file generation condition is not met. The storage layer system continues to wait for the next data storage request.

[0127] (3) The storage layer system generates a new leaf node, and constructs an initial target tree according to the new leaf node, the new leaf node containing the file identifier of the new file, and the minimum key value and the maximum key value in all key-value pairs in the new file.

[0128] Optionally, if the storage layer system dispersively stores all files in different disks, the new leaf node can further contain the disk identifier of the disk where the new file is located.

[0129] After generating the new leaf node, the storage layer system can newly create a branch node as the parent node of the new leaf node, the branch node containing a child node list, the child node list including the minimum key value and the maximum key value in the new leaf node. Since the number of current leaf nodes is 1, the branch node is the root node, and thus the construction of the initial target tree is completed.

[0130] In some embodiments, a branch node in the target tree can include a child node list. The child node list in a branch node contains the node information of each child node of the branch node.

[0131] For example, the node information of a node can include the pointer of the node, the minimum key value and the maximum key value in the node.

[0132] The pointer of a node is used to point to the memory space where the node is located.

[0133] In this case, the child node list in a branch node includes the node information of all child nodes of the branch node, and the node information of the all child nodes is in sequence in the child node list.

[0134] In some cases, after the storage layer system completes the construction of the initial target tree, it can also save in memory a pointer to the root node of the target tree and a pointer to the last leaf node in the target tree. In this way, the storage layer system can subsequently quickly index to the root node in the target tree according to the pointer to the root node maintained in memory and quickly index to the last leaf node in the target tree according to the pointer to the leaf node maintained in memory.

[0135] In some embodiments, for any one data storage request, if all the key-value pairs carried by the data storage request have been stored in the form of files in the disk and the target tree has been constructed or updated accordingly, the storage layer system can return a data storage success message to the external system that sent the data storage request. For example, the data storage success message can carry the file identifier of the file in which the key-value pairs in the data storage request are located, the disk identifier of the disk in which the file is located, and the like, and the present embodiments do not limit this.

[0136] As an example, the target tree in the present embodiments does not need to be persisted, that is, does not need to be stored in the disk, but can be saved in the memory, so that storage resources can be saved.

[0137] In this case, if the storage layer system restarts due to a crash or the like, after the restart, the storage layer system can reconstruct the target tree in the memory according to all the files stored in the disk. It should be noted that the "file" referred to in the present embodiments refers to the file containing key-value pairs referred to in the present embodiments.

[0138] Specifically, since the file identifiers are globally ordered and each leaf node saves the file identifier of a file and the minimum key value and the maximum key value in the file, the storage layer system can construct all the leaf nodes in the last layer of the target tree in the memory according to all the files stored in the disk, and construct branch nodes layer by layer upwards according to the leaf nodes in the last layer, until the root node is constructed, so that the target tree is reconstructed. After the target tree is reconstructed, the pointer to the root node of the reconstructed target tree and the pointer to the last leaf node in the target tree are maintained in the memory.

[0139] After the initial target tree is constructed through the above steps (1) to (3), the storage layer system can subsequently perform data processing based on the target tree, such as data query, data storage, and the like. The data processing method provided by the present embodiments will be explained and described in detail below.

[0140] Figure 4 is a flowchart of a data processing method provided by the present embodiments, which can be applied to a storage layer system. Referring to Figure 4 , the method comprises the following steps:

[0141] Step 401: The storage layer system receives a data storage request carrying a key-value pair to be stored, the key value in the key-value pair being globally ordered.

[0142] The operation of step 401 is similar to the operation of step (1) described above, and thus will not be repeated here.

[0143] Step 402: The storage layer system generates a new file containing one or more key-value pairs according to the order of the key value in the key-value pair, and stores the new file; wherein the file identifiers of all files containing key-value pairs are globally ordered.

[0144] The operation of step 402 is similar to the operation of step (2) described above, and thus will not be repeated here.

[0145] Step 403: The storage layer system increases a new leaf node after the last leaf node in the target tree, updates the target tree according to the new leaf node, and the new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file.

[0146] Optionally, the storage layer system can first obtain the pointer of the last leaf node in the target tree in the memory, and then increase a new leaf node after the leaf node according to the pointer of the leaf node.

[0147] For example, the last leaf node in the target tree can be located at the rightmost side of the last layer of the target tree, in which case the insertion order of the leaf node is from left to right. Then the storage layer system increases a new leaf node after the last leaf node in the target tree, that is, inserts the new leaf node at the rightmost side of the last layer of the target tree.

[0148] Alternatively, the last leaf node in the target tree can be located at the leftmost side of the last layer of the target tree, in which case the insertion order of the leaf node is from right to left. Then the storage layer system increases a new leaf node after the last leaf node in the target tree, that is, inserts the new leaf node at the leftmost side of the last layer of the target tree.

[0149] In the embodiments of the present application, when data storage is performed, indexing in the target tree is not required, and a new file can be directly generated for storage, and accordingly a new leaf node can be directly added in the target tree to update the target tree, thereby improving the data storage efficiency.

[0150] It should be noted that the updating logic of the target tree in the embodiments of the present application is only one, that is, a new leaf node is inserted after the last leaf node of the whole tree (such as the rightmost or leftmost of the last layer). The traditional tree structure adopts a random insertion logic, which needs to index from the root node to the leaf node first, then insert and store data, and then backtrack to update or split the parent node. Compared with the random insertion logic in the traditional tree structure, the updating logic of the target tree in the embodiments of the present application can avoid the indexing process from the root node to the leaf node, thereby improving the data storage efficiency.

[0151] It should be noted that, since the file identifiers are globally ordered, and each leaf node stores a file identifier of a file, the file identifiers in all leaf nodes in the last layer of the target tree are ordered from left to right or from right to left.

[0152] It should be noted that, in the embodiments of the present application, the tree shape of the whole target tree is relatively fixed, and except for the nodes in the path from the root node to the last leaf node (i.e. the rightmost path or the leftmost path), the remaining nodes are static nodes. A static node refers to a node that does not undergo a modification process and has a determined overall shape.

[0153] In some embodiments, the target tree is an n-ary tree, where n is an integer greater than or equal to 2. In the case of an n-ary tree, the target tree does not need to be split during the process of updating the target tree according to the new leaf node, thereby improving the updating efficiency.

[0154] Optionally, the operation of the storage layer system for updating the target tree according to the new leaf node can be: the storage layer system updates the nodes layer by layer upwards in the target tree according to the new leaf node until the root node is updated. During the process of updating the nodes layer by layer upwards, if an original node in the target tree needs to be updated, a write lock is added to the original node, and the write lock is released after the update is completed. During the process of updating the nodes layer by layer upwards, if a new node is added in the target tree, the new node does not need to be added with a write lock.

[0155] It should be noted that the original node in the target tree is added with a write lock before it is updated in the embodiments of the present application. In this case, during the updating process of the original node, if a query operation on the original node occurs, the query operation needs to wait until the original node is updated and the write lock is released, so as to ensure the accuracy of data query.

[0156] The new node added in the updating process of the target tree does not need to be added with a write lock in the embodiments of the present application. This is because the new node is not visible to the outside before the updating of the target tree is completed, so the data query process naturally does not pass through the new node, and there is no need to add a write lock.

[0157] In some embodiments, the operation of the storage layer system of updating the nodes in the target tree according to the new leaf node layer by layer upwards can include steps a to f.

[0158] Step a: Let the new leaf node be the current node.

[0159] Step b: Determine whether the current node is a new node or an original node in the target tree. If the current node is a new node, perform steps c to e; if the current node is an original node, perform step f.

[0160] Step c: If the current node is a new node, determine whether the current node can be a child node of the last node in the upper layer. If the current node can be a child node of the last node in the upper layer, perform step d; if the current node cannot be a child node of the last node in the upper layer, perform step e.

[0161] For example, in the case of an n-ary tree as the target tree, if the number of child nodes of the last node in the upper layer is less than n, the current node can be a child node of the last node in the upper layer; if the number of child nodes of the last node in the upper layer is equal to n, the current node cannot be a child node of the last node in the upper layer.

[0162] Step d: If the current node can be a child node of the last node in the upper layer, take the last node in the upper layer as the parent node of the current node, add a write lock to the parent node of the current node, update the parent node of the current node, release the write lock of the parent node of the current node after the update, let the parent node of the current node be the current node, and re-perform step b and the subsequent steps.

[0163] If the current node can be a child node of the last node in the upper layer, the last node in the upper layer can be taken as the parent node of the current node for updating. Here, the update is to add the node information of the current node to the child node list in the last node in the upper layer.

[0164] Step e: If the current node cannot be a child node of the last node in the upper layer, add a new node in the upper layer as the parent node of the current node, let the parent node of the current node be the current node, and re-perform step b and the subsequent steps.

[0165] If the current node cannot be a child node of the last node in the upper layer, a new node can be added after the last node in the upper layer as the parent node of the current node. In this case, the parent node of the current node includes a child node list, and the child node list includes the node information of the current node.

[0166] Step f: if the current node is the original node, then a write lock is added to the parent node of the current node, the parent node of the current node is updated, the write lock of the parent node of the current node is released after the update, the parent node of the current node is set as the current node, and step b and the subsequent steps are re-executed.

[0167] If the current node is the original node, the parent node of the current node already exists in the target tree, and thus the parent node of the current node can be directly updated at this time. The update here is to update the node information of the current node included in the child node list in the parent node of the current node.

[0168] It should be noted that after the current node is re-determined in step d, step e or step f, it can be determined whether the current node is the original root node of the target tree before step b is re-executed.

[0169] If the current node is not the original root node of the target tree and is not a node at the same level as the original root node, then step b and the subsequent steps are re-executed.

[0170] If the current node is the original root node of the target tree, it is determined that the update of the target tree is completed.

[0171] If the current node is not the original root node of the target tree and is a node at the same level as the original root node, a new root node needs to be added at the upper level as the parent node of the original root node and the current node, and the new root node includes a child node list, which includes the node information of the original root node and the node information of the current node. In this way, the update of the target tree is completed. In this case, the height of the target tree increases by one level.

[0172] It should be noted that after the target tree is updated, the pointer to the root node in the target tree and the pointer to the last leaf node in the target tree maintained in the memory can be updated.

[0173] The operation of updating the pointer to the root node in the memory and the operation of obtaining the pointer to the root node in the memory cannot be concurrent. That is, if the pointer to the root node in the memory is being updated, the operation of obtaining the pointer to the root node in the memory cannot be executed until the update is completed; if the pointer to the root node in the memory is being obtained, the operation of updating the pointer to the root node in the memory cannot be executed until the pointer to the root node is successfully obtained. This is because the pointer to the root node can be in a transitional state, such as being empty, during the update of the pointer to the root node, and in this case, an error will occur if the pointer to the root node is obtained.

[0174] In some cases, external systems need to query data stored in the storage tier system. In such cases, the external system can send a data query request to the storage tier system. For example, these data query requests can include two types: the first is for querying the value in a key-value pair, and the second is for querying the file containing the key-value pairs. These two query methods are explained in detail below.

[0175] The first method may include steps A through C.

[0176] Step A: The storage layer system receives the first data query request, which carries the target key value.

[0177] The target key value is the key value in the key-value pair to which the value to be queried belongs.

[0178] The first data query request is used to request the value of the key-value pair to which the target key belongs.

[0179] The first data query request may be sent by an external system. For example, it may be sent by a database system, a blockchain system, etc., but this application embodiment does not limit this.

[0180] Step B: The storage layer system determines the first leaf node in the target tree where the target key value is indexed.

[0181] The target key value is greater than or equal to the minimum key value in the first leaf node, and less than or equal to the maximum key value in the first leaf node.

[0182] For example, the first leaf node to which the target key value is indexed in the target tree can also be referred to as the first leaf node to which the target key value is distributed in the target tree.

[0183] Optionally, the storage layer system can index the first leaf node in the target tree using a binary search based on the target key value. Of course, other methods can also be used to index the first leaf node in the target tree, and this application embodiment does not limit this.

[0184] For example, if the target tree is an n-ary tree, then the complexity of this indexing process satisfies O(logN) × log2branch. Here, the path complexity of the tree index is O(logN), and the complexity of finding a specified child node within each node using binary search is a constant value log2branch, where branch is a fixed number of branches, a constant n.

[0185] In some embodiments, step B can be performed as follows: the storage layer system indexes nodes in the target tree from the root node down layer by layer according to the target key value, until the first leaf node is indexed.

[0186] As an example, in the process of indexing nodes layer by layer downwards, each time a node is indexed, a read lock is added to the node, the next node is indexed according to the node and the target key value, and the read lock of the node is released after the next node is indexed; the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0187] Specifically, after the storage layer system receives the first data query request, the pointer of the root node can be obtained in the memory first, and then the root node is indexed according to the pointer of the root node. A read lock is added to the root node after the root node is indexed.

[0188] Then, it is determined in which key range of the child nodes of the root node the target key value is, and the next node is indexed. After it is determined that the target key value is in the key range of a child node of the root node, the child node is the next node that is indexed, and the read lock of the root node is released and a read lock is added to the latest indexed node.

[0189] If the node is a branch node, it can be determined in which key range of the child nodes of the node the target key value is, i.e., the next node is indexed. The indexed node is a leaf node.

[0190] If the node is a leaf node, it can be determined that the node is the first leaf node, and the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0191] As another example, in the process of indexing nodes layer by layer downwards, each time a node is indexed, a read lock is added to the node in the case that the node is the last child node of all child nodes of the parent node of the node, the next node is indexed according to the node and the target key value, and the read lock of the node is released after the next node is indexed; and in the case that the node is not the last child node of all child nodes of the parent node of the node, no read lock is added to the node, the next node is directly indexed according to the node and the target key value, and in this case, no read lock is added to all nodes that are indexed subsequently, so that the lock resource can be saved.

[0192] In the case that the read lock is added to the first leaf node in the indexing process, the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0193] It should be noted that the read lock and the read lock in the embodiments of the present application are shared locks, and the read lock and the write lock are mutual exclusion locks.

[0194] Optionally, the storage layer system in the embodiments of the present application can be a concurrent model of single-threaded writing and multi-threaded reading.

[0195] In this case, one thread adding a read lock to a node does not affect another thread adding a read lock to the node.

[0196] However, after one thread adds a read lock to a node, another thread cannot add a write lock to the node and has to wait until the read lock of the node is released, and then add a read lock to the node and continue the operation (such as continuing to update the node).

[0197] In an embodiment of the present application, the tree index process and the tree update process will at most only have one lock conflict (also referred to as lock preemption) once. That is, the tree structure of the target tree in the embodiment of the present application can maximize the avoidance of path-level lock conflicts, thereby maximizing the system concurrency and reducing the main business process latency.

[0198] It should be noted that all data query behaviors in the embodiment of the present application can be concurrent, which is very friendly to the scene of concurrent reading of data in the system. For example, if the blockchain system supports concurrent execution of transactions, all behaviors of obtaining ledger data during the concurrent execution of transactions can also be completely concurrent in the system.

[0199] In addition, the node insertion behavior in the embodiment of the present application cannot be concurrent, that is, the next update can be performed only after the completion of the one-time update of the target tree.

[0200] Step C: The storage layer system obtains the value value in the key-value pair to which the target key value belongs from the file identified by the file identifier in the first leaf node.

[0201] Optionally, after obtaining the value value, the storage layer system can return a data query result to the external system, and the data query result carries the value value.

[0202] The second method can include the following step a to the following step c.

[0203] Step a: The storage layer system receives a second data query request, and the second data query request carries a target file identifier.

[0204] The target file identifier is the file identifier of the file to be queried.

[0205] The second data query request is used to request to query the file identified by the target file identifier.

[0206] The second data query request can be sent by an external system. For example, it can be sent by a database system, a blockchain system, etc., and the embodiment of the present application does not limit this.

[0207] Step b: The storage layer system determines the index information in the target tree according to the target file identifier, n, and the height of the target tree, in the case that the target tree is an n-ary tree.

[0208] The index information in the target tree is information needed when indexing a leaf node in the target tree.

[0209] Optionally, assuming that the height of the target tree is m, m is an integer greater than or equal to 2. Then the index information in the target tree can include the index information of each layer in the target tree, that is, it can include the index information of each layer in the 2nd layer to the mth layer.

[0210] The index information of the ith layer is used to indicate the ordering number of the node to be indexed in the ith layer among all the child nodes of its parent node, where i is greater than or equal to 2 and less than or equal to m.

[0211] That is, after indexing a node in the i-1th layer, it is necessary to continue to index the node in the ith layer, in which case, the next node can be indexed from all the child nodes of the node in the i-1th layer according to the index information of the ith layer. For example, the index information of the 2nd layer is 0, then after indexing the node in the 1st layer, i.e., the root node, the child node with an ordering number of 0 among all the child nodes of the root node can be taken as the next node to be indexed, which is the node in the 2nd layer.

[0212] As an example, the storage layer system can obtain the file identifier in the last leaf node in the target tree according to the pointer of the last leaf node in the target tree maintained in the memory, add 1 to the target value corresponding to the file identifier to obtain the number of leaf nodes of the target tree. In the case that the number of leaf nodes of the target tree and the branching number n of the target tree are known, the height m of the target tree can be determined.

[0213] The target value corresponding to a file identifier is the ordering number of the file identified by the file identifier among all files. The ordering number starts from 0 and increases by 1. That is, the ordering number can be 0, 1, 2, 3, ….

[0214] There is a corresponding relationship between the file identifier and the target value. The storage layer system can take the ordering number of a new file among all files as the target value corresponding to the file identifier of the new file every time a new file is added.

[0215] Since the file identifiers are globally ordered, for any file identifier, its corresponding target value can be determined.

[0216] For example, if the file identifiers start from 0 and increase by 1, i.e., the file identifiers are 0, 1, 2, 3, …, then a file identifier is the same as its corresponding target value.

[0217] For example, if the file identifiers are sequentially increased by a fixed value, and the fixed value is j, where j is an integer greater than or equal to 2, the file identifiers are 0, j, 2j, 3j, …, and so on. In this case, the target value corresponding to a file identifier is the value obtained by dividing the file identifier by j.

[0218] Of course, the file identifiers can also be sequentially arranged in other forms. For example, the file identifiers can be a, b, c, …, and so on. In this case, the file identifiers can be converted into corresponding target values according to certain rules. For example, the target value corresponding to the file identifier a is 0, the target value corresponding to the file identifier b is 1, the target value corresponding to the file identifier c is 2, and so on.

[0219] Optionally, the height of the target tree is m, where m is an integer greater than or equal to 2. The operation of step b can be as follows: the storage layer system adds a read lock to the root node of the target tree. The target value corresponding to the target file identifier is determined, where the target value corresponding to the target file identifier is the sequential number of the file identified by the target file identifier in all files. Let i be m, and let the specified parameter be the target value corresponding to the target file identifier. The specified parameter is divided by n to obtain a quotient and a remainder, and the remainder is used as the index information of the i-th layer. If i is not 2, let i be i-1, let the specified parameter be the quotient, and re-execute the above step of dividing the specified parameter by n to obtain a quotient and a remainder, and the subsequent steps, until i is 2. If i is 2, index the root node, and release the read lock of the root node after indexing the root node.

[0220] It should be noted that the embodiments of the present application need to add a read lock to the root node in the process of determining the index information in the target tree. This is because if a new root node appears in the process of determining the index information in the target tree, the index information determined according to the old root node will start from the new root node, which will cause an error. Therefore, the embodiments of the present application add a read lock to the root node before determining the index information in the target tree, so as to avoid the modification of the root node by other threads. After the index information in the target tree is determined, the root node is indexed, and the read lock of the root node is released after indexing the root node. In this way, the root node at the beginning of the indexing process is consistent with the root node when the index information is determined, thereby ensuring the correctness of the indexing process.

[0221] In addition, the embodiments of the present application add a read lock to the root node in the process of determining the index information of the target tree, which does not affect other data indexing processes. That is, different processes of indexing leaf nodes through file identifiers can be completely concurrent.

[0222] It should be noted that dividing the specified parameter by n yields a quotient and a remainder. The quotient represents the sorting index of the parent node of the node to be indexed at level i among all nodes at level i-1. The remainder represents the sorting index of the node to be indexed at level i among all child nodes of its parent node. Therefore, after indexing nodes at level i-1, indexing can continue to nodes at level i.

[0223] In this embodiment, the index information in the target tree is determined starting from the index information at level m, then sequentially determining the index information at level (m-1), level (m-2), and so on, until the index information at level 2 is determined. During indexing, after reaching the root node, the index is sequentially determined from level 2, level 3, and so on, down to the leaf nodes.

[0224] The following is combined Figure 5 The process of determining the index information in the target tree is illustrated with an example.

[0225] like Figure 5 As shown in Figure (b), the target tree is a binary tree with a height of 4.

[0226] Assuming the target file identifier corresponds to a target value of 6, such as Figure 5 As shown in Figure (a), first, the index information of the 4th level is determined. Specifically, let the specified parameter be 6, divide 6 by 2, and get a quotient of 3 and a remainder of 0. Then the index information of the 4th level is determined to be 0.

[0227] Then, we continue to determine the index information of the third level. Specifically, let the specified parameter be 3, divide 3 by 2, and get a quotient of 1 and a remainder of 1. Then we determine that the index information of the third level is 1.

[0228] Next, we continue to determine the index information of the second level. Specifically, we set the specified parameter to 1, divide 1 by 2, and get a quotient of 0 and a remainder of 1. Then we determine that the index information of the second level is 1.

[0229] At this point, the index information for each level from level 2 to level 4 in the target tree has been determined.

[0230] Subsequently, when indexing based on the index information of each level from level 2 to level 4, such as... Figure 5As shown in (b) of FIG. 1, after indexing to the node of the first layer, i.e., the root node, according to the index information of the second layer, it can be determined that the node of the second layer to be indexed is the child node with the sequence number 1 among all child nodes of the root node. After indexing to a node of the second layer, according to the index information of the third layer, it can be determined that the node of the third layer to be indexed is the child node with the sequence number 1 among all child nodes of the node of the second layer. After indexing to a node of the third layer, according to the index information of the fourth layer, it can be determined that the node of the fourth layer to be indexed is the child node with the sequence number 0 among all child nodes of the node of the third layer. In this way, the corresponding leaf node is indexed.

[0231] Step c: the storage layer system indexes the second leaf node in the target tree according to the index information in the target tree, the second leaf node being a leaf node containing the target file identifier.

[0232] It should be noted that, according to the index information in the target tree, the index path from the root node to the second leaf node can be directly determined, and in this process, the node data does not need to be read, so the process of indexing the second leaf node according to the index information in the target tree does not need to add a read lock.

[0233] In the embodiment of the present application, only m-1 times of calculation is needed to determine the index information in the target tree, and the index path can be directly determined according to the index information in the target tree. In this way, the complexity of the indexing process is low, and the indexing efficiency is high.

[0234] In some cases, the disk identifier corresponding to the file identifier can be saved in the leaf node.

[0235] In this case, after indexing to the second leaf node, the storage layer system can obtain the disk identifier corresponding to the target file identifier from the second leaf node, and then can obtain the file identified by the target file identifier from the disk identified by the disk identifier.

[0236] Optionally, after obtaining the file identified by the target file identifier, the storage layer system can return a data query result to the external system, the data query result carrying the file.

[0237] In the embodiment of the present application, a data storage request is received, the data storage request carrying a key-value pair to be stored, the key values in the key-value pair being globally ordered. Then, a new file containing one or more key-value pairs is generated according to the order of the key values in the key-value pair, and the new file is stored; wherein the file identifiers of all files containing key-value pairs are globally ordered. A new leaf node is added after the last leaf node in the target tree, and the target tree is updated according to the new leaf node, the new leaf node containing the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file. In the embodiment of the present application, when data is stored, indexing in the target tree is not required, and a new file can be directly generated for storage, and accordingly a new leaf node can be directly added in the target tree to update the target tree, so that the data storage efficiency can be improved.

[0238] Figure 6 Figure 1 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application. The apparatus can be realized by software, hardware or a combination of both as part or all of a computer device, which can be a computer device as shown in Figure 2. Figure 7 Figure 6 The apparatus comprises a first receiving module 601, a generating module 602 and an updating module 603.

[0239] The first receiving module 601 is configured to receive a data storage request, the data storage request carrying a key-value pair to be stored, the key values in the key-value pair being globally ordered.

[0240] The generating module 602 is configured to generate a new file containing one or more key-value pairs according to the order of the key values in the key-value pair, and store the new file; wherein the file identifiers of all files containing key-value pairs are globally ordered.

[0241] The updating module 603 is configured to add a new leaf node after the last leaf node in the target tree, and update the target tree according to the new leaf node, the new leaf node containing the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file.

[0242] Optionally, the target tree is an n-ary tree, n being an integer greater than or equal to 2.

[0243] Optionally, the branch node in the target tree comprises a child node list, the child node list comprising the minimum key value and the maximum key value in each child node of the branch node.

[0244] Optionally, the target tree is located in the memory, and the file is stored in the disk, and the apparatus further comprises:

[0245] ​A building module is configured to rebuild the target tree in the memory according to all the files stored in the disk after a restart.

[0246] Optionally, the key-value pair comprises a target key-value pair, the key value in the target key-value pair comprises a block number and a key value in a block identified by the block number, and the value value in the target key-value pair is a value value in the block.

[0247] Optionally, the generating module 602 is configured to:

[0248] Upon receiving each data storage request, save all the key-value pairs carried by the data storage request in the memory in the order of the key values in the key-value pairs;

[0249] If the difference between the total data amount of one or more key-value pairs saved in sequence in the memory and the preset data amount is less than or equal to the preset difference, generate a new file containing the one or more key-value pairs;

[0250] Store the new file in the disk.

[0251] Optionally, the updating module 603 is configured to:

[0252] Update the nodes in the target tree layer by layer upwards according to the new leaf node until the root node is updated;

[0253] In the process of updating the nodes layer by layer upwards, if an original node in the target tree needs to be updated, the original node is added with a write lock, and the write lock is released after the update is completed.

[0254] Optionally, the apparatus further comprises:

[0255] The second receiving module is configured to receive a first data query request, and the first data query request carries a target key value;

[0256] The first determining module is configured to determine a first leaf node indexed by the target key value in the target tree;

[0257] The obtaining module is configured to obtain a value value in a key-value pair to which the target key value belongs from a file identified by the file identifier in the file identified by the first leaf node.

[0258] Optionally, the first determining module is configured to:

[0259] Index the nodes in the target tree layer by layer downwards from the root node according to the target key value until the first leaf node is indexed;

[0260] In the process of indexing the nodes layer by layer downwards, a read lock is added to each indexed node, the next node is indexed according to the indexed node and the target key value, and the read lock of the indexed node is released after the next node is indexed; the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0261] Optionally, the target tree is an n-ary tree, n is an integer greater than or equal to 2, and the apparatus further comprises:

[0262] The third receiving module is configured to receive a second data query request, the second data query request carrying a target file identifier.

[0263] The second determining module is configured to determine index information in the target tree according to the target file identifier, n, and a tree height of the target tree.

[0264] The indexing module is configured to index a second leaf node in the target tree according to the index information in the target tree, the second leaf node being a leaf node containing the target file identifier.

[0265] Optionally, the second determining module is configured to:

[0266] A read lock is added to a root node of the target tree, the tree height of the target tree being m, m being an integer greater than or equal to 2.

[0267] A target value corresponding to the target file identifier is determined, the target value corresponding to the target file identifier being an ordering sequence number of a file identified by the target file identifier in all files.

[0268] Let i be m, and let the specified parameter be the target value corresponding to the target file identifier.

[0269] The specified parameter is divided by n to obtain a quotient and a remainder.

[0270] The remainder is taken as index information of the ith layer, the index information of the ith layer being used to indicate an ordering sequence number of a node of the ith layer to be indexed in all child nodes of a parent node.

[0271] If i is not 2, let i be i-1, let the specified parameter be the quotient, and re-execute the step of dividing the specified parameter by n to obtain a quotient and a remainder and the subsequent steps until i is 2.

[0272] If i is 2, the root node is indexed, and the read lock of the root node is released after the root node is indexed.

[0273] In this embodiment, a data storage request is received, carrying key-value pairs to be stored, where the keys in the key-value pairs are globally ordered. Then, based on the order of the keys in the key-value pairs, a new file containing one or more key-value pairs is generated and stored; wherein the file identifiers of all files containing key-value pairs are globally ordered. Then, a new leaf node is added after the last leaf node in the target tree, and the target tree is updated based on this new leaf node, which contains the file identifier of the new file, the minimum key value, and the maximum key value among all key-value pairs in the new file. In this embodiment, when storing data, there is no need to index the target tree; new files can be directly generated for storage, and new leaf nodes can be directly added to the target tree to update it, thus improving data storage efficiency.

[0274] It should be noted that the data processing device provided in the above embodiments is only illustrated by the division of the above functional modules when processing data. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0275] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0276] The data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept. The specific working process and technical effects of the units and modules in the above embodiments can be found in the method embodiment section, and will not be repeated here.

[0277] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 7 As shown, the computer device 7 includes a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, it implements the steps in the data processing method in the above embodiments.

[0278] The computer device 7 can be a general purpose computer device or a special purpose computer device. In specific implementations, the computer device 7 can be the storage layer system described above, and can be a desktop computer, a laptop computer, a network server, a palmtop computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device, and the embodiments of the present application do not limit the type of the computer device 7. Those skilled in the art can understand that Figure 7 The computer device 7 is merely an example and does not constitute a limitation on the computer device 7, and can include more or fewer components than shown, or combine certain components, or different components, such as an input / output device, a network access device, and the like.

[0279] The processor 70 can be a central processing unit (CPU), and can also be other general purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, and the like. The general purpose processor can be a microprocessor or can be any conventional processor.

[0280] The memory 71 can be an internal storage unit of the computer device 7, such as a hard disk or a memory of the computer device 7 in some embodiments. The memory 71 can also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like, in other embodiments. Further, the memory 71 can include both the internal storage unit and the external storage device of the computer device 7. The memory 71 is used to store an operating system, application programs, a boot loader, data, and other programs, and the like. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0281] The embodiments of the present application also provide a computer device, which includes at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor implements the steps in any of the method embodiments described above when executing the computer program.

[0282] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned various method embodiments.

[0283] The embodiment of the present application provides a computer program product, which, when running on a computer, causes the computer to execute the steps in the above-mentioned various method embodiments.

[0284] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above-mentioned method embodiments by a computer program to instruct related hardware to complete, which can be stored in a computer readable storage medium. The computer program, when executed by a processor, can implement the steps in the above-mentioned various method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device, etc. The computer readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0285] It should be understood that all or part of the steps in the above-mentioned embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product entirely or partially. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-mentioned computer readable storage medium.

[0286] In the above-mentioned embodiments, the description of each embodiment has its own focus. The parts not described in detail or recorded in a certain embodiment can be referred to the relevant description of other embodiments.

[0287] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0288] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely schematic. The division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0289] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0290] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A data processing method, characterized by, The method comprises: receiving a data storage request, the data storage request carrying a key-value pair to be stored, the key values in the key-value pair being globally ordered; generating a new file containing one or more key-value pairs according to the order of the key values in the key-value pair, and storing the new file; wherein the file identifiers of all files containing the key-value pair are globally ordered; adding a new leaf node after the last leaf node in the target tree, updating the target tree according to the new leaf node, the new leaf node containing the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file; wherein the target tree is an n-ary tree, n is an integer greater than or equal to 2, and the method further comprises: receiving a second data query request, the second data query request carrying a target file identifier; locking the root node of the target tree, the height of the target tree being m, m being an integer greater than or equal to 2; determining a target value corresponding to the target file identifier, the target value corresponding to the target file identifier being the ordering number of the file identified by the target file identifier among all files; let i be m, and let the specified parameter be the target value corresponding to the target file identifier; divide the specified parameter by n to obtain quotient and remainder; use the remainder as the index information of the i-th layer, the index information of the i-th layer being used to indicate the ordering number of the i-th layer node to be indexed among all child nodes of the parent node; if i is not 2, let i = i-1, and let the specified parameter be the quotient, re-execute the step of dividing the specified parameter by n to obtain quotient and remainder, and subsequent steps until i is 2; if i is 2, index the root node, and release the read lock of the root node after indexing the root node; indexing a second leaf node in the target tree according to the index information, the second leaf node being a leaf node containing the target file identifier.

2. The method of claim 1, wherein, The branch node in the target tree includes a child node list, and the child node list includes the minimum key value and the maximum key value in each child node of the branch node.

3. The method of claim 1, wherein, The target tree is located in the memory, and the files are stored in the disk, and the method further comprises: After restarting, reconstruct the target tree in the memory according to all files stored in the disk.

4. The method of claim 1, wherein, The key-value pair includes a target key-value pair, the key value in the target key-value pair includes a block number and a key value in a block identified by the block number, and the value value in the target key-value pair is a value value in the block.

5. The method according to any one of claims 1 to 4, characterized in that, The generating a new file containing one or more key-value pairs according to the order of the key values in the key-value pair, and storing the new file, comprises: Upon receiving each data storage request, save all key-value pairs carried by the data storage request in the memory according to the order of the key values in all key-value pairs carried by the data storage request; If a difference between the total data amount of the one or more key-value pairs stored in the memory in sequence and the preset data amount is less than or equal to a preset difference, a new file containing the one or more key-value pairs is generated; The new file is stored in the disk.

6. The method of any one of claims 1 to 4, wherein, The updating of the target tree according to the new leaf node comprises: The nodes are updated layer by layer upwards in the target tree according to the new leaf node until the root node is updated; wherein, in the process of updating the nodes layer by layer upwards, if the original node in the target tree needs to be updated, the original node is added with a write lock, and the write lock is released after the update is completed.

7. The method of any one of claims 1 to 4, wherein, The method further comprises: receiving a first data query request, the first data query request carrying a target key value; determining a first leaf node to which the target key value is indexed in the target tree; obtaining a value value in a key-value pair to which the target key value belongs from a file identified by a file identifier in the first leaf node.

8. The method of claim 7, wherein, The determining of the first leaf node to which the target key value is indexed in the target tree comprises: The nodes are indexed layer by layer downwards in the target tree according to the target key value from the root node until the first leaf node is indexed; wherein, in the process of indexing the nodes layer by layer downwards, each time a node is indexed, the node is added with a read lock, the next node is indexed according to the node and the target key value, and the read lock of the node is released after the next node is indexed; the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

9. A computer device, comprising: The computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is executed by the processor to implement the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method according to any one of claims 1 to 8.

Images