Data processing method, computer device, and storage medium

Abstract

The present application relates to the technical field of computers, and discloses a data processing method, a computer device, and a storage medium. The method comprises: receiving a data storage request, wherein the data storage request carries key-value pairs to be stored, and key values in the key-value pairs are globally ordered; on the basis of an order of the key values in the key-value pairs, generating a new file comprising one or more key-value pairs, and storing the new file, wherein file identifiers of all files comprising the key-value pairs are globally ordered; adding a new leaf node after the last leaf node in a target tree, and updating the target tree on the basis of the new leaf node, wherein the new leaf node comprises a file identifier of the new file, and the minimum key value and the maximum key value in all key-value pairs in the new file. In the present application, during data storage, there is no need to perform indexing in a target tree, a new file can be directly generated for storage, and accordingly, a new leaf node can be directly added to update the target tree. In this way, the data storage efficiency can be improved.

Description

Data processing method, computer device and storage medium

[0001] The present application claims priority from the Chinese patent application No. 202410853478.2 filed on June 27, 2024 in the China Patent Office and entitled "Data processing method, computer device and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of computer, in particular to a data processing method, a computer device and a storage medium. BACKGROUND

[0003] At present, data storage is often based on tree structure. Specifically, after receiving a key-value pair to be stored, the key value in the key-value pair is used to index from the root node to a leaf node in the tree. Then the key-value pair is added to the data of the leaf node to update the leaf node, and the updated leaf node is used to update the nodes layer by layer upwards until the root node is updated to complete the update of the tree. Then, the data of each node in the updated tree is stored to the disk to complete the persistence of the tree. However, the data storage efficiency of this method is low. TECHNICAL PROBLEM

[0004] The present application provides a data processing method, a computer device and a storage medium, which can improve the data storage efficiency. TECHNICAL SOLUTION

[0005] In a first aspect, a data processing method is provided, comprising:

[0006] receiving a data storage request carrying a key-value pair to be stored, the key values in the key-value pair being globally ordered;

[0007] generating a new file containing one or more key-value pairs according to the order of the key values in the key-value pair, and storing the new file; wherein the file identifiers of all files containing the key-value pair are globally ordered;

[0008] adding a new leaf node after the last leaf node in the target tree, updating the target tree according to the new leaf node, the new leaf node containing the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file.

[0009] Optionally, the target tree is an n-ary tree, and n is an integer greater than or equal to 2.

[0010] Optionally, the branch node in the target tree comprises a child node list, and the child node list comprises a minimum key value and a maximum key value in each child node of the branch node.

[0011] Optionally, the target tree is located in a memory, and the files are stored in a disk, and the method further comprises:

[0012] After a restart, the target tree is reconstructed in the memory according to all the files stored in the disk.

[0013] Optionally, the key-value pair comprises a target key-value pair, the key value in the target key-value pair comprises a block number and a key value in a block identified by the block number, and the value value in the target key-value pair is a value value in the block.

[0014] Optionally, the generating a new file containing one or more key-value pairs according to the order of the key values in the key-value pairs and storing the new file comprises:

[0015] According to the order of the key values in all the key-value pairs carried by the data storage request, saving all the key-value pairs carried by the data storage request in the memory;

[0016] If the difference between the total data amount of one or more key-value pairs saved in order in the memory and the preset data amount is less than or equal to the preset difference, a new file containing the one or more key-value pairs is generated;

[0017] The new file is stored in the disk.

[0018] Optionally, the updating the target tree according to the new leaf node comprises:

[0019] According to the new leaf node, a node is updated layer by layer upwards in the target tree until the root node is updated; wherein, in the process of updating the node layer by layer upwards, if an original node in the target tree needs to be updated, the original node is added with a write lock, and the write lock is released after the update is completed.

[0020] Optionally, the method further comprises:

[0021] A first data query request is received, and the first data query request carries a target key value;

[0022] A first leaf node to which the target key value is indexed in the target tree is determined;

[0023] A value value in a key-value pair to which the target key value belongs is obtained from a file identified by a file identifier in the first leaf node.

[0024] Optionally, the determining the first leaf node to which the target key value is indexed in the target tree comprises:

[0025] According to the target key value, a node is indexed layer by layer from a root node downwards in the target tree until the first leaf node is indexed.

[0026] In the process of indexing the node layer by layer downwards, a read lock is added to one node each time a node is indexed, a next node is indexed according to the one node and the target key value, and the read lock of the one node is released after the next node is indexed; and the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0027] Optionally, the target tree is an n-ary tree, the n is an integer greater than or equal to 2, and the method further comprises:

[0028] receiving a second data query request, the second data query request carrying a target file identifier;

[0029] According to the target file identifier, the n and the height of the target tree, determining index information in the target tree;

[0030] According to the index information, indexing a second leaf node in the target tree, the second leaf node being a leaf node containing the target file identifier.

[0031] Optionally, the determining the index information in the target tree according to the target file identifier, the n and the height of the target tree comprises:

[0032] adding a read lock to a root node of the target tree, the height of the target tree being m, the m being an integer greater than or equal to 2;

[0033] determining a target value corresponding to the target file identifier, the target value corresponding to the target file identifier being an ordering sequence number of a file identified by the target file identifier in all files;

[0034] letting i be the m, and letting a specified parameter be the target value corresponding to the target file identifier;

[0035] dividing the specified parameter by the n to obtain a quotient and a remainder;

[0036] taking the remainder as index information of the i-th layer, the index information of the i-th layer being used to indicate an ordering sequence number of a node of the i-th layer to be indexed in all child nodes of a parent node;

[0037] If i is not 2, let i = i-1, let the specified parameter be the quotient, re-execute the step of dividing the specified parameter by the n to obtain the quotient and the remainder and the subsequent steps until i is 2;

[0038] If i is 2, index the root node, and release the read lock of the root node after indexing the root node.

[0039] In a second aspect, the present application provides a data processing device, the device comprising:

[0040] A first receiving module, configured to receive a data storage request, the data storage request carrying a key-value pair to be stored, the key values in the key-value pair being globally ordered;

[0041] A generating module, configured to generate a new file containing one or more key-value pairs according to the order of the key values in the key-value pair, and store the new file; wherein the file identifiers of all files containing the key-value pair are globally ordered;

[0042] An updating module, configured to add a new leaf node after the last leaf node in a target tree, update the target tree according to the new leaf node, the new leaf node containing the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file.

[0043] In a third aspect, a computer device is provided, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being executed by the processor to implement the data processing method of the first aspect.

[0044] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the data processing method of the first aspect.

[0045] In a fifth aspect, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the steps of the data processing method of the first aspect. Advantageous effects

[0046] In the present application, when data is stored, indexing in a target tree is not required, and a new file can be directly generated for storage, and accordingly a new leaf node can be directly added in the target tree to update the target tree, so that the data storage efficiency can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced. Obviously, the drawings in the following description only constitute some embodiments of the present application. For those skilled in the art, other drawings can also be obtained from these drawings without creative effort.

[0048] Fig. 1 is a schematic diagram of a target tree provided by an embodiment of the present application;

[0049] Fig. 2 is a schematic diagram of a key-value pair provided by an embodiment of the present application;

[0050] Fig. 3 is a schematic diagram of another key-value pair provided by an embodiment of the present application;

[0051] Fig. 4 is a flowchart of a data processing method provided by an embodiment of the present application;

[0052] Fig. 5 is a schematic diagram of index information provided by an embodiment of the present application;

[0053] Fig. 6 is a schematic diagram of the structure of a data processing apparatus provided by an embodiment of the present application;

[0054] Fig. 7 is a schematic diagram of the structure of a computer device provided by an embodiment of the present application. Embodiments of the present application

[0055] In order to make the objects, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0056] It should be understood that the "multiple" mentioned in the present application refers to two or more. In the description of the present application, unless otherwise specified, " / " represents the meaning of or, for example, A / B can represent A or B; "and / or" in the present application only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, in order to clearly describe the technical solutions of the present application, the same items or similar items with basically the same functions and effects are distinguished by using "first", "second", etc. The skilled in the art can understand that "first", "second", etc. do not limit the quantity and execution order, and "first", "second", etc. also do not necessarily mean different.

[0057] The phrase "one embodiment" or "some embodiments" appearing in the present application are used as antecedents for the phrase "including a feature, structure, or characteristic" where no other feature, structure, or characteristic to which the phrase "one embodiment" or "some embodiments" refers is present. Use of the phrase "in one embodiment" or "in some embodiments" is not a limitation on the scope or on the meaning of the phrase "including a feature, structure, or characteristic" such that the feature, structure, or characteristic in question is present in one embodiment but is not present in all embodiments, unless specifically stated otherwise.

[0058] The application scenarios of the embodiments of the present application are described below.

[0059] With the continuous development of computer technology, data has become an important resource and asset. The importance of data is reflected in supporting decision-making, promoting business development, promoting innovation, improving customer experience, improving efficiency and effectiveness, managing risks, and promoting governance. With the continuous generation of data, how to efficiently store data is a problem that needs to be solved.

[0060] In the related art, data is usually stored based on a tree structure (such as a B+ tree). Specifically, after receiving a key-value pair to be stored, the key-value pair is first indexed to a leaf node in the tree according to the key value in the key-value pair, and then the key-value pair is added to the leaf node and the tree is updated. Finally, each node in the updated tree is stored to the disk. However, this way of storing data is inefficient.

[0061] Therefore, the embodiments of the present application propose a data processing method. When storing data, no indexing in the target tree is required, and a new file can be directly generated for storage. Accordingly, a new leaf node can be directly added to update the target tree, thereby improving the data storage efficiency.

[0062] The data processing method provided by the embodiments of the present application can be applied in a system that has a data processing requirement. In the embodiments of the present application, the system is referred to as a storage layer system. The storage layer system is described below.

[0063] The embodiments of the present application are applied to a scenario in which an external system stores data based on the storage layer system. For example, the embodiments of the present application can be applied to a scenario in which data is stored based on a target tree.

[0064] For example, after a blockchain system executes all transactions in a block, a series of ledger data is generated. The ledger data can include a plurality of key-value pairs. In this case, the data processing method provided by the embodiments of the present application can be used to store the ledger data based on the target tree.

[0065] The external system to which the storage layer system interfaces is a system that has a data storage requirement. For example, the external system to which the storage layer system interfaces can be a database system, a blockchain system, and the like, and the embodiments of the present application do not limit this.

[0066] The storage layer system is configured to receive, process, and reply to various requests sent by the external system. For example, the request sent by the external system can include a data storage request, a data query request, and the like, and the embodiments of the present application do not limit this.

[0067] In some embodiments, the storage layer system can interface with a database system. The database system can interface with other systems that have a data storage requirement, such as a blockchain system and the like. For example, when the blockchain system needs to store data, the data can be first sent to the database system, and then sent to the storage layer system by the database system, and the storage layer system can store the data based on the target tree.

[0068] Optionally, the storage layer system and the database system can be independent systems, in which case the storage layer system and the database system can run on independent physical machines. Alternatively, the database system can include the storage layer system, in which case the database system includes a specific system that interfaces with the blockchain system and runs on the same physical machine as the storage layer system.

[0069] Next, the structure of the target tree provided by the embodiments of the present application is described.

[0070] The target tree is a tree structure. The target tree includes branch nodes and leaf nodes at the lowest level of the target tree, wherein a node with a child node number of 0 is a leaf node, a node with a child node number greater than or equal to 1 is a branch node, and the branch nodes and the leaf nodes are all logical nodes. The uppermost branch node in the target tree can be referred to as a root node, and the root node is a branch node without a parent node in the target tree.

[0071] Optionally, the target tree can be an n-ary tree, that is, the number of child nodes of each branch node in the target tree is at most n, and n is an integer greater than or equal to 2. Any branch node in the n-ary tree can have at most n child nodes. Moreover, for branch nodes at the same level in the n-ary tree, a new branch node can be added after the last branch node at the same level only when the number of child nodes of the last branch node at the same level reaches n.

[0072] The leaf node stores a file identifier, a maximum key value, and a minimum key value.

[0073] The file identifier is used to identify a file stored in the storage layer system. For example, the file identifier can be a file identity (ID), a file name, and the like, and the embodiments of the present application do not limit this.

[0074] The minimum key value in the leaf node is the minimum key value in the file identified by the file identifier in the leaf node. The maximum key value in the leaf node is the maximum key value in the file identified by the file identifier in the leaf node.

[0075] In the embodiment of the present application, the leaf nodes in the target tree correspond to files one by one. That is, one leaf node corresponds to the file identifier of a file and the minimum key value and the maximum key value in the file.

[0076] In the embodiment of the present application, the storage layer system can store one or more files, and each file can contain one or more key-value pairs. The file identifiers of all the files stored in the storage layer system are globally ordered, that is, the file identifier of any file is ordered in all the files stored in the storage layer system. In addition, the key values in all the key-value pairs contained in the files are also globally ordered, that is, the key value in any key-value pair contained in any file is ordered in all the key-value pairs in all the files stored in the storage layer system.

[0077] Optionally, the global order of the key values in the key-value pairs can be in the order of increasing dictionary order.

[0078] In this case, the minimum key value in a file refers to the key value with the earliest order among the key values in all the key-value pairs in the file. The maximum key value in a file refers to the key value with the latest order among the key values in all the key-value pairs in the file.

[0079] It should be noted that in the computer field, dictionary order can be used to indicate the size relationship between any two strings. The size relationship between two strings depends on the size relationship of the American Standard Code for Information Interchange (ASCII) codes of the characters from left to right in the string.

[0080] Optionally, the global order of the file identifiers of the files can be in the order of increasing dictionary order.

[0081] For example, the file identifier of a file can start from 0 and increase by a preset value.

[0082] The preset value can be set in advance. For example, the preset value can be set to 1, 2, 3, etc., which is not limited in the embodiment of the present application.

[0083] In this case, the file identifiers of all the files are an arithmetic sequence. For example, assuming that the preset value is 1, the file identifiers of the first file, the second file, the third file, …, the kth file generated can be 0, 1, 2, …, k-1, respectively.

[0084] In some embodiments, a global continuously increasing counter can be maintained inside the storage layer system, and the counter can be used to generate the file identifier. In this case, each time the storage layer system generates a new file, the current value of the counter can be used as the file identifier of the new file, so that the files stored in the storage layer system are a batch of files with continuously increasing file identifiers.

[0085] The branch node includes a child node list, and the child node list includes the minimum key value and the maximum key value in each child node of the branch node. For example, the range from the minimum key value in a child node to the maximum key value in the child node can be referred to as the key range of the child node.

[0086] The child nodes of a branch node can be branch nodes or leaf nodes of the next layer. The minimum key value in the branch node is the minimum key value in all the key values in the child node list of the branch node, that is, the key value with the earliest order in all the key values in the child node list; and the maximum key value in the branch node is the maximum key value in all the key values in the child node list of the branch node, that is, the key value with the latest order in all the key values in the child node list. The minimum key value and the maximum key value in a leaf node are the minimum key value and the maximum key value stored in the leaf node.

[0087] For example, FIG. 1 is a schematic diagram of a target tree provided by an embodiment of the present application. Referring to FIG. 1, the target tree is a 2-ary tree, and the target tree includes multiple leaf nodes (for example, nodes n4, n5, and n6 in FIG. 1) and multiple branch nodes (for example, nodes n1, n2, and n3 in FIG. 1). The child nodes of the node n1 include the nodes n2 and n3, the child nodes of the node n2 include the nodes n4 and n5, and the child nodes of the node n3 include the node n6. In the embodiment of the present application, the branch node at the uppermost layer in the target tree can be referred to as a root node, for example, the node n1 in FIG. 1 is a root node.

[0088] The leaf node n4 in FIG. 1 stores the file identifier 0, the minimum key value 1, and the maximum key value 3. The leaf node n5 stores the file identifier 1, the minimum key value 4, and the maximum key value 6. The leaf node n6 stores the file identifier 2, the minimum key value 7, and the maximum key value 9.

[0089] The branch node n2 in FIG. 1 stores a child node list including the minimum key value 1 and the maximum key value 3 in the leaf node n4, and including the minimum key value 4 and the maximum key value 6 in the leaf node n5.

[0090] The branch node n3 in FIG. 1 stores a child node list including the minimum key value 7 and the maximum key value 9 in the leaf node n6.

[0091] The branch node n1 in FIG. 1 stores a child node list including the minimum key value 1 and the maximum key value 6 in the branch node n2, and including the minimum key value 7 and the maximum key value 9 in the branch node n3.

[0092] In an embodiment of the present application, the storage layer system can store the key-value pairs sent by the external system based on the target tree. The target tree can also be referred to as a solid branch tree.

[0093] In some cases, the key values in the key-value pairs sent by the external system to the storage layer system are globally ordered, which has been explained above and will not be repeated here.

[0094] In some embodiments, the storage layer system can be connected to a database system, and the database system can be connected to a blockchain system. In this case, the blockchain system can send the key-value pairs to be stored to the database system, and the database system can send a data storage request carrying the key-value pairs to the storage layer system after receiving the key-value pairs.

[0095] As an example, the key-value pairs to be stored sent by the blockchain system to the database system can be block data. Since the block data is strictly organized according to the block number, i.e., the key values in the block data are block numbers, the key values in the block data are naturally globally ordered. In this case, the database system can directly send the block data to the storage layer system for storage in the data storage request.

[0096] As another example, the key-value pairs to be stored sent by the blockchain system to the database system can be account data in a block. The account data in a block includes multiple key-value pairs, and the key values in the multiple key-value pairs in the same block are different, but the key values in the key-value pairs in different blocks can be the same.

[0097] In this case, the database system can first process the account book data, specifically, the database system can introduce a block number in the account book data to construct key-value pairs with globally ordered keys, and then send the constructed key-value pairs to the storage layer system in a data storage request for storage. In this way, it can be ensured that the keys in the key-value pairs carried in the data storage request sent by the database system to the storage layer system are globally ordered.

[0098] Optionally, the key-value pair can be referred to as a target key-value pair in the embodiments of the present application, the key in the target key-value pair includes a block number and a key in a block identified by the block number (i.e. the key in the key-value pair in the account book data), and the value in the target key-value pair is the value in the block (i.e. the value in the key-value pair in the account book data).

[0099] In some embodiments, the key in the target key-value pair can be obtained by concatenating the block number and the key in the block, of course, the key in the target key-value pair can also be obtained by other ways, which are not limited in the embodiments of the present application.

[0100] Since the block number is naturally globally ordered, and the keys in a block are different, the key in the target key-value pair containing the block number and the key in the block identified by the block number is also globally ordered.

[0101] For example, the block with the block number 00000001 includes a plurality of key-value pairs, for each key-value pair in the plurality of key-value pairs, the block number 00000001 can be added before the key in the key-value pair to obtain a new key.

[0102] For example, as shown in (a) of FIG. 2, the block 1 with the block number 00000001 includes key-value pairs bbbb:{value1...}, bggg:{value2...}, caaa:{value3...}, xaab:{value4...}. As shown in (b) of FIG. 2, the block 2 with the block number 00000002 includes key-value pairs aabb:{value5...}, bggg:{value6...}, bcde:{value7...}.

[0103] Then, after the blockchain system sends the key-value pairs to the database system, the database system can introduce the block number 00000001 into the key-value pairs in block 1, specifically, concatenate 00000001 before the key value in each key-value pair in block 1 to obtain the target key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. The block number 00000002 is introduced into the key-value pairs in block 2, specifically, 00000002 is concatenated before the key value in each key-value pair in block 2 to obtain the target key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, and 00000002bcde:{value7...}. The key values in the target key-value pairs obtained in this way are globally ordered.

[0104] It should be noted that the storage layer system in the embodiments of the present application is not limited to storing the key-value pairs in the blockchain system described above, but can also store other key-value pairs, as long as the key values in the key-value pairs are globally ordered.

[0105] The process of constructing an initial target tree by the storage layer system will be described below, which can include the following steps (1)-(3):

[0106] (1) The storage layer system receives a data storage request, which carries a key-value pair to be stored, and the key value in the key-value pair is globally ordered.

[0107] The data storage request is used to instruct the storage layer system to store the key-value pair to be stored. In some embodiments, the data storage request can be sent by an external system. For example, the blockchain system or the database system.

[0108] In this case, the key values in all key-value pairs to be stored sent by the external system to the storage layer system are ordered. After the storage layer system stores these key-value pairs, the key values in all key-value pairs stored in the storage layer system are also ordered.

[0109] (2) The storage layer system generates a new file containing one or more key-value pairs according to the order of the key values in the key-value pairs, and stores the new file; wherein the file identifiers of all files containing key-value pairs are globally ordered.

[0110] The key values in the key-value pairs received by the storage layer system are in order. In this case, the storage layer system generates a new file according to the order of the key values in the key-value pairs, and the key values in the key-value pairs in the new file are also in order.

[0111] Optionally, when storing the new file, the storage layer system can store the new file in a disk.

[0112] For example, the storage layer system can include one or more disks, and the storage layer system can store all the files in the same disk or store all the files in different disks, and the embodiments of the present application do not limit this.

[0113] Optionally, if the storage layer system stores all the files in different disks, the storage layer system can also obtain the disk identifier of the disk where the new file is stored after storing the new file.

[0114] In some embodiments, the operation of step (2) can be that the storage layer system saves all the key-value pairs carried by the data storage request in the memory according to the order of the key values in the key-value pairs carried by the data storage request each time the storage layer system receives a data storage request. If the difference between the total data amount of one or more key-value pairs saved in order in the memory and the preset data amount is less than or equal to the preset difference value, the storage layer system generates a new file containing the one or more key-value pairs. The storage layer system stores the new file in a disk.

[0115] The preset data amount can be set in advance. For example, the preset data amount can be set to 8 megabytes (MB), 9 MB, 10 MB, etc., and the embodiments of the present application do not limit this.

[0116] Optionally, the preset data amount can be set according to an integer multiple of the block size of the storage layer system. For example, if the block size is 4 MB, the preset data amount can be set to 4 MB, 8 MB, 12 MB, etc. The block size of the storage layer system refers to the smallest unit used by the storage layer system to organize and manage data storage.

[0117] The preset difference value can be set in advance. For example, the preset difference value can be set to 100 kilobytes (KB), 200 KB, 300 KB, etc., and the embodiments of the present application do not limit this.

[0118] It should be noted that the data amount of each key-value pair is generally close, and the data amount of a key-value pair is usually within a predetermined range. In the embodiments of the present application, the preset data amount can be much larger than the data amount of a key-value pair. The preset difference value can be close to the data amount of a key-value pair, such as equal to or slightly larger than the data amount of a key-value pair.

[0119] The storage layer system can save all the key-value pairs carried by each received data storage request in order in the memory. In this case, if the total data amount of one or more key-value pairs currently saved in order in the memory is close to the preset data amount (i.e., the difference between the total data amount and the preset data amount is less than or equal to the preset difference), the one or more key-value pairs can be stored in the form of a new file in the disk. In this way, batch storage of key-value pairs can be implemented, thereby improving the data storage efficiency. Moreover, in this case, the file size of each file stored in the disk is basically consistent, and is close to the preset data amount, thereby facilitating file management.

[0120] It should be noted that if the total data amount of all the key-value pairs currently saved in the memory is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference, it indicates that the total data amount of the key-value pairs currently saved in the memory is greatly different from the preset data amount, and the condition for generating a new file is not met. In this case, the storage layer system can continue to receive data storage requests sent by the external system, and save the key-value pairs carried by the data storage requests in the memory, until the difference between the total data amount of the key-value pairs saved in order in the memory and the preset data amount is less than or equal to the preset difference, and then a new file is generated and stored.

[0121] The generated new file is stored in the disk, i.e., the persistence of the new file is implemented. Moreover, after the new file is stored in the disk, one or more key-value pairs in the new file are deleted from the memory.

[0122] It should be noted that the key values in the key-value pairs saved in the memory are in order. In this case, the new file is generated according to the key-value pairs saved in the memory, and the key values in all the key-value pairs in the new file are also in order.

[0123] It should be noted that since one file can contain one or more key-value pairs, the key-value pairs in one file can cover the data of one block or multiple blocks.

[0124] For example, as shown in Figure 3, the storage layer system receives data storage request 1, which carries key-value pairs: 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. The storage layer system saves the key-value pairs carried in data storage request 1 into memory in sequence. Therefore, the key-value pairs currently saved in memory include 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. Since the total data volume of these key-value pairs is less than the preset data volume, and the difference between them is greater than the preset difference, the conditions for generating a new file are not met.

[0125] Subsequently, the storage layer system receives data storage request 2, which carries key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, and 00000002bcde:{value7...}. The storage layer system stores the key-value pairs carried in data storage request 2 in memory in sequence. After storing key-value pair 00000002bggg:{value6...} in memory, the storage layer system finds that the difference between the total amount of data in memory containing key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} and the preset amount of data is less than or equal to the preset difference, satisfying the condition for generating a new file. Therefore, a new file containing key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} is generated and stored on disk. Furthermore, the storage layer system stores the key-value pair 00000002bcde:{value7...} carried in data storage request 2 in memory.

[0126] Subsequently, the storage layer system receives data storage request 3, which carries the key-value pair 00000003bcbb:{value1...}. The storage layer system saves the key-value pairs carried in data storage request 3 into memory in sequence. Therefore, the key-value pairs currently saved in memory include 00000002bcde:{value7...} and 00000003bcbb:{value1...}. Since the total data volume of these key-value pairs is less than the preset data volume, and the difference between them is greater than the preset difference, the conditions for generating a new file are not met. The storage layer system continues to wait for the next data storage request.

[0127] (3) The storage layer system generates a new leaf node and constructs an initial target tree based on the new leaf node. The new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value among all key-value pairs in the new file.

[0128] Optionally, if the storage layer system distributes all files across different disks, the new leaf node may also contain the disk identifier of the disk where the new file is located.

[0129] After generating the new leaf node, the storage layer system can create a new branch node as the parent node of the new leaf node. This branch node stores a list of child nodes, which includes the minimum and maximum key values ​​of the new leaf node. Since the current number of leaf nodes is 1, this branch node is the root node, thus completing the initial construction of the target tree.

[0130] In some embodiments, a branch node in the target tree may include a list of child nodes. The list of child nodes in a branch node stores node information for each of the branch node's child nodes.

[0131] For example, node information can include a pointer to the node, the minimum key value, and the maximum key value in the node.

[0132] A pointer to a node is used to point to the memory space where that node resides.

[0133] In this case, the list of child nodes in a branch node includes the node information of all child nodes of this branch node, and the node information of all child nodes exists in the child node list in order.

[0134] In some cases, after the storage layer system completes the initial construction of the target tree, it can also store pointers to the root node and the last leaf node of the target tree in memory. This allows the storage layer system to quickly index to the root node of the target tree using the pointer to the root node maintained in memory, and to quickly index to the last leaf node of the target tree using the pointers to the leaf nodes maintained in memory.

[0135] In some embodiments, for any data storage request, if all key-value pairs carried in the data storage request have been stored on the disk as files, and the target tree has been constructed or updated accordingly, then the storage layer system can return a data storage success message to the external system that sent the data storage request. For example, the data storage success message may carry the file identifier of the file containing the key-value pairs in the data storage request, the disk identifier of the disk containing them, etc., but this application embodiment does not limit this.

[0136] As an example, the target tree in this embodiment does not need to be persisted, that is, it does not need to be stored on disk. Instead, the target tree can be kept in memory, which can save storage resources.

[0137] In this scenario, if the storage layer system restarts due to a crash or other reasons, it can reconstruct the target tree in memory based on all files stored on the disk after the restart. It should be noted that the "files" mentioned in this embodiment refer to files containing key-value pairs as described in this embodiment.

[0138] Specifically, because file identifiers are globally ordered, and each leaf node stores a file identifier and the minimum and maximum key values ​​within that file, the storage layer system can construct all leaf nodes of the last level of the target tree in memory based on all files stored on the disk. Then, it builds branch nodes upwards layer by layer from the last level leaf nodes until the root node is constructed, thus reconstructing the target tree. After reconstructing the target tree, it maintains pointers to the root node and the last leaf node in the constructed target tree in memory.

[0139] After constructing the initial target tree through steps (1) to (3) above, the subsequent storage layer system can perform data processing based on the target tree, such as data querying and data storage. The data processing method provided in the embodiments of this application will be explained in detail below.

[0140] Figure 4 is a flowchart of a data processing method provided in an embodiment of this application. This method can be applied to a storage layer system. Referring to Figure 4, the method includes the following steps:

[0141] Step 401: The storage layer system receives a data storage request, which carries a key-value pair to be stored, and the key values ​​in the key-value pair are globally ordered.

[0142] The operation of step 401 is similar to the operation of step (1) above, and will not be described again in this embodiment.

[0143] Step 402: The storage layer system generates a new file containing one or more key-value pairs according to the order of the key values ​​in the key-value pairs, and stores the new file; wherein, the file identifiers of all files containing key-value pairs are globally ordered.

[0144] The operation of step 402 is similar to that of step (2) above, and will not be described again in this embodiment.

[0145] Step 403: The storage layer system adds a new leaf node after the last leaf node in the target tree and updates the target tree based on the new leaf node. The new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value among all key-value pairs in the new file.

[0146] Optionally, the storage layer system can first obtain a pointer to the last leaf node in the target tree in memory, and then add a new leaf node after the leaf node based on the pointer of the leaf node.

[0147] For example, the last leaf node in the target tree can be located at the rightmost edge of the last level of the target tree. In this case, the leaf nodes are inserted from left to right. Therefore, when the storage layer system adds a new leaf node after the last leaf node in the target tree, it inserts the new leaf node at the rightmost edge of the last level of the target tree.

[0148] Alternatively, the last leaf node in the target tree can be located at the leftmost position of the last level of the target tree. In this case, the leaf nodes are inserted from right to left. Therefore, the storage layer system adds a new leaf node after the last leaf node in the target tree by inserting it at the leftmost position of the last level of the target tree.

[0149] In this embodiment of the application, when storing data, there is no need to index the target tree. New files can be generated directly for storage, and new leaf nodes can be added directly to the target tree to update the target tree, thereby improving data storage efficiency.

[0150] It should be noted that the target tree update logic in this embodiment has only one form: inserting a new leaf node after the last leaf node of the entire tree (such as the rightmost or leftmost node of the last level). Traditional tree structures, however, use random insertion logic, which requires indexing down from the root node to the leaf node before inserting and storing data, followed by backtracking to update or split the parent node. Compared to the random insertion logic in traditional tree structures, the target tree update logic in this embodiment avoids the indexing process from the root node to the leaf node, thereby improving data storage efficiency.

[0151] It should be noted that, since the file identifiers are globally ordered, and each leaf node stores the file identifier of a file, the file identifiers in all leaf nodes at the last level of the target tree exist in order from left to right or from right to left.

[0152] Another point to note is that in this embodiment, the overall tree shape of the target tree is relatively fixed. Except for the nodes along the path from the root node to the last leaf node (i.e., the rightmost or leftmost path), all other nodes are static nodes. Static nodes are nodes that do not undergo modification and whose overall shape is determined.

[0153] In some embodiments, the target tree is an n-ary tree, where n is an integer greater than or equal to 2. When the target tree is an n-ary tree, no node splitting is required during the update of the target tree based on the new leaf node, thereby improving update efficiency.

[0154] Optionally, the storage layer system can update the target tree based on the new leaf node as follows: the storage layer system updates nodes upwards in the target tree layer by layer based on the new leaf node until the root node is reached. During the upward node update process, if an existing node in the target tree needs to be updated, a write lock is acquired on that existing node, and released after the update is complete. If a new node is added to the target tree during the upward node update process, no write lock needs to be acquired on that new node.

[0155] It should be noted that, in this embodiment of the application, a write lock is first acquired on the existing node when updating the existing node in the target tree. In this case, if a query operation on the existing node occurs during the update process, it must wait until the update of the existing node is completed and the write lock is released before it can be performed, thus ensuring the accuracy of data query.

[0156] In this embodiment, new nodes added during the target tree update process do not require write locks. This is because new nodes are not visible to the outside world before the target tree update is complete, so the data query process will not pass through the new nodes, and therefore there is no need to acquire write locks.

[0157] In some embodiments, the operation of the storage layer system updating nodes layer by layer upwards in the target tree based on the new leaf node may include the following steps a to f.

[0158] Step a: Set the new leaf node as the current node.

[0159] Step b: Determine whether the current node is a new node or an existing node in the target tree. If the current node is a new node, proceed to steps c through e; if the current node is an existing node, proceed to step f.

[0160] Step c: If the current node is a new node, determine whether the current node can be a child node of the last node in the parent hierarchy. If the current node can be a child node of the last node in the parent hierarchy, proceed to step d; if the current node cannot be a child node of the last node in the parent hierarchy, proceed to step e.

[0161] For example, if the target tree is an n-ary tree, the current node can be a child of the last node in the upper level if the number of child nodes of the last node in the upper level is less than n; if the number of child nodes of the last node in the upper level is equal to n, the current node cannot be a child of the last node in the upper level.

[0162] Step d: If the current node can be the child node of the last node in the upper layer, then make the last node in the upper layer the parent node of the current node, acquire a write lock on the parent node of the current node, update the parent node of the current node, release the write lock on the parent node of the current node after the update is complete, make the parent node of the current node the current node, and re-execute step b and subsequent steps.

[0163] If the current node can be a child of the last node in the parent hierarchy, then the last node in the parent hierarchy can be used as the parent node of the current node for updating. This update involves adding the current node's information to the list of child nodes of the last node in the parent hierarchy.

[0164] Step e: If the current node cannot be the child node of the last node in the upper layer, then add a new node in the upper layer as the parent node of the current node, make the parent node of the current node the current node, and re-execute step b and subsequent steps.

[0165] If the current node cannot be a child of the last node in the parent hierarchy, a new node can be added after the last node in the parent hierarchy to serve as the parent node of the current node. In this case, the parent node of the current node includes a list of child nodes, which contains the node information of the current node.

[0166] Step f: If the current node is an existing node, acquire a write lock on the parent node of the current node, update the parent node of the current node, release the write lock on the parent node of the current node after the update is complete, make the parent node of the current node the current node, and re-execute step b and subsequent steps.

[0167] If the current node is an existing node, then its parent node already exists in the target tree, so we can directly update the parent node. This update involves updating the node information of the current node, including its child nodes, within the list of its parent node.

[0168] It should be noted that after re-determining the current node in step d, step e, or step f, before re-executing step b, it is possible to determine whether the current node is the original root node of the target tree.

[0169] If the current node is not the original root node of the target tree, and is not a node at the same level as the original root node, then re-execute step b and subsequent steps.

[0170] If the current node is the original root node of the target tree, then the update of the target tree is complete.

[0171] If the current node is not the original root node of the target tree, but is at the same level as the original root node, then a new root node needs to be added at the upper level as the parent node of both the original root node and the current node. The new root node includes a list of child nodes, which contains the node information of the original root node and the node information of the current node. This completes the update of the target tree. In this case, the height of the target tree will increase by one level.

[0172] It should be noted that after updating the target tree, the pointers to the root node and the last leaf node in the target tree maintained in memory can be updated.

[0173] Updating the pointer to the root node in memory and retrieving the pointer to the root node in memory cannot be done concurrently. That is, if the pointer to the root node in memory is being updated, the operation to retrieve it must wait until the update is complete; conversely, if the pointer to the root node in memory is being retrieved, the operation to update it must wait until the pointer is successfully retrieved. This is because updating the pointer may result in the pointer being in a transitional state, such as being null. In this case, retrieving the pointer would lead to an error.

[0174] In some cases, external systems need to query data stored in the storage tier system. In such cases, the external system can send a data query request to the storage tier system. For example, these data query requests can include two types: the first is for querying the value in a key-value pair, and the second is for querying the file containing the key-value pairs. These two query methods are explained in detail below.

[0175] The first method may include steps A through C.

[0176] Step A: The storage layer system receives the first data query request, which carries the target key value.

[0177] The target key value is the key value in the key-value pair to which the value to be queried belongs.

[0178] The first data query request is used to request the value of the key-value pair to which the target key belongs.

[0179] The first data query request may be sent by an external system. For example, it may be sent by a database system, a blockchain system, etc., but this application embodiment does not limit this.

[0180] Step B: The storage layer system determines the first leaf node in the target tree where the target key value is indexed.

[0181] The target key value is greater than or equal to the minimum key value in the first leaf node, and less than or equal to the maximum key value in the first leaf node.

[0182] For example, the first leaf node to which the target key value is indexed in the target tree can also be referred to as the first leaf node to which the target key value is distributed in the target tree.

[0183] Optionally, the storage layer system can index the first leaf node in the target tree using a binary search based on the target key value. Of course, other methods can also be used to index the first leaf node in the target tree, and this application embodiment does not limit this.

[0184] For example, if the target tree is an n-ary tree, then the complexity of this indexing process satisfies O(logN) × log2branch. Here, the path complexity of the tree index is O(logN), and the complexity of finding a specified child node within each node using binary search is a constant value log2branch, where branch is a fixed number of branches, a constant n.

[0185] In some embodiments, step B can be performed as follows: the storage layer system indexes nodes in the target tree from the root node down layer by layer according to the target key value, until the first leaf node is indexed.

[0186] As an example, during the process of indexing nodes layer by layer downwards, a read lock is acquired for each node indexed, and the next node is indexed based on this node and the target key value. After indexing the next node, the read lock of this node is released; the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0187] Specifically, after receiving the first data query request, the storage layer system can first obtain a pointer to the root node in memory, and then index the root node based on that pointer. After indexing the root node, a read lock is acquired on the root node.

[0188] Next, determine which child node's key range the target key value falls within; this process is called indexing the next node. Once it's determined that the target key value falls within the key range of a child node of the root node, that child node is the next node to be indexed. At this point, release the read lock on the root node and acquire a read lock on the newly indexed node.

[0189] If this node is a branch node, then we can continue to determine which child node's key range the target key value belongs to, i.e., continue indexing the next node. This continues until the indexed node is a leaf node.

[0190] If this node is a leaf node, then it can be determined that this node is the first leaf node. Subsequently, after obtaining the file identifier in the first leaf node, the read lock of the first leaf node will be released.

[0191] As another example, during the process of indexing nodes layer by layer downwards, when a node is indexed, if the node is the last child node among all child nodes of its parent node, a read lock is acquired on the node, and the next node is indexed based on the node and the target key value. After indexing the next node, the read lock on the node is released. However, if the node is not the last child node among all child nodes of its parent node, no read lock is acquired on the node, and the next node is directly indexed based on the node and the target key value. Furthermore, in this case, no read lock is needed for any of the subsequently indexed nodes, thus saving lock resources.

[0192] If a read lock is acquired on the first leaf node during the indexing process, the read lock on the first leaf node will be released after the file identifier in the first leaf node is acquired.

[0193] It should be noted that in the embodiments of this application, the read lock and the read lock are shared locks, and the read lock and the write lock are mutually exclusive locks.

[0194] Optionally, the storage layer system in this application embodiment can be a concurrent model of single-threaded writing and multi-threaded reading.

[0195] In this scenario, if one thread acquires a read lock on a node, it does not affect another thread's ability to acquire a read lock on the same node.

[0196] If one thread acquires a read lock on a node, another thread cannot acquire a write lock on that node. It must wait until the read lock on that node is released before it can acquire a write lock and continue its operations (such as updating the node). Similarly, if one thread acquires a write lock on a node, another thread cannot acquire a read lock on that node. It must wait until the write lock on that node is released before it can acquire a read lock and continue its operations (such as indexing the node).

[0197] In this embodiment, a lock conflict (also known as lock preemption) will occur at most once during a tree indexing process and a tree update process. That is, the tree structure of the target tree in this embodiment can avoid path-level lock conflicts to the greatest extent, thereby maximizing system concurrency and reducing the latency of the main business process.

[0198] It should be noted that all data query behaviors in this application embodiment can be performed concurrently, which is very beneficial for scenarios involving concurrent data reading within the system. For example, if the blockchain system supports concurrent transaction execution, then all behaviors of obtaining ledger data during the concurrent execution of transactions can also be performed concurrently within this system.

[0199] In addition, the node insertion behavior in this embodiment cannot be concurrent, that is, the next update can only be performed after the target tree has been updated once.

[0200] Step C: The storage layer system retrieves the value of the key-value pair containing the target key from the file identified by the file identifier in the first leaf node.

[0201] Optionally, after obtaining the value, the storage layer system can return the data query result to the external system, which carries the value.

[0202] The second approach may include steps a through c.

[0203] Step a: The storage layer system receives a second data query request, which carries the target file identifier.

[0204] The target file identifier is the file identifier of the file to be queried.

[0205] The second data query request is used to request the file identified by the target file identifier.

[0206] The second data query request can be sent by an external system. For example, it can be sent by a database system, a blockchain system, etc., but this application embodiment does not limit this.

[0207] Step b: When the target tree is an n-ary tree, the storage layer system determines the index information in the target tree based on the target file identifier, n, and the height of the target tree.

[0208] The index information in the target tree is the information needed when indexing leaf nodes in the target tree.

[0209] Optionally, assuming the target tree has a height of m, where m is an integer greater than or equal to 2, the index information in the target tree can include the index information of each level in the target tree, that is, it can include the index information of each level from level 2 to level m.

[0210] The index information of the i-th level is used to indicate the sorting number of the node in the i-th level that needs to be indexed among all the child nodes of its parent node, where i is greater than or equal to 2 and less than or equal to m.

[0211] In other words, after indexing a node at level i-1, it is necessary to continue indexing nodes at level i. In this case, the next node can be indexed from all the child nodes of the node at level i-1, based on the index information of level i. For example, if the index information of level 2 is 0, then after indexing a node at level 1, i.e., the root node, the child node with the sort index 0 among all the child nodes of the root node can be taken as the next node to be indexed, and this next node is the node at level 2.

[0212] As an example, the storage layer system can obtain the file identifier in the last leaf node of the target tree maintained in memory based on the pointer to that leaf node. It can then increment the target value corresponding to that file identifier by 1 to obtain the number of leaf nodes in the target tree. Knowing the number of leaf nodes and the number of branches n in the target tree, the height m of the target tree can be determined.

[0213] The target value corresponding to a file identifier is the sort number of the file it identifies among all files. The sort number starts from 0 and increments by 1. That is, the sort number can be 0, 1, 2, 3, ...

[0214] There is a correspondence between file identifiers and target values. When a new file is added to the storage layer system, its sort number among all files can be used as the target value corresponding to its file identifier.

[0215] Because file identifiers are globally ordered, the target value corresponding to any given file identifier can be determined.

[0216] For example, if the file identifier starts from 0 and increments by 1, i.e., the file identifier is 0, 1, 2, 3, ..., then a file identifier is the same as its corresponding target value.

[0217] For example, if file identifiers start from 0 and increment by a fixed value, say j, where j is an integer greater than or equal to 2, then the file identifiers would be 0, j, 2j, 3j, ... In this case, the target value corresponding to a file identifier is the value obtained by dividing the file identifier by j.

[0218] Of course, file identifiers can also exist in other ordered forms, such as file identifiers being a, b, c, etc. In this case, the file identifiers can be converted into corresponding target values ​​according to certain rules, such as the target value corresponding to file identifier a being 0, the target value corresponding to file identifier b being 1, the target value corresponding to file identifier c being 2, etc.

[0219] Optionally, assume the height of the target tree is m, where m is an integer greater than or equal to 2. Step b can be performed as follows: The storage layer system acquires a read lock on the root node of the target tree. Determine the target value corresponding to the target file identifier, which is the sorting index of the file identified by the target file identifier among all files. Let i be m, and let the specified parameter be the target value corresponding to the target file identifier. Divide the specified parameter by n to obtain the quotient and remainder, and use the remainder as the index information of the i-th level. If i is not 2, let i = i-1, let the specified parameter be the quotient, and repeat the steps of dividing the specified parameter by n to obtain the quotient and remainder, and the subsequent steps, until i is 2; if i is 2, index the root node, and release the read lock on the root node after indexing the root node.

[0220] It should be noted that this embodiment requires acquiring a read lock on the root node during the process of determining the index information in the target tree. This is because if a new root node is added during the determination of the index information in the target tree, it will lead to an error where the index information determined based on the old root node starts indexing from the new root node. Therefore, this embodiment acquires a read lock on the root node before determining the index information in the target tree to prevent other threads from modifying the root node. After determining the index information in the target tree, the root node is indexed first, and the read lock on the root node is released after indexing the root node. This ensures that the root node at the start of the indexing process is consistent with the root node when the index information is determined, thereby ensuring the correctness of the indexing process.

[0221] Furthermore, in this embodiment, a read lock is acquired on the root node during the process of determining the index information of the target tree, which does not affect other data indexing processes. In other words, different processes of indexing leaf nodes through file identifiers can run concurrently.

[0222] It should be noted that dividing the specified parameter by n yields a quotient and a remainder. The quotient represents the sorting index of the parent node of the node to be indexed at level i among all nodes at level i-1. The remainder represents the sorting index of the node to be indexed at level i among all child nodes of its parent node. Therefore, after indexing nodes at level i-1, indexing can continue to nodes at level i.

[0223] In this embodiment, the index information in the target tree is determined starting from the index information at level m, then sequentially determining the index information at level (m-1), level (m-2), and so on, until the index information at level 2 is determined. During indexing, after reaching the root node, the index is sequentially determined from level 2, level 3, and so on, down to the leaf nodes.

[0224] The process of determining the index information in the target tree is illustrated below with reference to Figure 5.

[0225] As shown in Figure 5(b), the target tree is a binary tree with a height of 4.

[0226] Assuming the target file identifier corresponds to a target value of 6, as shown in Figure 5(a), firstly, determine the index information of the 4th level. Specifically, let the specified parameter be 6, divide 6 by 2, and get a quotient of 3 and a remainder of 0. Then, determine the index information of the 4th level as 0.

[0227] Then, we continue to determine the index information of the third level. Specifically, let the specified parameter be 3, divide 3 by 2, and get a quotient of 1 and a remainder of 1. Then we determine that the index information of the third level is 1.

[0228] Next, we continue to determine the index information of the second level. Specifically, we set the specified parameter to 1, divide 1 by 2, and get a quotient of 0 and a remainder of 1. Then we determine that the index information of the second level is 1.

[0229] At this point, the index information for each level from level 2 to level 4 in the target tree has been determined.

[0230] Subsequently, when indexing based on the index information of each level from level 2 to level 4, as shown in Figure 5(b), after indexing a node in level 1 (the root node), based on the index information of level 2, it can be determined that the node to be indexed in level 2 is the child node with a sorting index of 1 among all the child nodes of the root node. After indexing a node in level 2, based on the index information of level 3, it can be determined that the node to be indexed in level 3 is the child node with a sorting index of 1 among all the child nodes of this node in level 2. After indexing a node in level 3, based on the index information of level 4, it can be determined that the node to be indexed in level 4 is the child node with a sorting index of 0 among all the child nodes of this node in level 3. In this way, the corresponding leaf nodes are indexed.

[0231] Step c: The storage layer system indexes the second leaf node in the target tree based on the index information in the target tree. The second leaf node is a leaf node containing the target file identifier.

[0232] It should be noted that, in this embodiment of the application, the index path from the root node to the second leaf node can be directly determined based on the index information in the target tree. In this process, there is no need to read node data. Therefore, the process of indexing the second leaf node based on the index information in the target tree does not require a read lock.

[0233] In this embodiment, only m-1 calculations are needed to determine the index information in the target tree, and the index path can be directly determined based on the index information in the target tree. Thus, the indexing process has low complexity and high efficiency.

[0234] In some cases, the leaf nodes can store the disk identifier corresponding to the file identifier.

[0235] In this case, after the storage layer system indexes the second leaf node, it can obtain the disk identifier corresponding to the target file identifier from the second leaf node, and then obtain the file identified by the target file identifier from the disk identified by that disk identifier.

[0236] Optionally, after obtaining the file identified by the target file identifier, the storage layer system can return data query results to the external system, which contain the file information.

[0237] In this embodiment, a data storage request is received, carrying key-value pairs to be stored, where the keys in the key-value pairs are globally ordered. Then, based on the order of the keys in the key-value pairs, a new file containing one or more key-value pairs is generated and stored; wherein the file identifiers of all files containing key-value pairs are globally ordered. Then, a new leaf node is added after the last leaf node in the target tree, and the target tree is updated based on this new leaf node, which contains the file identifier of the new file, the minimum key value, and the maximum key value among all key-value pairs in the new file. In this embodiment, when storing data, there is no need to index the target tree; new files can be directly generated for storage, and new leaf nodes can be directly added to the target tree to update it, thus improving data storage efficiency.

[0238] Figure 6 is a schematic diagram of a data processing device provided in an embodiment of this application. This device can be implemented as part or all of a computer device by software, hardware, or a combination of both, and the computer device can be the computer device shown in Figure 7 below. Referring to Figure 6, the device includes: a first receiving module 601, a generating module 602, and an updating module 603.

[0239] The first receiving module 601 is used to receive a data storage request, which carries a key-value pair to be stored, wherein the key values ​​in the key-value pair are globally ordered.

[0240] The generation module 602 is used to generate a new file containing one or more key-value pairs according to the order of the key values ​​in the key-value pairs, and to store the new file; wherein the file identifiers of all files containing key-value pairs are globally ordered;

[0241] Update module 603 is used to add a new leaf node after the last leaf node in the target tree, and update the target tree based on the new leaf node. The new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value among all key-value pairs in the new file.

[0242] Optionally, the target tree is an n-ary tree, where n is an integer greater than or equal to 2.

[0243] Optionally, the branch nodes in the target tree include a list of child nodes, which includes the minimum and maximum key values ​​in each child node of the branch node.

[0244] Optionally, the target tree resides in memory, and the file is stored on disk. The device also includes:

[0245] The building module is used to rebuild the target tree in memory based on all files stored on the disk after a restart.

[0246] Optionally, the key-value pair includes a target key-value pair, where the key value in the target key-value pair includes the block number and the key value within the block identified by the block number, and the value value in the target key-value pair is the value within that block.

[0247] Optionally, the generation module 602 is used for:

[0248] Upon receiving a data storage request, all key-value pairs carried by the data storage request are stored in memory according to the order of the key values ​​in all key-value pairs carried by the data storage request.

[0249] If the difference between the total amount of data of one or more key-value pairs stored in memory in sequence and the preset amount of data is less than or equal to the preset difference, a new file containing the one or more key-value pairs will be generated.

[0250] Save the new file to the disk.

[0251] Optionally, update module 603 is used for:

[0252] Based on the new leaf node, update the nodes upwards in the target tree level by level until the root node is updated;

[0253] In the process of updating nodes layer by layer upwards, if it is necessary to update an existing node in the target tree, a write lock is acquired on the existing node, and the write lock is released after the update is completed.

[0254] Optionally, the device further includes:

[0255] The second receiving module is used to receive the first data query request, which carries the target key value.

[0256] The first determining module is used to determine the first leaf node indexed by the target key value in the target tree;

[0257] The retrieval module is used to retrieve the value of the key-value pair to which the target key belongs from the file identified by the file identifier in the first leaf node.

[0258] Optionally, the first determining module is used for:

[0259] Based on the target key value, index the nodes in the target tree from the root node down layer by layer until the first leaf node is indexed;

[0260] In the process of indexing nodes layer by layer downwards, a read lock is acquired for each node indexed, and the next node is indexed based on this node and the target key value. The read lock of the next node is released after it is indexed. The read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0261] Optionally, the target tree is an n-ary tree, where n is an integer greater than or equal to 2, and the device further includes:

[0262] The third receiving module is used to receive the second data query request, which carries the target file identifier.

[0263] The second determining module is used to determine the index information in the target tree based on the target file identifier, n, and the height of the target tree;

[0264] The indexing module is used to index the second leaf node in the target tree based on the index information in the target tree. The second leaf node is a leaf node containing the target file identifier.

[0265] Optionally, the second determining module is used for:

[0266] Add a read lock to the root node of the target tree. The height of the target tree is m, where m is an integer greater than or equal to 2.

[0267] Determine the target value corresponding to the target file identifier. The target value corresponding to the target file identifier is the sort number of the file identified by the target file identifier among all files.

[0268] Let i be m, and let the specified parameter be the target value corresponding to the target file identifier;

[0269] Divide the specified parameter by n to obtain the quotient and remainder;

[0270] The remainder is used as the index information of the i-th level, which indicates the sorting number of the node in the i-th level among all the child nodes of the parent node.

[0271] If i is not 2, let i = i-1, let the specified parameter be the quotient, and repeat the steps of dividing the specified parameter by n to obtain the quotient and remainder, and the subsequent steps, until i is 2.

[0272] If i is 2, then the root node is indexed, and the read lock on the root node is released after indexing the root node.

[0273] In this embodiment, a data storage request is received, carrying key-value pairs to be stored, where the keys in the key-value pairs are globally ordered. Then, based on the order of the keys in the key-value pairs, a new file containing one or more key-value pairs is generated and stored; wherein the file identifiers of all files containing key-value pairs are globally ordered. Then, a new leaf node is added after the last leaf node in the target tree, and the target tree is updated based on this new leaf node, which contains the file identifier of the new file, the minimum key value, and the maximum key value among all key-value pairs in the new file. In this embodiment, when storing data, there is no need to index the target tree; new files can be directly generated for storage, and new leaf nodes can be directly added to the target tree to update it, thus improving data storage efficiency.

[0274] It should be noted that the data processing device provided in the above embodiments is only illustrated by the division of the above functional modules when processing data. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0275] The functional units and modules in the above embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of the embodiments of this application.

[0276] The data processing apparatus and data processing method embodiments provided in the above embodiments belong to the same concept. The specific working process and technical effects of the units and modules in the above embodiments can be found in the method embodiment section, and will not be repeated here.

[0277] Figure 7 is a schematic diagram of the structure of a computer device provided in an embodiment of this application. As shown in Figure 7, the computer device 7 includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, it implements the steps in the data processing method in the above embodiment.

[0278] Computer device 7 can be a general-purpose computer device or a special-purpose computer device. In specific implementations, computer device 7 can be the aforementioned storage layer system, specifically a desktop computer, portable computer, network server, handheld computer, mobile phone, tablet computer, wireless terminal device, communication device, or embedded device. The embodiments of this application do not limit the type of computer device 7. Those skilled in the art will understand that FIG7 is merely an example of computer device 7 and does not constitute a limitation on computer device 7. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0279] Processor 70 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0280] In some embodiments, memory 71 may be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, memory 71 may be an external storage device of the computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., provided on the computer device 7. Furthermore, memory 71 may include both internal storage units and external storage devices of the computer device 7. Memory 71 is used to store the operating system, applications, boot loader, data, and other programs. Memory 71 may also be used to temporarily store data that has been output or will be output.

[0281] This application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0282] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.

[0283] This application provides a computer program product that, when run on a computer, causes the computer to perform the steps described in the various method embodiments above.

[0284] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above method embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage devices. The computer-readable storage medium mentioned in this application can be a non-volatile storage medium; in other words, it can be a non-transient storage medium.

[0285] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented in whole or in part as a computer program product. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above-described computer-readable storage medium.

[0286] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0287] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0288] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0289] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0290] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, The method includes: Receive a data storage request, the data storage request carrying a key-value pair to be stored, the key values ​​in the key-value pair being globally ordered; Based on the order of the key values ​​in the key-value pairs, a new file containing one or more key-value pairs is generated and stored; wherein, the file identifiers of all files containing the key-value pairs are globally ordered. A new leaf node is added after the last leaf node in the target tree, and the target tree is updated based on the new leaf node. The new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value among all key-value pairs in the new file.

2. The method as described in claim 1, characterized in that, The branch nodes in the target tree include a list of child nodes, and the list of child nodes includes the minimum key value and the maximum key value in each child node of the branch node.

3. The method as described in claim 1, characterized in that, The target tree is located in memory, the file is stored on disk, and the method further includes: After restarting, the target tree is reconstructed in memory based on all files stored on the disk.

4. The method as described in claim 1, characterized in that, The key-value pair includes a target key-value pair, wherein the key value in the target key-value pair includes a block number and a key value within the block identified by the block number, and the value value in the target key-value pair is the value within the block.

5. The method as described in any one of claims 1 to 4, characterized in that, The step of generating a new file containing one or more key-value pairs according to the order of the key values ​​in the key-value pairs, and storing the new file, includes: Upon receiving each data storage request, all key-value pairs carried by the data storage request are stored in memory according to the order of the key values ​​in all key-value pairs carried by the data storage request. If the difference between the total amount of data of one or more key-value pairs stored sequentially in memory and the preset amount of data is less than or equal to the preset difference, a new file containing the one or more key-value pairs is generated. Store the new file to the disk.

6. The method according to any one of claims 1 to 4, characterized in that, The step of updating the target tree based on the new leaf nodes includes: The new leaf node is used to update the nodes in the target tree layer by layer until the root node is updated. During the process of updating the nodes layer by layer, if it is necessary to update the existing nodes in the target tree, a write lock is acquired on the existing nodes, and the write lock is released after the update is completed.

7. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Receive a first data query request, the first data query request carrying a target key value; Determine the first leaf node indexed by the target key value in the target tree; Retrieve the value from the key-value pair to which the target key belongs from the file identified by the file identifier in the first leaf node.

8. The method as described in claim 7, characterized in that, Determining the first leaf node indexed by the target key value in the target tree includes: Based on the target key value, the nodes in the target tree are indexed layer by layer from the root node down until the first leaf node is indexed; In the process of indexing nodes layer by layer downwards, a read lock is acquired for each node indexed, and the next node is indexed based on the node and the target key value. The read lock of the node is released after the next node is indexed. The read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

9. The method according to any one of claims 1 to 4, characterized in that, The target tree is an n-ary tree, where n is an integer greater than or equal to 2. The method further includes: Receive a second data query request, which carries a target file identifier; Based on the target file identifier, n, and the height of the target tree, determine the index information in the target tree; The second leaf node is indexed in the target tree according to the index information. The second leaf node is a leaf node containing the identifier of the target file.

10. The method as described in claim 9, characterized in that, The step of determining the index information in the target tree based on the target file identifier, n, and the tree height of the target tree includes: Add a read lock to the root node of the target tree, where the height of the target tree is m, and m is an integer greater than or equal to 2; Determine the target value corresponding to the target file identifier, wherein the target value corresponding to the target file identifier is the sort number of the file identified by the target file identifier among all files; Let i be the value of m, and let the specified parameter be the target value corresponding to the target file identifier; Divide the specified parameter by n to obtain the quotient and remainder; The remainder is used as the index information of the i-th layer, which is used to indicate the sorting number of the node to be indexed in the i-th layer among all the child nodes of the parent node. If i is not 2, then let i = i-1, let the specified parameter be the quotient, and repeat the step of dividing the specified parameter by n to obtain the quotient and remainder, and the subsequent steps, until i is 2; If i is 2, then the root node is indexed, and the read lock on the root node is released after the root node is indexed.

11. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 10.

Images