Data processing method, computer device, and storage medium

Abstract

The present application relates to the technical field of computers, and discloses a data processing method, a computer device, and a storage medium. The method comprises: upon reception of a data query request, acquiring a target version number, wherein the data query request carries a target key value; then determining a target leaf node indexed by the target key value in a target tree, wherein a leaf node in the target tree contains a file identifier, the minimum key value and the maximum key value in all key value pairs in a file identified by the file identifier, and a mapping table; acquiring a target storage module identifier from a mapping table in the target leaf node on the basis of the target version number; and acquiring, from a file of which a file identifier in a storage module identified by the target storage module identifier is identical to the file identifier in the target leaf node, a value in a key value pair to which the target key value belongs. The present application can accurately implement data query.

Description

Data processing method, computer device and storage medium

[0001] The present application claims priority to the Chinese patent application No. 202410852801.4, filed on June 27, 2024 in the China Patent Office and entitled "Data processing method, computer device and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computer, in particular to a data processing method, a computer device and a storage medium. BACKGROUND

[0003] Multi-version concurrency control algorithm is a classic concurrency control algorithm in the field of database, and is an important function to realize transaction concurrency and rollback. The main idea is to save the historical version of data, and to realize the concurrency control of database by managing multiple versions of data rows. In this way, it can be determined whether the data is displayed by comparing the version number, and the transaction isolation effect can be guaranteed when reading data without locking.

[0004] However, the current multi-version concurrency control algorithm only considers single machine data storage scenarios, and it is often difficult to guarantee the accuracy of query data in distributed data storage scenarios. TECHNICAL PROBLEM

[0005] The present application provides a data processing method, a computer device and a storage medium, which can guarantee the accuracy of query data. TECHNICAL SOLUTION

[0006] In a first aspect, a data processing method is provided, which is applied to a storage layer system, the storage layer system comprising a plurality of storage modules, each of the plurality of storage modules being configured to store a file, and the plurality of storage modules being capable of file migration, the method comprising:

[0007] If a data query request is received, a target version number is obtained, the data query request carrying a target key value, the target version number being the latest system version number when the data query request is received, and the update of the system version number being triggered by any one of a file storage operation or a file migration operation;

[0008] A target leaf node to which the target key value is indexed in a target tree is determined, the leaf nodes in the target tree comprising a file identifier, a minimum key value and a maximum key value in all key-value pairs in a file identified by the file identifier, and a mapping table, the mapping table comprising a mapping relationship between a version number and a storage module identifier, and the mapping table being configured to indicate the storage module in which the file identified by the file identifier is located at each version.

[0009] obtain a target storage module identifier from the mapping table in the target leaf node according to the target version number;

[0010] obtain a value in a key-value pair to which the target key value belongs from a file identified by the file identifier in the storage module identified by the target storage module identifier.

[0011] In a second aspect, a data processing apparatus is provided, which is applied to a storage layer system, the storage layer system comprising a plurality of storage modules, each of the plurality of storage modules being configured to store files, and the plurality of storage modules being capable of file migration, the apparatus comprising:

[0012] a first obtaining module, configured to obtain a target version number if a data query request is received, the data query request carrying a target key value, the target version number being a latest system version number at the time when the data query request is received, and the update of the system version number being triggered by any one of a file storage operation and a file migration operation;

[0013] a determining module, configured to determine a target leaf node to which the target key value is indexed in a target tree, each leaf node in the target tree comprising a file identifier, minimum and maximum key values in all key-value pairs in a file identified by the file identifier, and a mapping table, the mapping table comprising a mapping relationship between version numbers and storage module identifiers, and the mapping table being configured to indicate storage modules in which the file identified by the file identifier is located at different versions;

[0014] a second obtaining module, configured to obtain a target storage module identifier from the mapping table in the target leaf node according to the target version number;

[0015] a third obtaining module, configured to obtain a value in a key-value pair to which the target key value belongs from a file identified by the file identifier in the storage module identified by the target storage module identifier.

[0016] In a third aspect, a computer device is provided, the computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the computer program, when executed by the processor, implements the data processing method of the first aspect.

[0017] In a fourth aspect, a computer readable storage medium is provided, the computer readable storage medium storing a computer program, and the computer program, when executed by a processor, implements the data processing method of the first aspect.

[0018] In a fifth aspect, a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the steps of the data processing method of the first aspect described above. Advantages

[0019] In the present application, the storage layer system stores files through a plurality of storage modules, and file migration can be performed between the plurality of storage modules, so that the data storage capability and data storage flexibility can be improved. In this case, both the file storage operation and the file migration operation trigger version update. Therefore, the present application embodiment also saves a mapping table in the leaf node in the target tree, according to which the storage module where the file is located under each version can be determined, so that the required value can be obtained from the corresponding storage module according to the data query request, so that accurate data query can be ensured. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0021] FIG. 1 is a schematic diagram of a storage layer system according to an embodiment of the present application;

[0022] FIG. 2 is a schematic diagram of a target tree according to an embodiment of the present application;

[0023] FIG. 3 is a schematic diagram of a key-value pair according to an embodiment of the present application;

[0024] FIG. 4 is a schematic diagram of another key-value pair according to an embodiment of the present application;

[0025] FIG. 5 is a schematic diagram of index information according to an embodiment of the present application;

[0026] FIG. 6 is a flowchart of a data processing method according to an embodiment of the present application;

[0027] FIG. 7 is a structural schematic diagram of a data processing apparatus according to an embodiment of the present application;

[0028] FIG. 8 is a structural schematic diagram of a computer device according to an embodiment of the present application. Embodiments of the present application

[0029] In order to make the purpose, technical solutions and advantages of the present application more clear, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0030] The data processing method provided in the embodiments of the present application can be applied in a system with data processing requirements, which is referred to as a storage layer system in the embodiments of the present application. The storage layer system is described below.

[0031] The embodiments of the present application are applied to a scenario in which an external system performs data query based on the storage layer system. For example, the embodiments of the present application can be applied to a scenario in which data query is performed based on a target tree.

[0032] For example, a key (i.e., a key value) that needs to be queried is generated in the process in which a blockchain system executes a transaction in a block. In this case, the data processing method provided in the embodiments of the present application can be used to query a value corresponding to the key value based on the target tree.

[0033] The external system connected to the storage layer system is a system that has data query requirements and data storage requirements. For example, the external system connected to the storage layer system can be a database system, a blockchain system, and the like. The embodiments of the present application do not limit the external system.

[0034] The storage layer system is configured to receive, process, and reply to various requests sent by the external system. For example, the requests sent by the external system can include data storage requests, data query requests, and the like. The embodiments of the present application do not limit the requests.

[0035] For example, FIG. 1 is a schematic diagram of a storage layer system provided in the embodiments of the present application. As shown in FIG. 1, the storage layer system can include a plurality of storage modules, for example, storage module 1, storage module 2, …, and storage module m.

[0036] Each of the plurality of storage modules is configured to store files, and the plurality of storage modules can perform file migration.

[0037] The files in the storage modules include data, such as key-value pairs, sent by the external system and needing to be stored. For example, the storage modules can be disks or other devices that can be used to store data.

[0038] File migration refers to that the storage layer system stores files in one storage module in another storage module. For example, the storage layer system can migrate files in the storage module 1 to the storage module 2 for storage.

[0039] Optionally, one storage module can correspond to one instance, and the instance can bind a path of the corresponding storage module. In this way, the storage layer system can store data in or read data from the corresponding storage module through the instance.

[0040] The storage modules have identifiers, i.e., storage module identifiers. One storage module identifier is used to identify one storage module.

[0041] In some embodiments, the storage layer system has version numbers. One version number corresponds to one version, and the version corresponding to one version number can represent a file distribution state in the storage layer system, which indicates which files are stored in each storage module.

[0042] For example, storing a new file can trigger a version update, and migrating a file between storage modules can also trigger a version update. The version update is strictly serially executed in the storage layer system.

[0043] The storage layer system has a system version number, which can be a continuously increasing integer. The system version number is used to indicate the current latest version number of the storage layer system. For example, the system version number can be 0, 1, 2, 3, 4, ….

[0044] For example, the update of the system version number can be triggered by any one of the file storage operation or the file migration operation.

[0045] Specifically, when the storage layer system performs a file storage operation or a file migration operation, a new version number is generated, and the system version number can be updated to the new version number after the storage layer system completes the file storage operation or the file migration operation.

[0046] In some embodiments, the storage layer system can interface with a database system. The database system can interface with other systems that have data storage needs, such as a blockchain system. For example, when the blockchain system needs to store data, the data can be sent to the database system first, and then sent to the storage layer system by the database system, and the storage layer system can store the data based on the target tree.

[0047] Alternatively, the storage layer system and the database system can be independent systems, in which case the storage layer system and the database system can run on independent physical machines. Alternatively, the database system can include the storage layer system, in which case the database system includes a specific system that interfaces with the blockchain system and runs on the same physical machine as the storage layer system.

[0048] Next, the structure of the target tree provided by the embodiments of the present application is described.

[0049] The target tree is a tree structure. The target tree includes branch nodes and leaf nodes at the lowest level of the target tree, wherein a node with a child node number of 0 is a leaf node, a node with a child node number greater than or equal to 1 is a branch node, and both branch nodes and leaf nodes are logical nodes. The uppermost branch node in the target tree can be referred to as the root node, which is a branch node without a parent node in the target tree.

[0050] Optionally, the target tree can be an n-ary tree, i.e., the number of child nodes of each branch node in the target tree is at most n, where n is an integer greater than or equal to 2. Any branch node in the n-ary tree can have at most n child nodes. Also, for branch nodes in the same level of the n-ary tree, a new branch node can be added after the last branch node in the same level only when the last branch node has n child nodes.

[0051] The leaf node stores a file identifier, a maximum key value, a minimum key value, and a mapping table.

[0052] The file identifier is used to identify a file stored in the storage layer system. For example, the file identifier can be a file identity (ID), a file name, etc., which is not limited in the embodiments of the present application.

[0053] The minimum key value in the leaf node is the minimum key value in the file identified by the file identifier in the leaf node. The maximum key value in the leaf node is the maximum key value in the file identified by the file identifier in the leaf node.

[0054] In the embodiments of the present application, the leaf nodes in the target tree correspond to files one by one. That is, one leaf node corresponds to the file identifier of a file and the minimum key value and the maximum key value in the file.

[0055] The mapping table in the leaf node is used to indicate the storage module where the file identified by the file identifier in the leaf node is stored under each version. The mapping table can include a mapping relationship between version numbers and storage module identifiers. One version number and its mapped storage module identifier in the mapping relationship are used to indicate that the file is stored in the storage module identified by the storage module identifier when the file is in the version corresponding to the version number.

[0056] In some embodiments, the mapping table can be a double-linked list, and each node in the double-linked list includes a version number and a storage module identifier. In this case, the double-linked list can maintain two node pointers, one of which is a head pointer used to indicate the head node of the double-linked list, and the other of which is a tail pointer used to indicate the tail node of the double-linked list.

[0057] In the embodiments of the present application, the storage layer system can store one or more files, and each file can contain one or more key-value pairs. The file identifiers of all the files stored in the storage layer system are globally ordered, that is, the file identifier of any file is ordered in all the files stored in the storage layer system. In addition, the key values in all the key-value pairs contained in the files are also globally ordered, that is, the key value in any key-value pair contained in any file is ordered in all the key-value pairs in all the files stored in the storage layer system.

[0058] Optionally, the global order of the key values in the key-value pairs can be in the order of increasing dictionary order.

[0059] In this case, the minimum key value in a file refers to the key value with the earliest order among the key values in all the key-value pairs in the file. The maximum key value in a file refers to the key value with the latest order among the key values in all the key-value pairs in the file.

[0060] It should be noted that in the computer field, dictionary order can be used to indicate the size relationship between any two strings. The size relationship between two strings depends on the size relationship of the American Standard Code for Information Interchange (ASCII) codes of the characters in the string from left to right.

[0061] Optionally, the global order of the file identifiers of the files can be in the order of increasing dictionary order.

[0062] For example, the file identifier of a file can be increased by a preset value starting from 0.

[0063] The preset value can be set in advance. For example, the preset value can be set to 1, 2, 3, etc., which is not limited in the embodiments of the present application.

[0064] In this case, the file identifiers of all the files are an arithmetic sequence. For example, assuming that the preset value is 1, then the file identifiers of the first file, the second file, the third file, …, the kth file generated can be 0, 1, 2, …, k-1, respectively.

[0065] In some embodiments, the storage layer system can maintain a globally continuously increasing counter inside the storage layer system, which is used to generate file identifiers. In this case, the storage layer system can take the current value of the counter as the file identifier of a new file each time a new file is generated, so that the files stored in the storage layer system are a batch of files with continuously increasing file identifiers.

[0066] The branch node includes a child node list, which includes the minimum key value and the maximum key value in each child node of the branch node. For example, the range from the minimum key value in a child node to the maximum key value in the child node can be referred to as the key range of the child node.

[0067] The child node of a branch node can be a branch node or a leaf node of the next layer. The minimum key value in a branch node is the minimum key value in all key values in the child node list in the branch node, that is, the key value with the earliest order in all key values in the child node list; and the maximum key value in the branch node is the maximum key value in all key values in the child node list in the branch node, that is, the key value with the latest order in all key values in the child node list. The minimum key value and the maximum key value in a leaf node are the minimum key value and the maximum key value stored in the leaf node.

[0068] For example, FIG. 2 is a schematic diagram of a target tree provided by an embodiment of the present application. Referring to FIG. 2, the target tree is a 2-ary tree, and the target tree includes a plurality of leaf nodes (for example, nodes n4, n5, and n6 in FIG. 2) and a plurality of branch nodes (for example, nodes n1, n2, and n3 in FIG. 2). The child nodes of the node n1 include the nodes n2 and n3, the child nodes of the node n2 include the nodes n4 and n5, and the child nodes of the node n3 include the node n6. In the embodiment of the present application, the branch node at the uppermost layer in the target tree can be referred to as a root node, for example, the node n1 in FIG. 2 is a root node.

[0069] The leaf node n4 in FIG. 2 stores a file identifier 0, a minimum key value 1, a maximum key value 3, and a mapping table 0. The leaf node n5 stores a file identifier 1, a minimum key value 4, a maximum key value 6, and a mapping table 1. The leaf node n6 stores a file identifier 2, a minimum key value 7, a maximum key value 9, and a mapping table 2.

[0070] The branch node n2 in FIG. 2 stores a child node list, which includes the minimum key value 1 and the maximum key value 3 in the leaf node n4, and includes the minimum key value 4 and the maximum key value 6 in the leaf node n5.

[0071] The branch node n3 in FIG. 2 stores a child node list, which includes the minimum key value 7 and the maximum key value 9 in the leaf node n6.

[0072] The branch node n1 in FIG. 2 stores a child node list, which includes the minimum key value 1 and the maximum key value 6 in the branch node n2, and includes the minimum key value 7 and the maximum key value 9 in the branch node n3.

[0073] In the embodiments of the present application, the storage layer system can store the key-value pairs sent by the external system based on the target tree. The target tree can also be referred to as a solid branch tree.

[0074] In some cases, the key values in the key-value pairs sent by the external system to the storage layer system are globally ordered. The concept of global order has been explained above and will not be repeated here.

[0075] In some embodiments, the storage layer system can be connected to a database system, and the database system can be connected to a blockchain system. In this case, the blockchain system can send the key-value pairs to be stored to the database system, and the database system can send a data storage request carrying the key-value pairs to the storage layer system after receiving the key-value pairs.

[0076] As an example, the key-value pairs to be stored sent by the blockchain system to the database system can be block data. Since the block data is strictly organized according to the block number, that is, the key values in the block data are block numbers, the key values in the block data are naturally globally ordered. In this case, the database system can directly send the block data to the storage layer system for storage in the data storage request.

[0077] As another example, the key-value pairs to be stored sent by the blockchain system to the database system can be account data in a block. The account data in a block includes multiple key-value pairs, and the key values in the multiple key-value pairs in the same block are different, but the key values in the key-value pairs in different blocks can be the same.

[0078] In this case, the database system can first process the account data. Specifically, the database system can introduce a block number in the account data to construct key-value pairs with globally ordered key values, and then send the constructed key-value pairs to the storage layer system for storage in the data storage request. In this way, it can be ensured that the key values in the key-value pairs carried in the data storage request sent by the database system to the storage layer system are globally ordered.

[0079] Optionally, the key-value pairs can be referred to as target key-value pairs in the embodiments of the present application. The key values in the target key-value pairs include a block number and a key value in a block identified by the block number (i.e., a key value in the key-value pair in the account data), and the value in the target key-value pair is a value in the block (i.e., a value in the key-value pair in the account data).

[0080] In some embodiments, the key value in the target key-value pair can be obtained by splicing the block number and the key value within the block, of course, the key value in the target key-value pair can also be obtained by other ways, which are not limited in the embodiments of the present application.

[0081] Since the block number is naturally globally ordered, and the key values within a block are different, the key values in the target key-value pair containing the block number and the key values within the block identified by the block number are also globally ordered.

[0082] For example, the block with block number 00000001 includes a plurality of key-value pairs, for each key-value pair in the plurality of key-value pairs, the block number 00000001 can be added before the key value in the key-value pair to obtain a new key value.

[0083] For example, as shown in (a) of FIG. 3, the block 1 with block number 00000001 includes key-value pairs bbbb:{value1...}, bggg:{value2...}, caaa:{value3...}, xaab:{value4...}. As shown in (b) of FIG. 3, the block 2 with block number 00000002 includes key-value pairs aabb:{value5...}, bggg:{value6...}, bcde:{value7...}.

[0084] Then, after the blockchain system sends these key-value pairs to the database system, the database system can introduce the block number 00000001 in the key-value pairs in the block 1, specifically, splice 00000001 before the key value in each key-value pair in the block 1 to obtain target key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, 00000001xaab:{value4...}. Introduce the block number 00000002 in the key-value pairs in the block 2, specifically, splice 00000002 before the key value in each key-value pair in the block 2 to obtain target key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, 00000002bcde:{value7...}. The key values in the target key-value pairs obtained in this way are globally ordered.

[0085] It should be noted that the storage layer system in the embodiments of the present application is not limited to storing the key-value pairs in the blockchain system proposed above, but can also store other key-value pairs, as long as the key values in these key-value pairs are globally ordered.

[0086] The following describes a process of constructing an initial target tree for a storage layer system, which can include the following (1)-(5) steps:

[0087] (1) The storage layer system receives a data storage request carrying a key-value pair to be stored, the key value in the key-value pair being globally ordered.

[0088] The data storage request is used to instruct the storage layer system to store the key-value pair to be stored. In some embodiments, the data storage request can be sent by an external system, such as a blockchain system or a database system.

[0089] In this case, the key values in all key-value pairs to be stored sent by the external system to the storage layer system are ordered. After the storage layer system stores these key-value pairs, the key values in all key-value pairs stored in the storage layer system are also ordered.

[0090] (2) The storage layer system generates a new file containing one or more key-value pairs according to the order of the key values in the key-value pairs, and stores the new file in a first storage module, which is one of the plurality of storage modules; wherein the file identifiers of all files containing key-value pairs are globally ordered.

[0091] The first storage module is the storage module that stores the new file. For example, the first storage module can be a default one of the plurality of storage modules, or the first storage module can be a randomly selected one of the plurality of storage modules, and the present application does not limit this.

[0092] The key values in the key-value pairs to be stored received by the storage layer system are ordered. In this case, the storage layer system generates a new file according to the order of the key values in the key-value pairs, and the key values in the key-value pairs in the new file are also ordered.

[0093] After the storage layer system stores the new file in the first storage module, the identifier of the first storage module can be obtained.

[0094] In some embodiments, the operation of step (2) can be: the storage layer system receives each data storage request, and saves all key-value pairs carried by the data storage request in the memory according to the order of the key values in the key-value pairs. If the difference between the total data amount of one or more key-value pairs saved in order in the memory and the preset data amount is less than or equal to the preset difference, the storage layer system generates a new file containing the one or more key-value pairs. The storage layer system stores the new file in the first storage module.

[0095] The preset data amount can be preset. For example, the preset data amount can be set as 8 megabytes (MB), 9 MB, 10 MB, etc., and the embodiments of the present application do not limit this.

[0096] Optionally, the preset data amount can be set according to an integer multiple of the block size of the storage layer system. For example, if the block size is 4 MB, the preset data amount can be set as 4 MB, 8 MB, 12 MB, etc. The block size of the storage layer system refers to the smallest unit used by the storage layer system to organize and manage data storage.

[0097] The preset difference value can be preset. For example, the preset difference value can be set as 100 kilobytes (KB), 200 KB, 300 KB, etc., and the embodiments of the present application do not limit this.

[0098] It should be noted that the data amounts of the key-value pairs are generally close to each other, and the data amount of one key-value pair is usually within a predetermined range. In the embodiments of the present application, the preset data amount can be much larger than the data amount of one key-value pair. The preset difference value can be close to the data amount of one key-value pair, such as being equal to or slightly larger than the data amount of one key-value pair.

[0099] The storage layer system can save all the key-value pairs carried by each received data storage request in sequence in the memory. In this case, if the total data amount of all the key-value pairs currently saved in sequence in the memory is close to the preset data amount (i.e., the difference between the total data amount and the preset data amount is less than or equal to the preset difference value), all the key-value pairs currently saved in the memory can be stored in the first storage module in the form of a new file. In this way, batch storage of key-value pairs can be implemented, thereby improving the data storage efficiency. Moreover, in this case, the file sizes of each file stored in the first storage module are basically consistent, and are close to the preset data amount, thereby facilitating file management.

[0100] It should be noted that if the total data amount of all the key-value pairs currently saved in the memory is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference value, it indicates that the total data amount of the key-value pairs currently saved in the memory is greatly different from the preset data amount, and does not meet the condition for generating a new file. In this case, the storage layer system can continue to receive data storage requests sent by the external system, and save the key-value pairs carried by the data storage requests in the memory, until the difference between the total data amount of the key-value pairs currently saved in sequence in the memory and the preset data amount is less than or equal to the preset difference value, and then a new file is generated and stored.

[0101] The generated new file is stored in the first storage module, i.e., the persistence of the new file is implemented. Moreover, after the new file is stored in the first storage module, one or more key-value pairs in the new file are deleted from the memory.

[0102] It should be noted that the key values in the key-value pairs stored in the memory are in order. In this case, a new file is generated according to the key-value pairs stored in the memory, and the key values in all the key-value pairs in the new file are also in order.

[0103] It should be noted that since one file can contain one or more key-value pairs, the key-value pairs in one file can cover the data of one block or multiple blocks.

[0104] As shown in FIG. 4, the storage layer system receives a data storage request 1 carrying key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. The storage layer system stores the key-value pairs carried in the data storage request 1 in the memory, and the key-value pairs currently stored in the memory include 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. Since the total data amount of these key-value pairs is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference, the condition for generating a new file is not met.

[0105] Then, the storage layer system receives a data storage request 2 carrying key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, and 00000002bcde:{value7...}. The storage layer system stores the key-value pairs carried in the data storage request 2 in the memory in order. After storing the key-value pair 00000002bggg:{value6...} in the memory, the storage layer system finds that the difference between the total data amount of the key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} currently stored in the memory and the preset data amount is less than or equal to the preset difference, and the condition for generating a new file is met. Therefore, a new file containing the key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} is generated and stored in the first storage module. In addition, the storage layer system stores the key-value pair 00000002bcde:{value7...} carried in the data storage request 2 in the memory.

[0106] Afterwards, the storage layer system receives a data storage request 3 carrying a key-value pair 00000003bcbb:{value1...}. The storage layer system saves the key-value pair carried in the data storage request 3 in sequence to the memory, and then the key-value pairs currently saved in the memory include 00000002bcde:{value7...} and 00000003bcbb:{value1...}. Since the total data amount of these key-value pairs is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference, the new file generation condition is not met. The storage layer system continues to wait for the next data storage request.

[0107] (3) The storage layer system generates an initial version number.

[0108] The initial version number can be preset. For example, the initial version number can be 0.

[0109] Since the storage layer system stores the file for the first time, the storage layer system can generate the initial version number.

[0110] After the storage layer system stores the new file to the first storage module and generates the initial version number, the storage layer system can determine the mapping relationship between the initial version number and the identifier of the first storage module, which is used to indicate that the new file is stored in the first storage module at the version corresponding to the initial version number.

[0111] (4) The storage layer system generates a new leaf node, and constructs an initial target tree according to the new leaf node. The new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file, and a mapping table including the mapping relationship between the initial version number and the identifier of the first storage module.

[0112] After generating the new leaf node, the storage layer system can newly create a branch node as the parent node of the new leaf node. The branch node saves a child node list, and the child node list includes the minimum key value and the maximum key value in the new leaf node. Since the number of current leaf nodes is 1, the branch node is the root node, and thus the construction of the initial target tree is completed.

[0113] In some embodiments, a branch node in the target tree can include a child node list. The child node list in a branch node saves the node information of each child node of the branch node.

[0114] For example, the node information of a node can include the pointer of the node, the minimum key value and the maximum key value in the node.

[0115] The pointer of a node is used to point to the memory space where the node is located.

[0116] In this case, the child node list in one branch node includes node information of all child nodes of the branch node, and the node information of all child nodes is in order in the child node list.

[0117] In some cases, after the storage layer system completes the construction of the initial target tree, the storage layer system can also save in memory a pointer to the root node of the target tree and a pointer to the last leaf node in the target tree. In this way, the storage layer system can subsequently quickly index to the root node in the target tree according to the pointer to the root node maintained in the memory and quickly index to the last leaf node in the target tree according to the pointer to the leaf node maintained in the memory.

[0118] (5) The storage layer system creates a system version number and updates the system version number to the initial version number.

[0119] After the storage layer system completes the construction of the initial target tree, the storage layer system can create a system version number in memory and update the system version number to the initial version number.

[0120] In some embodiments, for any data storage request, if all key-value pairs carried by the data storage request have been stored in the storage module in the form of files and the target tree has been constructed or updated and the system version number has been updated, the storage layer system can return a data storage success message to the external system that sent the data storage request. For example, the data storage success message can carry the file identifier of the file in which the key-value pairs in the data storage request are located, the identifier of the storage module, and the like, and the embodiments of the present application do not limit this.

[0121] In some embodiments, after the storage layer system updates the system version number each time, the storage layer system can store version information of the version corresponding to the updated system version number to the disk, the version information including the version number and version increment information, the version increment information being used to indicate the change in the files in the storage module under the version corresponding to the version number compared to the previous version. That is, the version increment information is used to indicate which storage module decreases which files and which storage module increases which files under the version compared to the previous version. It should be noted that the "file" referred to in the embodiments of the present application refers to the file containing key-value pairs referred to in the embodiments of the present application.

[0122] It should be noted that the decrease in the files in the storage module in the embodiments of the present application is caused by file migration, and the increase in the files in the storage module can be caused by file storage or file migration.

[0123] Optionally, the version information can also include the largest file identifier (i.e., the file identifier at the back of the order) under the version.

[0124] In this case, the version information of the version corresponding to the updated system version number can be stored into the disk every time the system version number is updated, so that the storage layer system stores the version information of each version into the disk in sequence, that is, one or more version information stored in the disk is in order.

[0125] In some embodiments, the storage layer system can update the file distribution state stored in the disk according to one or more version information stored in the disk, the file distribution state containing the correspondence between the version number, the storage module identifier and the file identifier. The file distribution state is used to indicate the files stored in each storage module in the plurality of storage modules under the corresponding version. Wherein, the file identifier corresponding to any one of the storage module identifiers in the correspondence between the storage module identifier and the file identifier is the file identifier of each file in all files stored in the storage module identified by the storage module identifier.

[0126] Since one or more version information stored in the disk is in order, the storage layer system updates the file distribution state stored in the disk in sequence, that is, sequentially according to the version order.

[0127] When the storage layer system updates the file distribution state stored in the disk according to a version information, the version number in the file distribution state can be updated to the version number in the version information, and the correspondence between the storage module identifier and the file identifier in the file distribution state can be updated according to the version increment information in the version information. Then, after updating the file distribution state according to the version information of a version, the updated file distribution state is used to indicate the files stored in each storage module in the plurality of storage modules under the version.

[0128] After the storage layer system updates the file distribution state according to a version information, the version information can be deleted from the disk.

[0129] In some cases, the file distribution state can be updated asynchronously by a thread. That is, the storage layer system records version information every time a version update occurs, so that there are multiple version information after multiple version updates. There can be a thread that updates the file distribution state according to the recorded multiple version information in sequence.

[0130] For example, Table 1 below shows multiple version information and the file distribution state updated in sequence according to each version information in the multiple version information.

[0131] Specifically, when the storage layer system stores a new file 3 in the storage module 1 to obtain a version corresponding to version number 3, the version information of the version is "version number: 3, version increment information: the storage module 1 adds the file 3". According to the version information, the file distribution state can be updated to "version number: 3, the correspondence between the storage module identifier and the file identifier: storage module 1: {file 2, file 3}, storage module 2: {file 1}".

[0132] Then, when the storage layer system stores a new file 4 in the storage module 1 to obtain a version corresponding to version number 4, the version information of the version is "version number: 4, version increment information: the storage module 1 adds the file 4". According to the version information, the file distribution state can be updated to "version number: 4, the correspondence between the storage module identifier and the file identifier: storage module 1: {file 2, file 3, file 4}, storage module 2: {file 1}".

[0133] Then, when the storage layer system stores a new file 4 in the storage module 1 to obtain a version corresponding to version number 4, the version information of the version is "version number: 4, version increment information: the storage module 1 adds the file 4". According to the version information, the file distribution state can be updated to "version number: 4, the correspondence between the storage module identifier and the file identifier: storage module 1: {file 2, file 3, file 4}, storage module 2: {file 1}".

[0134] Table 1

[0135] The present application only takes Table 1 as an example to exemplarily illustrate the version information and the file distribution state, and Table 1 does not limit the present application.

[0136] As an example, the target tree in the present application does not need to be persisted, that is, does not need to be stored in a disk, but can be saved in a memory, so that storage resources can be saved.

[0137] In this case, if the storage layer system restarts due to a crash or the like, the storage layer system can obtain the version number in the latest version information stored in the disk as the system version number after the restart, and save the system version number in the memory. In addition, the storage layer system can update the file distribution state stored in the disk according to all the version information stored in the disk, and reconstruct the target tree in the memory according to the updated file distribution state. In this case, the updated file distribution state is the file distribution state under the latest version (i.e., the version corresponding to the system version number), which can indicate the files stored in each storage module under the latest version.

[0138] Since the file identifiers are globally ordered and one-to-one corresponding to the leaf nodes, all the leaf nodes in the last layer of the target tree can be constructed according to all the file identifiers in the file distribution state and the minimum key value and the maximum key value in the file identified by each file identifier, and the mapping table in each leaf node can be constructed according to the version number (i.e., the version number of the latest version) in the file distribution state and the identifier of the storage module where each file is located. At this time, the mapping table only contains the mapping relationship between the version number and the corresponding storage module identifier. Then, the branch nodes are constructed layer by layer upwards according to the leaf nodes in the last layer until the root node is constructed, so that the target tree is reconstructed. After the target tree is reconstructed, the pointer of the root node in the reconstructed target tree and the pointer of the last leaf node in the target tree are maintained in the memory.

[0139] It should be noted that after the target tree is constructed, the subsequent storage layer system can perform data processing based on the target tree, such as data storage, file migration, data query, and the like.

[0140] Next, the process in which the storage layer system in the data processing method provided by the embodiment of the present application performs data storage and updates the target tree will be described, which can include the following steps A to E:

[0141] Step A: The storage layer system receives a data storage request, and the data storage request carries a key-value pair to be stored, and the key value in the key-value pair is globally ordered.

[0142] The operation of step A is similar to the operation of step (1) described above, and the embodiment of the present application will not be described again.

[0143] Step B: The storage layer system generates a new file containing one or more key-value pairs according to the order of the key value in the key-value pair, and stores the new file in a first storage module, and the first storage module is one of the plurality of storage modules; wherein the file identifiers of all the files containing the key-value pairs are globally ordered.

[0144] The operation of Step B is similar to that of Step (2) described above, and will not be repeated here.

[0145] Step C: The storage layer system acquires a first version number, which is the sum of the latest system version number at the time when the new file is stored and 1.

[0146] The first version number is a version number generated by the storage layer system after storing the new file, that is, a version number obtained by adding 1 to the latest system version number at the time when the new file is stored. For example, assuming that the latest system version number at the time when the new file is stored is 5, then the first version number is 6. In this case, version updating is strictly serially executed in the storage layer system.

[0147] After the storage layer system acquires the first version number, it can determine the mapping relationship between the first version number and the identifier of the first storage module, which is used to indicate that the new file is stored in the first storage module under the version corresponding to the first version number.

[0148] Step D: The storage layer system adds a first leaf node after the last leaf node in the target tree, updates the target tree according to the first leaf node, and the first leaf node contains the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file, and the first mapping table, which includes the mapping relationship between the first version number and the identifier of the first storage module.

[0149] The first leaf node is a leaf node newly added after the last leaf node in the target tree, that is, a new leaf node.

[0150] Optionally, the storage layer system can first acquire the pointer of the last leaf node in the target tree in the memory, and then add the first leaf node after the leaf node according to the pointer of the leaf node.

[0151] For example, the last leaf node in the target tree can be located at the rightmost side of the last layer of the target tree, in which case the insertion order of the leaf nodes is from left to right. Then, the storage layer system adds the first leaf node after the last leaf node in the target tree, that is, inserts the first leaf node at the rightmost side of the last layer of the target tree.

[0152] Alternatively, the last leaf node in the target tree can be located at the leftmost side of the last layer of the target tree, in which case the insertion order of the leaf nodes is from right to left. Then, the storage layer system adds the first leaf node after the last leaf node in the target tree, that is, inserts the first leaf node at the leftmost side of the last layer of the target tree.

[0153] In the embodiments of the present application, when data is stored, no indexing in the target tree is needed, and a new file can be directly generated for storage, and accordingly a new leaf node can be directly added in the target tree to update the target tree, thereby improving the data storage efficiency.

[0154] It should be noted that the update logic of the target tree in the embodiments of the present application is only one, that is, a new leaf node is inserted after the last leaf node of the whole tree (such as the rightmost or leftmost of the last layer). The traditional tree structure adopts a random insertion logic, which needs to index from the root node to the leaf node first, then insert and store data, and then backtrack to update or split the parent node. Compared with the random insertion logic in the traditional tree structure, the update logic of the target tree in the embodiments of the present application can avoid the indexing process from the root node to the leaf node, thereby improving the data storage efficiency.

[0155] It should be noted that, since the file identifiers are globally ordered, and each leaf node stores a file identifier of a file, the file identifiers in all leaf nodes in the last layer of the target tree are ordered from left to right or from right to left.

[0156] It should be noted that, in the embodiments of the present application, the tree shape of the whole target tree is relatively fixed, and except for the nodes in the path from the root node to the last leaf node (i.e. the rightmost path or the leftmost path), the remaining nodes are static nodes. A static node refers to a node that does not have a modification process and has a determined overall shape.

[0157] In some embodiments, the target tree is an n-ary tree, where n is an integer greater than or equal to 2. In the case of an n-ary tree, no node splitting is needed in the process of updating the target tree according to the first leaf node, thereby improving the update efficiency.

[0158] Optionally, the operation of updating the target tree according to the first leaf node by the storage layer system can be: the storage layer system updates the nodes layer by layer upwards in the target tree according to the first leaf node until the root node is updated. In the process of updating the nodes layer by layer upwards, if an original node in the target tree needs to be updated, a write lock is added to the original node, and the write lock is released after the update is completed. In the process of updating the nodes layer by layer upwards, if a new node is added in the target tree, no write lock is needed for the new node.

[0159] It should be noted that, in the embodiments of the present application, a write lock is added to an original node in the target tree before the original node is updated. In this case, if a query operation on the original node occurs during the update process of the original node, the query operation can be performed only after the write lock of the original node is released, thereby ensuring the accuracy of data query.

[0160] The new node added in the updating process of the target tree in the embodiment of the present application does not need to be added with a write lock. This is because the new node is invisible to the outside before the updating of the target tree is completed, so the data query process naturally does not pass through the new node, and there is no need to add a write lock.

[0161] In some embodiments, the operation of the storage layer system of updating the nodes layer by layer upwards in the target tree according to the first leaf node can include the following steps a to f.

[0162] Step a: let the first leaf node be a current node.

[0163] Step b: judge whether the current node is a new node or an original node in the target tree. If the current node is a new node, execute the following steps c to e; if the current node is an original node, execute the following step f.

[0164] Step c: if the current node is a new node, judge whether the current node can be a child node of the last node in the upper layer. If the current node can be a child node of the last node in the upper layer, execute the following step d; if the current node cannot be a child node of the last node in the upper layer, execute the following step e.

[0165] For example, in the case of an n-ary tree as the target tree, if the number of child nodes of the last node in the upper layer is less than n, the current node can be a child node of the last node in the upper layer; if the number of child nodes of the last node in the upper layer is equal to n, the current node cannot be a child node of the last node in the upper layer.

[0166] Step d: if the current node can be a child node of the last node in the upper layer, add the last node in the upper layer as a parent node of the current node, add a write lock to the parent node of the current node, update the parent node of the current node, release the write lock of the parent node of the current node after the update, let the parent node of the current node be the current node, and re-execute step b and the subsequent steps.

[0167] If the current node can be a child node of the last node in the upper layer, the last node in the upper layer can be added as a parent node of the current node for updating. The update here is to add the node information of the current node in the child node list of the last node in the upper layer.

[0168] Step e: if the current node cannot be a child node of the last node in the upper layer, add a new node in the upper layer as a parent node of the current node, let the parent node of the current node be the current node, and re-execute step b and the subsequent steps.

[0169] If the current node cannot be a child node of the last node in the upper layer, a new node can be added after the last node in the upper layer as the parent node of the current node. In this case, the parent node of the current node includes a child node list, which includes the node information of the current node.

[0170] Step f: If the current node is an original node, a write lock is added to the parent node of the current node, the parent node of the current node is updated, the write lock of the parent node of the current node is released after the update, the parent node of the current node is set as the current node, and step b and the subsequent steps are executed again.

[0171] If the current node is an original node, the parent node of the current node already exists in the target tree, and thus the parent node of the current node can be directly updated at this time. The update here is to update the node information of the current node included in the child node list in the parent node of the current node.

[0172] It should be noted that after the current node is re-determined in step d, step e or step f, it can be determined whether the current node is an original root node of the target tree before step b is executed again.

[0173] If the current node is not an original root node of the target tree and is not a node in the same layer as the original root node, step b and the subsequent steps are executed again.

[0174] If the current node is an original root node of the target tree, it is determined that the update of the target tree is completed.

[0175] If the current node is not an original root node of the target tree and is a node in the same layer as the original root node, a new root node needs to be added in the upper layer as the parent node of the original root node and the current node, and the new root node includes a child node list, which includes the node information of the original root node and the node information of the current node. In this way, the update of the target tree is completed. In this case, the height of the target tree increases by one layer.

[0176] It should be noted that after the target tree is updated, the pointer of the root node in the target tree and the pointer of the last leaf node in the target tree maintained in the memory can be updated.

[0177] The operation of updating the pointer of the root node in the memory cannot be concurrent with the operation of obtaining the pointer of the root node in the memory. That is, if the pointer of the root node in the memory is being updated, the operation of obtaining the pointer of the root node in the memory needs to wait until the updating is completed; if the pointer of the root node in the memory is being obtained, the operation of updating the pointer of the root node in the memory needs to wait until the pointer of the root node is successfully obtained. This is because the pointer of the root node can be in a transitional state, such as being empty, during the process of updating the pointer of the root node, and in this case, an error will occur if the pointer of the root node is obtained.

[0178] Step E: The storage layer system updates the system version number to the first version number.

[0179] The storage layer system can update the system version number to the first version number after the target tree is updated. In this way, the storage layer system updates the system version number after the file storage operation is performed.

[0180] In some embodiments, after updating the system version number, the storage layer system can store the version information of the version corresponding to the updated system version number to the disk.

[0181] Next, the process of file migration of the storage layer system in the data processing method provided by the embodiments of the present application is described, which can include the following steps A to step E:

[0182] Step A: The storage layer system migrates the first file in the second storage module to the third storage module, and the second storage module and the third storage module are different storage modules in the plurality of storage modules.

[0183] The second storage module is the storage module that stores the first file before migration.

[0184] The first file is a file to be migrated. The number of the first file can be one or more.

[0185] The third storage module is the storage module to which the first file is migrated.

[0186] In some cases, if a storage module stores too many files, the storage layer system can migrate at least part of the files in this storage module to other storage modules (such as storage modules that store fewer files). Of course, the storage layer system can also migrate files in a storage module to other storage modules in other cases, which is not limited by the embodiments of the present application.

[0187] Step B: The storage layer system obtains the second version number, which is the sum of the latest system version number at the time when the first file migration is completed and 1.

[0188] The second version number is a version number generated after the first file migration is completed, that is, a version number obtained by adding 1 to the latest system version number at the time when the first file migration is completed.

[0189] After obtaining the second version number, the storage layer system can determine a mapping relationship between the second version number and the identifier of the third storage module, the mapping relationship being used to indicate that the first file is stored in the third storage module at a version corresponding to the second version number.

[0190] Step C: The storage layer system indexes the second leaf node in the target tree according to the file identifier of the first file, the second leaf node being a leaf node containing the file identifier of the first file.

[0191] In some embodiments, the operation of step C can include the following steps a to b:

[0192] Step a: In the case where the target tree is an n-ary tree, the storage layer system determines index information in the target tree according to the file identifier of the first file, n, and the height of the target tree.

[0193] The index information in the target tree is information needed when indexing a leaf node in the target tree.

[0194] Optionally, assuming that the height of the target tree is m, m being an integer greater than or equal to 2. Then the index information in the target tree can include index information of each layer in the target tree, that is, can include index information of each layer in the 2nd layer to the mth layer.

[0195] The index information of the ith layer is used to indicate the ordering serial number of the node to be indexed in the ith layer among all child nodes of its parent node, where i is greater than or equal to 2 and less than or equal to m.

[0196] That is, after indexing a node in the i-1th layer, it is necessary to continue to index a node in the ith layer, in which case, the next node can be indexed from all child nodes of the node in the i-1th layer according to the index information of the ith layer. For example, the index information of the 2nd layer is 0, then after indexing a node in the 1st layer, that is, the root node, a child node with an ordering serial number of 0 among all child nodes of the root node can be taken as the next node to be indexed, the next node being a node in the 2nd layer.

[0197] As an example, the storage layer system can obtain the file identifier in the last leaf node in the target tree according to the pointer of the last leaf node in the target tree maintained in the memory, add 1 to the target value corresponding to the file identifier to obtain the number of leaf nodes of the target tree. After learning the number of leaf nodes of the target tree and the branching number n of the target tree, the height m of the target tree can be determined.

[0198] The target value corresponding to a file identifier is the rank number of the file identified by the file identifier among all files. The rank number starts from 0 and increases by 1. That is, the rank number can be 0, 1, 2, 3, ….

[0199] The file identifier and the target value have a corresponding relationship. When the storage layer system adds a new file, the rank number of the new file among all files can be used as the target value corresponding to the file identifier of the new file.

[0200] Since the file identifiers are globally ordered, for any file identifier, the corresponding target value can be determined.

[0201] For example, if the file identifiers start from 0 and increase by 1, that is, the file identifiers are 0, 1, 2, 3, …, then a file identifier and its corresponding target value are the same.

[0202] For another example, if the file identifiers start from 0 and increase by a fixed value, and the fixed value is j, j is an integer greater than or equal to 2, then the file identifiers are 0, j, 2j, 3j, … In this case, the target value corresponding to a file identifier is the value obtained by dividing the file identifier by j.

[0203] Of course, the file identifiers can also exist in other forms of order, such as a, b, c, … In this case, the file identifiers can be converted into corresponding target values according to certain rules, such as the target value corresponding to the file identifier a is 0, the target value corresponding to the file identifier b is 1, and the target value corresponding to the file identifier c is 2.

[0204] Optionally, assuming that the height of the target tree is m, m is an integer greater than or equal to 2. The operation of step a can be: the storage layer system adds a read lock to the root node of the target tree. Determine the target value corresponding to the file identifier of the first file, which is the rank number of the first file among all files. Let i be m, and let the specified parameter be the target value corresponding to the file identifier of the first file. Divide the specified parameter by n to obtain a quotient and a remainder, and use the remainder as the index information of the i-th layer. If i is not 2, let i = i-1, let the specified parameter be the quotient, and re-execute the above step of dividing the specified parameter by n to obtain a quotient and a remainder, and subsequent steps until i is 2; if i is 2, index the root node, and release the read lock of the root node after indexing the root node.

[0205] It should be noted that the embodiment of the present application needs to add a read lock to the root node in the process of determining the index information in the target tree. This is because if the case of adding a new root node occurs in the process of determining the index information in the target tree, the error case that the index information determined according to the old root node starts to index from the new root node will occur. Therefore, the embodiment of the present application adds a read lock to the root node before determining the index information in the target tree, so as to avoid modifying the root node by other threads. After the index information in the target tree is determined, the root node is indexed first, and the read lock of the root node is released after the root node is indexed. In this way, the root node starting the indexing process is consistent with the root node when the index information is determined, thereby ensuring the correctness of the indexing process.

[0206] In addition, the embodiment of the present application adds a read lock to the root node in the process of determining the index information in the target tree, which does not affect other data indexing processes. That is, different processes of indexing leaf nodes through file identifiers can be completely concurrent.

[0207] It should be noted that the specified parameter is divided by n to obtain a quotient and a remainder. The quotient represents the ordering sequence number of the parent node of the node to be indexed in the i-th layer in all nodes in the i-1-th layer. The remainder is the ordering sequence number of the node to be indexed in the i-th layer in all child nodes of the parent node. Accordingly, after indexing the nodes in the i-1-th layer, the nodes in the i-th layer can be continuously indexed.

[0208] In the embodiment of the present application, the index information in the target tree is determined from the index information in the m-th layer, and then the index information in the m-1-th layer, the index information in the m-2-th layer, and so on are determined until the index information in the second layer is determined. When indexing, after the root node is indexed, the index information in the second layer, the index information in the third layer, and so on are used to index down layer by layer to the leaf node according to the index information in the m-th layer.

[0209] Step b: The storage layer system indexes the second leaf node in the target tree according to the index information in the target tree.

[0210] It should be noted that the embodiment of the present application can directly determine the index path from the root node to the second leaf node according to the index information in the target tree, and in this process, the node data does not need to be read, so the process of indexing the second leaf node according to the index information in the target tree does not need to add a read lock.

[0211] In the embodiment of the present application, only m-1 times of calculation is needed to determine the index information in the target tree, and the index path can be directly determined according to the index information in the target tree. In this way, the complexity of the indexing process is low, and the indexing efficiency is high.

[0212] The process of determining the index information in the target tree is illustrated below in combination with FIG. 5.

[0213] As shown in the (b) diagram of FIG. 5, the target tree is a binary tree, and the height of the target tree is 4.

[0214] Suppose the file identifier of the first file corresponds to a target value of 6, as shown in the (a) diagram of FIG. 5, first, the index information of the 4th layer is determined, specifically, the specified parameter is 6, 6 is divided by 2, and the quotient is 3 and the remainder is 0, so the index information of the 4th layer is determined to be 0.

[0215] Then, the index information of the 3rd layer is determined, specifically, the specified parameter is 3, 3 is divided by 2, and the quotient is 1 and the remainder is 1, so the index information of the 3rd layer is determined to be 1.

[0216] Then, the index information of the 2nd layer is determined, specifically, the specified parameter is 1, 1 is divided by 2, and the quotient is 0 and the remainder is 1, so the index information of the 2nd layer is determined to be 1.

[0217] At this point, the index information of each layer from the 2nd layer to the 4th layer in the target tree is determined.

[0218] Subsequently, when indexing according to the index information of each layer from the 2nd layer to the 4th layer, as shown in the (b) diagram of FIG. 5, after indexing to the node of the 1st layer, i.e., the root node, according to the index information of the 2nd layer, it can be determined that the node of the 2nd layer to be indexed is the child node with the ordering sequence number of 1 among all child nodes of the root node. After indexing to a node of the 2nd layer, according to the index information of the 3rd layer, it can be determined that the node of the 3rd layer to be indexed is the child node with the ordering sequence number of 1 among all child nodes of the node of the 2nd layer. After indexing to a node of the 3rd layer, according to the index information of the 4th layer, it can be determined that the node of the 4th layer to be indexed is the child node with the ordering sequence number of 0 among all child nodes of the node of the 3rd layer. In this way, the corresponding leaf node, i.e., the second leaf node, is indexed.

[0219] Step D: The storage layer system adds a mapping relationship between the second version number and the identifier of the third storage module in the mapping table in the second leaf node.

[0220] In some embodiments, the mapping table is a doubly linked list. Each node in the doubly linked list contains a version number and a storage module identifier.

[0221] In this case, the operation of step D can be: the storage layer system adds a node containing the second version number and the identifier of the third storage module to the tail of the doubly linked list in the second leaf node.

[0222] It should be noted that, because the behavior of adding a node during the above file migration is based on the tail node of the double-linked list, the tail node (only a single node) needs a write lock to ensure its concurrent safety. That is, when a node needs to be added, the tail node is first given a write lock, then a node is added after the tail node, and then the write lock is released.

[0223] Step E: The storage layer system updates the system version number to the second version number.

[0224] After the storage layer system adds the mapping relationship between the second version number and the identifier of the third storage module in the mapping table in the second leaf node, the storage layer system can update the system version number to the second version number. In this way, the storage layer system updates the system version number after performing the file migration operation.

[0225] Next, the process of querying data by the storage layer system in the data processing method provided by the embodiments of the present application is described.

[0226] FIG. 6 is a flowchart of a data processing method provided by an embodiment of the present application, specifically a flowchart of a data query process. Referring to FIG. 6, the method can include the following steps 601 to 605:

[0227] Step 601: If the storage layer system receives a data query request, the target version number is obtained, the data query request carries a target key value, and the target version number is the latest system version number when the data query request is received. The update of the system version number is triggered by any one of a file storage operation or a file migration operation.

[0228] The target key value is the key value in the key-value pair to which the value value needs to be queried.

[0229] The data query request is used to request to query the value value in the key-value pair to which the target key value belongs.

[0230] The data query request can be sent by an external system. For example, it can be sent by a database system, a blockchain system, etc., which is not limited by the embodiments of the present application.

[0231] Step 602: The storage layer system determines the target leaf node indexed by the target key value in the target tree. The leaf node in the target tree includes a file identifier, the minimum key value and the maximum key value in all key-value pairs in the file identified by the file identifier, and a mapping table including the mapping relationship between the version number and the storage module identifier. The mapping table is used to indicate the storage module where the file identified by the file identifier is located at each version.

[0232] The target key value is greater than or equal to the minimum key value in the target leaf node and less than or equal to the maximum key value in the target leaf node.

[0233] For example, the target leaf node to which the target key value is indexed in the target tree can also be referred to as the target leaf node to which the target key value is distributed in the target tree.

[0234] Optionally, the storage layer system can index the target leaf node in the target tree according to the target key value by means of binary search, and of course, the target leaf node can also be indexed in the target tree by other means, which is not limited by the embodiments of the present application.

[0235] For example, if the target tree is an n-ary tree, the complexity of the indexing process satisfies O (logN) x log2branch. Wherein, the complexity of the tree indexing path is O (logN), and the complexity of specifying a child node in each node by means of binary search is a constant value log2branch, and the branch value is a fixed branch number constant n.

[0236] In some embodiments, the operation of step 603 can be that the storage layer system indexes the nodes from the root node down layer by layer in the target tree according to the target key value until the target leaf node is indexed.

[0237] As an example, in the process of indexing the nodes down layer by layer, a read lock is added to each node as it is indexed, the next node is indexed according to the node and the target key value, and the read lock of the node is released after the next node is indexed; the read lock of the target leaf node is released after the file identifier and the mapping table in the target leaf node are obtained.

[0238] Specifically, after the target version number is obtained, the storage layer system can first obtain the pointer of the root node in the memory, and then index the root node according to the pointer of the root node. A read lock is added to the root node after the root node is indexed.

[0239] Then, it is determined in which key range of the child node of the root node the target key value is, and this process is to index the next node. After it is determined that the target key value is in the key range of a child node of the root node, the child node is the next node indexed, and the read lock of the root node is released and a read lock is added to the latest indexed node.

[0240] If the node is a branch node, it can be determined in which key range of the child node of the node the target key value is, i.e. the next node is indexed. Until the indexed node is a leaf node.

[0241] If the node is a leaf node, it can be determined that the node is the target leaf node, and the read lock of the target leaf node is released after the file identifier and the mapping table in the target leaf node are obtained.

[0242] As another example, in the process of indexing nodes layer by layer downwards, each time a node is indexed, if the node is the last child node of all child nodes of its parent node, a read lock is added to the node, the next node is indexed according to the node and the target key value, and the read lock of the node is released after the next node is indexed; if the node is not the last child node of all child nodes of its parent node, no read lock is added to the node, and the next node is directly indexed according to the node and the target key value, and no read lock needs to be added to all nodes indexed subsequently in this case, so that the lock resource can be saved.

[0243] If a read lock is added to the target leaf node in the indexing process, the read lock of the target leaf node is released after the file identifier and the mapping table in the target leaf node are obtained.

[0244] It should be noted that the read lock and the read lock are shared locks, the read lock and the write lock are mutually exclusive locks, and the write lock and the write lock are mutually exclusive locks in the embodiments of the present application.

[0245] Optionally, the storage layer system in the embodiments of the present application can be a concurrent model of single-threaded writing and multi-threaded reading.

[0246] In this case, after one thread adds a read lock to a node, another thread cannot add a read lock to the node, and needs to wait until the read lock of the node is released before adding a read lock to the node and then continuing to operate (such as continuing to update the node).

[0247] After one thread adds a read lock to a node, another thread cannot add a write lock to the node, and needs to wait until the read lock of the node is released before adding a write lock to the node and then continuing to operate (such as continuing to update the node). Similarly, after one thread adds a write lock to a node, another thread cannot add a read lock to the node, and needs to wait until the write lock of the node is released before adding a read lock to the node and then continuing to operate (such as continuing to index the node).

[0248] In the embodiments of the present application, at most one lock conflict (also referred to as lock preemption) occurs in one tree indexing process and one tree updating process. That is, the tree structure of the target tree in the embodiments of the present application can avoid path-level lock conflicts to the greatest extent, thereby maximizing the system concurrency and reducing the main business process latency.

[0249] It should be noted that all data query behaviors in the embodiments of the present application can be concurrent, which is very friendly to the scene of concurrent reading of data in the system. For example, if the blockchain system supports concurrent execution of transactions, all behaviors of obtaining ledger data during concurrent execution of transactions can also be completely concurrent in the system.

[0250] In addition, the node insertion behavior in the embodiments of the present application cannot be concurrent, that is, the next update cannot be performed until the update of the target tree is completed.

[0251] Step 603: The storage layer system obtains the target storage module identifier from the mapping table in the target leaf node according to the target version number.

[0252] The storage module identified by the target storage module identifier is the storage module in which the file identified by the file identifier in the target leaf node is stored at the version corresponding to the target version number.

[0253] Since the target version number is the latest system version number of the storage layer system when the data query request is received, and the mapping relationship between the version number and the storage module identifier is saved in the mapping table in the target leaf node, the file in which the target key value is stored can be determined according to the target version number and the mapping table. storage module identifier (i.e., target storage module identifier) identified by the storage module identifier.

[0254] In some embodiments, the operation of step 603 can be: the storage layer system obtains, from the mapping table in the target leaf node, the storage module identifier to which the maximum version number in all version numbers less than or equal to the target version number is mapped as the target storage module identifier.

[0255] That is, if the target version number is recorded in the mapping table, it can be determined that the storage module identifier to which the target version number is mapped is the identifier of the storage module in which the file identified by the file identifier in the target leaf node is stored at the version corresponding to the target version number. Therefore, the storage module identifier can be used as the target storage module identifier.

[0256] If the target version number is not recorded in the mapping table, the maximum version number in all version numbers less than the target version number can be determined. The file identified by the file identifier in the target leaf node has not been migrated from the version corresponding to the maximum version number to the version corresponding to the target version number, so it can be determined that the storage module in which the file is stored at the version corresponding to the maximum version number is the storage module in which the file is stored at the version corresponding to the target version number. Therefore, the storage module identifier to which the maximum version number is mapped can be used as the target storage module identifier.

[0257] In some embodiments, when the mapping table is a double-linked list, the storage layer system can start searching from the tail of the double-linked list when obtaining the target storage module identifier from the double-linked list. This is because the latest system version number is usually used in data query, and thus searching from the tail of the double-linked list can quickly find the corresponding node (i.e., the node containing the maximum version number of all version numbers less than or equal to the target version number and the target storage module identifier).

[0258] It should be noted that, because the behavior of searching for a node is based on the tail node of the double-linked list, the tail node (only a single node) needs a read lock to ensure its concurrent safety. That is, when a node needs to be searched, a read lock is first added to the tail node, and then the tail node is searched forward, and the read lock is released after the corresponding node is found.

[0259] Step 604: The storage layer system obtains the value in the key-value pair to which the target key value belongs from the file identified by the file identifier in the file identified by the target storage module identifier.

[0260] Optionally, after obtaining the value, the storage layer system can return a data query result to the external system, and the data query result carries the value.

[0261] In some embodiments, the storage layer system can support concurrent reading, and thus multiple versions can coexist and run in the storage layer system. Therefore, control and release of multiple versions are needed.

[0262] In this case, after obtaining the target version number in step 601, the storage layer system can increase the reference count of the target version number by 1. After obtaining the value in the key-value pair to which the target key value belongs in step 604, the storage layer system can decrease the reference count of the target version number by 1.

[0263] As an example, the reference count has an initial value, and the initial value can be set to 0. That is, after updating the system version number, the storage layer system can set the reference count of the latest system version number to 0.

[0264] In this case, the reference count of the target version number is the total number of data query operations being performed under the version corresponding to the target version number.

[0265] In the process of managing multiple versions, the storage layer system needs to release historical versions (i.e., versions other than the version corresponding to the latest system version number), and thus the reference count can be used to determine whether a historical version meets the release condition.

[0266] Specifically, if the reference count of a historical version number (i.e. any version number except the latest system version number) is 0, it indicates that the storage layer system is not currently performing a data query operation based on the historical version corresponding to the historical version number, i.e. the historical version is not currently in use, and thus meets the release condition, and thus the historical version can be released.

[0267] If the reference count of a historical version number is greater than 0, it indicates that the storage layer system is currently performing one or more data query operations based on the historical version corresponding to the historical version number, i.e. the historical version is currently in use, and thus does not meet the release condition, and thus the historical version is not released.

[0268] As an example, the reference count has an initial value, and the initial value can be set to 1. That is, after the storage layer system updates the system version number each time, the storage layer system can set the reference count of the latest system version number to 1, and in this case can also decrease the reference count of the previous version number of the latest system version number by 1.

[0269] In this case, the reference count of the target version number can indicate whether the version corresponding to the target version number is in use. For example, if the reference count of the target version number is 1, it indicates that the version corresponding to the target version number can be the current latest version (i.e. the version corresponding to the latest system version number), or the version corresponding to the target version number can be a historical version and is currently performing a data query based on the historical version.

[0270] In the process of multi-version management by the storage layer system, version release is required, and thus the reference count can be used to determine whether a version meets the release condition.

[0271] Specifically, if the reference count of a version number (whether it is the latest system version number or a historical version number) is 0, it indicates that the storage layer system is not currently using the version corresponding to the version number, and thus the version meets the release condition, and thus the version can be released.

[0272] If the reference count of a version number is greater than 0, it indicates that the storage layer system is currently using the version corresponding to the version number, and thus the version does not meet the release condition, and thus the version is not released.

[0273] In some embodiments, the operation of the storage layer system to release the version can be: the storage layer system releases the version according to the version information of the next version of the version.

[0274] Specifically, if the version information of the next version indicates that only storage modules increase files, but no storage module decreases files, the release of the version is only to delete the reference count of the version number of the version. If the version information of the next version indicates that storage modules increase files and storage modules (which can be referred to as fourth storage modules) decrease files (which can be referred to as second files), the release of the version is to delete the second files from the fourth storage modules, and delete, from the mapping table in the leaf node of the file identifier of the second files in the target tree, all mapping relationships between version numbers and storage module identifiers, which are smaller than or equal to the version number of the version, and finally delete the reference count of the version number of the version.

[0275] In some embodiments, the mapping table in the leaf node is a doubly linked list, so that when the mapping relationship in the doubly linked list in the leaf node needs to be deleted during the release of a version, a node can be directly deleted from the head of the doubly linked list. This is because the deletion is usually of the oldest version.

[0276] It should be noted that, in the embodiments of the present application, the deletion scenario can be implemented based on the logic of lock-free deletion of the doubly linked list structure, because of the following two reasons:

[0277] 1. In the embodiments of the present application, the nodes corresponding to the same version in the doubly linked list satisfy the logical clock order of being created first and then deleted. Moreover, for a certain version, the release of the version is triggered again without any subsequent indexing behavior, that is, the deletion of the specified node in the doubly linked list in the leaf node is triggered. Therefore, the doubly linked list does not need additional concurrent control logic.

[0278] 2. The doubly linked list can realize that the node deletion behavior is completely controlled within the range of "only operating the node corresponding to the head pointer", so that the process can be lock-free.

[0279] Based on the above reasons, the doubly linked list in the embodiments of the present application can realize the maximum safety of concurrent read-write logic, thereby guaranteeing the throughput performance under high access frequency.

[0280] In the embodiments of the present application, the storage layer system stores files through a plurality of storage modules, and the plurality of storage modules can perform file migration, so that the data storage capacity and data storage flexibility can be improved. In this case, the file storage operation and the file migration operation will trigger version update. Therefore, the mapping table is further saved in the leaf node in the target tree, according to which the storage module where the file is located under each version can be determined, so that the required value can be obtained from the corresponding storage module according to the data query request, so that accurate data query can be realized.

[0281] Figure 7 is a structural schematic diagram of a data processing apparatus provided in an embodiment of the present application. The apparatus can be realized by software, hardware or a combination of both as part or all of a computer device, which can be the computer device shown in Figure 8 below. Referring to Figure 7, the apparatus comprises a first obtaining module 701, a determining module 702, a second obtaining module 703 and a third obtaining module 704.

[0282] The first obtaining module 701 is configured to, if a data query request is received, obtain a target version number, the data query request carrying a target key value, the target version number being the latest system version number at the time of receiving the data query request, the updating of the system version number being triggered by any one of a file storage operation or a file migration operation;

[0283] The determining module 702 is configured to determine a target leaf node to which the target key value is indexed in a target tree, the leaf node in the target tree comprising a file identifier, minimum and maximum key values in all key-value pairs in a file identified by the file identifier, and a mapping table, the mapping table comprising a mapping relationship between version numbers and storage module identifiers, the mapping table being used to indicate storage modules in which the file identified by the file identifier is located at various versions;

[0284] The second obtaining module 703 is configured to obtain a target storage module identifier from the mapping table in the target leaf node according to the target version number;

[0285] The third obtaining module 704 is configured to obtain a value value in a key-value pair to which the target key value belongs from a file identified by the file identifier in a storage module identified by the target storage module identifier.

[0286] In an embodiment of the present application, a storage layer system stores files through a plurality of storage modules, and file migration can be performed between the plurality of storage modules, so that data storage capacity and data storage flexibility can be improved. In this case, both the file storage operation and the file migration operation trigger version updating. Therefore, the mapping table is further saved in the leaf node in the target tree, according to which the storage module in which the file is located at various versions can be determined, so that the required value value can be obtained from the corresponding storage module after receiving the data query request, so that accurate data query can be ensured.

[0287] It should be noted that the data processing apparatus provided in the above embodiments is only used for example to divide the above functions into different function modules when processing data, and in actual application, the above functions can be completed by different function modules according to needs, that is, the internal structure of the apparatus is divided into different function modules to complete all or part of the above described functions.

[0288] The function units and modules in the above embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit, and the integrated unit can be realized in the form of hardware or in the form of a software function unit. In addition, the specific names of the function units and modules are only used for mutual distinction, and are not used to limit the protection scope of the embodiments of the application.

[0289] The data processing apparatus and the data processing method provided in the above embodiments belong to the same concept, and the specific working process of the units and modules in the above embodiments and the resulting technical effects can be referred to the method embodiment part, which will not be repeated here.

[0290] FIG. 8 is a structural schematic diagram of a computer device provided in the embodiments of the application. As shown in FIG. 8, the computer device 8 includes a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, and the processor 80 implements the steps in the data processing method in the above embodiments when executing the computer program 82.

[0291] The computer device 8 can be a general-purpose computer device or a special-purpose computer device. In a specific implementation, the computer device 8 can be the storage layer system described above, and specifically can be a desktop computer, a laptop computer, a network server, a palm computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device, and the embodiments of the application do not limit the type of the computer device 8. Those skilled in the art can understand that FIG. 8 is only an example of the computer device 8, and does not limit the computer device 8, and can include more or fewer components than the diagram, or combine certain components, or different components, such as can also include input and output devices, network access devices, etc.

[0292] The processor 80 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0293] The memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or a memory of the computer device 8 in some embodiments. The memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 8 in some other embodiments. Further, the memory 81 can include both the internal storage unit and the external storage device of the computer device 8. The memory 81 is used to store an operating system, application programs, a boot loader, data, and other programs, etc. The memory 81 can also be used to temporarily store data that has been output or is to be output.

[0294] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage equipment, etc. The computer readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0295] It should be understood that all or part of the steps of the above embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. The computer instructions can be stored in the computer readable storage medium mentioned above.

[0296] The above is the optional embodiment provided by the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. within the technical scope disclosed in the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that, The method is applied to a storage layer system, which includes multiple storage modules, each of which stores files, and the multiple storage modules are capable of file migration. The method includes: If a data query request is received, the target version number is obtained. The data query request carries a target key value. The target version number is the latest system version number when the data query request is received. The update of the system version number is triggered by either a file storage operation or a file migration operation. Determine the target leaf node indexed in the target tree for the target key value. The leaf node in the target tree contains a file identifier, the minimum and maximum key values ​​among all key-value pairs in the file identified by the file identifier, and a mapping table. The mapping table includes a mapping relationship between version numbers and storage module identifiers. The mapping table is used to indicate the storage module where the file identified by the file identifier is located under each version. The target storage module identifier is obtained from the mapping table in the target leaf node based on the target version number; From the file whose file identifier is the same as the file identifier in the target leaf node, obtain the value of the key-value pair to which the target key value belongs.

2. The method as described in claim 1, characterized in that, The method further includes: Receive a data storage request, the data storage request carrying a key-value pair to be stored, the key values ​​in the key-value pair being globally ordered; Based on the order of the key values ​​in the key-value pairs, a new file containing one or more key-value pairs is generated, and the new file is stored in a first storage module, which is one of the plurality of storage modules; wherein, the file identifiers of all files containing the key-value pairs are globally ordered; Obtain the first version number, which is the sum of the latest system version number when the new file is stored and 1; A first leaf node is added after the last leaf node in the target tree, and the target tree is updated according to the first leaf node. The first leaf node contains the file identifier of the new file, the minimum key value and the maximum key value among all key-value pairs in the new file, and a first mapping table. The first mapping table includes the mapping relationship between the first version number and the identifier of the first storage module. Update the system version number to the first version number.

3. The method as described in claim 1, characterized in that, The method further includes: The first file in the second storage module is migrated to the third storage module, wherein the second storage module and the third storage module are different storage modules among the plurality of storage modules; Obtain the second version number, which is the sum of the latest system version number when the first file migration is completed and 1; The second leaf node is indexed in the target tree based on the file identifier of the first file, and the second leaf node is a leaf node containing the file identifier of the first file; Add a mapping relationship between the second version number and the identifier of the third storage module to the mapping table in the second leaf node; Update the system version number to the second version number.

4. The method as described in claim 3, characterized in that, The step of obtaining the target storage module identifier from the mapping table in the target leaf node according to the target version number includes: From the mapping table in the target leaf node, obtain the storage module identifier mapped to the largest version number among all version numbers less than or equal to the target version number, and use it as the target storage module identifier.

5. The method as described in claim 1, characterized in that, The system version number is stored in memory, and the method further includes: After each update of the system version number, the version information corresponding to the updated system version number is stored in the disk. The version information includes the version number and version increment information. The version increment information is used to indicate the changes in the files in the storage module under the version number compared to the previous version.

6. The method as described in claim 5, characterized in that, The method further includes: Based on one or more version information stored in the disk, update the file distribution status stored in the disk. The file distribution status includes the correspondence between version number, storage module identifier and file identifier. The file distribution status is used to indicate the files stored in each of the multiple storage modules under the corresponding version.

7. The method as described in claim 6, characterized in that, The method further includes: After restarting, the version number in the latest version information stored on the disk is used as the system version number; The system version number is stored in the memory; The target tree is located in the memory, and the method further includes: After restarting, update the file distribution status stored on the disk according to all version information stored on the disk; The target tree is reconstructed in memory based on the updated file distribution status.

8. The method as described in claim 1, characterized in that, After obtaining the target version number, the process also includes: Increment the reference count of the target version number by 1; After obtaining the value from the key-value pair to which the target key belongs, the process further includes: Decrement the reference count of the target version number by 1; The method further includes: For any version number other than the latest system version number, if the reference count of the version number is 0, then the version corresponding to the version number is released.

9. The method according to any one of claims 1 to 8, characterized in that, The mapping table is a doubly linked list, and each node in the doubly linked list contains a version number and a storage module identifier.

10. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 9.

Images