Data processing method, computer device and storage medium

Abstract

The application discloses a data processing method, a computer device and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: if a data query request is received, a target version number is acquired, and the data query request carries a target key value. Then, a target leaf node indexed by the target key value in a target tree is determined, the leaf node in the target tree comprises a file identifier, minimum and maximum key values in all key-value pairs in a file identified by the file identifier, and a mapping table. A target storage module identifier is acquired from the mapping table in the target leaf node according to the target version number. A value in a key-value pair to which the target key value belongs is acquired from a file identified by the file identifier in the target leaf node in a storage module identified by the target storage module identifier. The application can accurately realize data query.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, a computer device, and a storage medium. Background Art

[0002] The multi-version concurrency control algorithm is a classic concurrency control algorithm in the database field and is crucial for implementing transaction concurrency and rollback. Its key concept is to preserve historical data versions and implement database concurrency control by managing multiple versions of data rows. This allows data to be displayed by comparing version numbers, ensuring transaction isolation without requiring locks when reading data.

[0003] However, the current multi-version concurrency control algorithm only considers the single-machine data storage scenario, and it is often difficult to ensure the accuracy of query data in distributed data storage scenarios. Summary of the Invention

[0004] This application provides a data processing method, computer device, and storage medium that can ensure the accuracy of query data. The technical solution is as follows:

[0005] In a first aspect, a data processing method is provided. The method is applied to a storage layer system, wherein the storage layer system includes multiple storage modules, each of the multiple storage modules is used to store files, and files can be migrated between the multiple storage modules. The method includes:

[0006] If a data query request is received, the target version number is obtained. The data query request carries a target key value. The target version number is the latest system version number when the data query request is received. The update of the system version number is triggered by any one of the file storage operation and file migration operation;

[0007] Determine a target leaf node indexed by the target key value in a target tree, where the leaf node in the target tree includes a file identifier, a minimum key value and a maximum key value among all key-value pairs in the file identified by the file identifier, and a mapping table, where the mapping table includes a mapping relationship between a version number and a storage module identifier, and the mapping table is used to indicate the storage module where the file identified by the file identifier is located under each version;

[0008] Obtaining a target storage module identifier from a mapping table in the target leaf node according to the target version number;

[0009] Obtain the value in the key-value pair to which the target key value belongs from the file whose file identifier in the storage module identified by the target storage module identifier is the file identifier in the target leaf node.

[0010] In the present application, the storage layer system stores files through multiple storage modules, and files can be migrated between the multiple storage modules, which can improve data storage capacity and data storage flexibility. In this case, both file storage operations and file migration operations will trigger version updates. For this reason, the embodiment of the present application also stores a mapping table in the leaf nodes of the target tree. According to the mapping table, the storage module where the file is located under each version can be determined, so that after receiving a data query request, the required value can be obtained from the corresponding storage module, thereby ensuring that data query can be accurately implemented.

[0011] In a second aspect, a data processing device is provided. The device is applied to a storage layer system, the storage layer system including multiple storage modules, each of the multiple storage modules is used to store files, and files can be migrated between the multiple storage modules. The device includes:

[0012] A first acquisition module is configured to acquire a target version number upon receiving a data query request, wherein the data query request carries a target key value, and the target version number is the latest system version number at the time the data query request is received. The update of the system version number is triggered by any one of a file storage operation and a file migration operation;

[0013] a determination module, configured to determine a target leaf node indexed by the target key value in a target tree, wherein the leaf node in the target tree includes a file identifier, a minimum key value and a maximum key value among all key-value pairs in the file identified by the file identifier, and a mapping table, wherein the mapping table includes a mapping relationship between a version number and a storage module identifier, and the mapping table is used to indicate the storage module where the file identified by the file identifier is located under each version;

[0014] A second acquisition module is configured to acquire a target storage module identifier from a mapping table in the target leaf node according to the target version number;

[0015] The third acquisition module is used to obtain the value in the key-value pair to which the target key value belongs from the file whose file identifier in the storage module identified by the target storage module identifier is the file identifier in the target leaf node.

[0016] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the data processing method described in the first aspect.

[0017] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program. The computer program, when executed by a processor, implements the data processing method of the first aspect.

[0018] In a fifth aspect, a computer program product containing instructions which, when executed on a computer, cause the computer to carry out the steps of the data processing method of the first aspect.

[0019] It can be understood that the beneficial effects of the second aspect, the third aspect, the fourth aspect and the fifth aspect can be referred to the related description of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0021] Figure 1 is a schematic diagram of a storage layer system provided by an embodiment of the present application;

[0022] Figure 2 is a schematic diagram of a target tree provided by an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a key-value pair provided by an embodiment of the present application;

[0024] Figure 4 is a schematic diagram of another key-value pair provided by an embodiment of the present application;

[0025] Figure 5 is a schematic diagram of index information provided by an embodiment of the present application;

[0026] Figure 6 is a flowchart of a data processing method provided by an embodiment of the present application;

[0027] Figure 7 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0028] Figure 8 is a structural schematic diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0030] The data processing method provided in the embodiment of the present application can be applied to a system with data processing requirements. In the embodiment of the present application, the system is referred to as a storage layer system, and the storage layer system is described below.

[0031] The embodiments of the present application are applied to a scenario where an external system performs data query based on a storage layer system. For example, the embodiments can be applied to a scenario where data query is performed based on a target tree.

[0032] For example, when executing a transaction in a block, a blockchain system may generate a key (i.e., a key value) that needs to be queried. In this case, the data processing method provided in the embodiment of the present application can be used to query the value corresponding to the key value based on the target tree.

[0033] The external system that the storage layer system interfaces with is a system with data query requirements and data storage requirements. For example, the external system that the storage layer system interfaces with can be a database system, a blockchain system, etc., which is not limited in this embodiment of the application.

[0034] The storage layer system is used to receive, process, and respond to various requests sent by external systems. For example, the requests sent by external systems may include data storage requests, data query requests, etc., which are not limited in this embodiment of the present application.

[0035] For example, Figure 1 Schematic diagram of a storage layer system provided by an embodiment of the present application. Figure 1 The storage layer system may include multiple storage modules, such as storage module 1, storage module 2, ..., storage module m.

[0036] Each storage module in the multiple storage modules is used to store files, and files can be migrated between the multiple storage modules.

[0037] The files in the storage module contain data sent by the external system that needs to be stored, such as key-value pairs, etc. For example, the storage module can be a disk or other device that can be used for data storage.

[0038] File migration refers to the storage layer system storing files in one storage module to another storage module. For example, the storage layer system can migrate files in storage module 1 to storage module 2 for storage.

[0039] Optionally, a storage module may correspond to an instance, and the instance may be bound to the path of the corresponding storage module, so that the storage layer system can store data to or read data from the corresponding storage module through the instance.

[0040] The storage module has an identifier, namely, a storage module identifier. A storage module identifier is used to identify a storage module.

[0041] In some embodiments, the storage layer system has a version number. Each version number corresponds to a version, and the version corresponding to a version number can represent a file distribution state within the storage layer system, where the file distribution state refers to which files are specifically stored in each storage module.

[0042] For example, storing a new file triggers a version update, and migrating a file between storage modules also triggers a version update. Version updates are strictly serialized within the storage layer system.

[0043] The storage layer system has a system version number, which can be a continuously increasing integer. The system version number is used to indicate the current latest version number of the storage layer system. For example, the system version number can be 0, 1, 2, 3, 4, ...

[0044] For example, the update of the system version number may be triggered by any one of a file storage operation and a file migration operation.

[0045] Specifically, when the storage layer system performs a file storage operation or a file migration operation, a new version number is generated. After the storage layer system completes the file storage operation or the file migration operation, the system version number can be updated to the new version number.

[0046] In some embodiments, the storage layer system can be connected to a database system. The database system can also be connected to other systems with data storage requirements, such as blockchain systems. For example, when a blockchain system needs to store data, it can first send the data to the database system, which then sends it to the storage layer system. The storage layer system can then store the data based on the target tree.

[0047] Alternatively, the storage layer system and the database system can be independent systems, in which case they can run on separate physical machines. Alternatively, the database system can include the storage layer system, in which case the database system includes a specific system that interfaces with the blockchain system and runs on the same physical machine as the storage layer system.

[0048] Next, the structure of the target tree provided in the embodiment of the present application is described.

[0049] The target tree is a tree structure. It consists of branch nodes and leaf nodes at the bottom of the target tree. A node with zero child nodes is a leaf node, while a node with one or more child nodes is a branch node. Both branch nodes and leaf nodes are logical nodes. The topmost branch node in the target tree is called the root node. A root node is a branch node in the target tree that has no parent node.

[0050] Optionally, the target tree may be an n-ary tree, that is, each branch node in the target tree has a maximum number of child nodes of n, where n is an integer greater than or equal to 2. Any branch node in an n-ary tree may have at most n child nodes. Furthermore, for branch nodes at the same level in an n-ary tree, a new branch node may only be added after the last branch node in that level has reached n child nodes.

[0051] The leaf node stores the file identifier, maximum key value, minimum key value and mapping table.

[0052] The file identifier is used to identify the file stored in the storage layer system. For example, the file identifier can be a file identity (ID), a file name, etc., which is not limited in the embodiment of the present application.

[0053] The minimum key value in a leaf node is the minimum key value in the file identified by the file identifier in the leaf node. The maximum key value in a leaf node is the maximum key value in the file identified by the file identifier in the leaf node.

[0054] In the embodiment of the present application, the leaf nodes in the target tree correspond to the files one by one, that is, a leaf node corresponds to a file identifier of a file and the minimum key value and the maximum key value in the file.

[0055] The mapping table in the leaf node is used to indicate the storage module where the file identified by the file identifier in the leaf node is located at each version. The mapping table may include a mapping relationship between version numbers and storage module identifiers. A version number and its mapped storage module identifier in the mapping relationship is used to indicate that the file is stored in the storage module identified by the storage module identifier at the version corresponding to the version number.

[0056] In some embodiments, the mapping table may be a doubly linked list, wherein each node in the doubly linked list includes a version number and a storage module identifier. In this case, the doubly linked list may maintain two node pointers, one being a head pointer for indicating the head node of the doubly linked list, and the other being a tail pointer for indicating the tail node of the doubly linked list.

[0057] In an embodiment of the present application, the storage layer system can store one or more files, and each file can contain one or more key-value pairs. The file identifiers of all files stored in the storage layer system are globally ordered, that is, the file identifier of any file exists in order in all files stored in the storage layer system. Moreover, the key values ​​in all key-value pairs contained in these files are also globally ordered, that is, the key values ​​in any key-value pair contained in any file exist in order in all key-value pairs in all files stored in the storage layer system.

[0058] Optionally, the global ordering of the key values ​​in the key-value pairs may be in increasing lexicographical order.

[0059] In this case, the minimum key value in a file refers to the key value that ranks first among all the key-value pairs in the file. The maximum key value in a file refers to the key value that ranks last among all the key-value pairs in the file.

[0060] It should be noted that in the computer field, lexicographic order can be used to indicate the size relationship between any two strings. The size relationship between the two strings is determined by the size relationship of the American Standard Code for Information Interchange (ASCII) codes of the characters in the strings from left to right.

[0061] Optionally, the global ordering of the file identifiers of the files may be in increasing lexicographical order.

[0062] For example, the file identifier of a file may start from 0 and increase by a preset value.

[0063] The preset value can be set in advance. For example, the preset value can be set to 1, 2, 3, etc., which is not limited in the embodiment of the present application.

[0064] In this case, the file identifiers of all files are an arithmetic progression. For example, assuming the preset value is 1, the file identifiers of the first file, the second file, the third file, ..., and the kth file can be generated in the order of: 0, 1, 2, ..., k-1.

[0065] In some embodiments, the storage layer system may maintain a global, continuously incrementing counter for generating file identifiers. In this case, each time the storage layer system generates a new file, the current value of the counter may be used as the file identifier of the new file. Thus, the files stored in the storage layer system are a batch of files with continuously incrementing file identifiers.

[0066] A branch node includes a list of child nodes, which includes the minimum and maximum key values ​​of each child node of the branch node. For example, the range from the minimum key value in a child node to the maximum key value in the child node can be called the key range of the child node.

[0067] The child nodes of a branch node can be branch nodes or leaf nodes at the next level. The minimum key value in a branch node is the minimum key value among all key values ​​in the list of child nodes in the branch node, that is, the key value that ranks highest among all key values ​​in the list of child nodes in the branch node; the maximum key value in a branch node is the maximum key value among all key values ​​in the list of child nodes in the branch node, that is, the key value that ranks lowest among all key values ​​in the list of child nodes in the branch node. The minimum and maximum key values ​​in a leaf node are the minimum and maximum key values ​​stored in the leaf node.

[0068] For example, Figure 2 This is a schematic diagram of a target tree provided in an embodiment of the present application. Figure 2 , the target tree is a binary tree, and the target tree includes multiple leaf nodes ( Figure 2 In this example, we take nodes n4, n5, and n6 as examples) and multiple branch nodes ( Figure 2 In the example of node n1, node n2, and node n3, the child nodes of node n1 include node n2 and node n3, the child nodes of node n2 include node n4 and node n5, and the child nodes of node n3 include node n6. In the embodiment of the present application, the topmost branch node in the target tree can be called the root node, for example, Figure 2 The node n1 in is the root node.

[0069] Figure 2 The leaf node n4 in the file stores the file identifier 0, the minimum key value 1, the maximum key value 3, and the mapping table 0. The leaf node n5 stores the file identifier 1, the minimum key value 4, the maximum key value 6, and the mapping table 1. The leaf node n6 stores the file identifier 2, the minimum key value 7, the maximum key value 9, and the mapping table 2.

[0070] Figure 2 The branch node n2 in the tree stores a child node list, which includes the minimum key value 1 and the maximum key value 3 in the leaf node n4, and the minimum key value 4 and the maximum key value 6 in the leaf node n5.

[0071] Figure 2 The branch node n3 in the tree stores a child node list, which includes the minimum key value 7 and the maximum key value 9 in the leaf node n6.

[0072] Figure 2 The branch node n1 in stores a child node list, which includes the minimum key value 1 and the maximum key value 6 in the branch node n2, and the minimum key value 7 and the maximum key value 9 in the branch node n3.

[0073] In an embodiment of the present application, the storage layer system can store the key-value pairs sent by the external system based on the target tree. For example, the target tree can also be called a solid branch tree (SolidBranchTree).

[0074] In some cases, the key values ​​in the key-value pairs sent by the external system to the storage layer system are globally ordered. The concept of global order has been explained above and will not be repeated here.

[0075] In some embodiments, the storage layer system can interface with a database system, which in turn can interface with a blockchain system. In this case, the blockchain system can send key-value pairs to be stored to the database system. Upon receiving these key-value pairs, the database system can send a data storage request containing these key-value pairs to the storage layer system.

[0076] As an example, the key-value pairs sent by the blockchain system to the database system for storage could be block data. Because block data is strictly organized by block number—that is, the key value in block data is the block number—the key values ​​in block data are naturally globally ordered. In this case, the database system can directly send the block data along with the data storage request to the storage layer system for storage.

[0077] As another example, the key-value pairs sent by the blockchain system to the database system for storage can be ledger data within a block. Ledger data within a block includes multiple key-value pairs. The key values ​​in multiple key-value pairs within the same block are different, but the key values ​​in key-value pairs in different blocks may be the same.

[0078] In this case, the database system can first process the ledger data. Specifically, it can introduce block numbers into the ledger data to construct globally ordered key-value pairs. These constructed key-value pairs are then sent to the storage layer system in data storage requests for storage. This ensures that the key values ​​in the key-value pairs carried in the data storage requests sent by the database system to the storage layer system are globally ordered.

[0079] Optionally, the key-value pair can be referred to as a target key-value pair in the embodiments of the present application, the key value in the target key-value pair includes a block number and a key value in a block identified by the block number (i.e. the key value in the key-value pair in the ledger data), and the value in the target key-value pair is a value in the block (i.e. the value in the key-value pair in the ledger data).

[0080] In some embodiments, the key value in the target key-value pair can be obtained by splicing the block number and the key value in the block, of course, the key value in the target key-value pair can also be obtained by other ways, and the embodiments of the present application do not limit this.

[0081] Since the block numbers are naturally globally ordered, and the key values in a block are different, the key value in the target key-value pair containing the block number and the key value in the block identified by the block number is also globally ordered.

[0082] For example, the block with the block number 00000001 includes a plurality of key-value pairs, for each key-value pair in the plurality of key-value pairs, the block number 00000001 can be added before the key value in the key-value pair to obtain a new key value.

[0083] For example, as shown in (a) of FIG. 1, Figure 3 As shown in (a) of FIG. 1, the block 1 with the block number 00000001 includes the key-value pairs bbbb:{value1...}, bggg:{value2...}, caaa:{value3...}, xaab:{value4...}. As shown in (b) of FIG. 1, Figure 3 As shown in (b) of FIG. 1, the block 2 with the block number 00000002 includes the key-value pairs aabb:{value5...}, bggg:{value6...}, bcde:{value7...}.

[0084] After the blockchain system sends these key-value pairs to the database system, the database system can introduce block number 00000001 into the key-value pairs in block 1 by appending 00000001 to the key value in each key-value pair in block 1, resulting in target key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, 00000001xaab:{value4...}. Block number 00000002 can be introduced into the key-value pairs in block 2 by appending 00000002 to the key value in each key-value pair in block 2, resulting in target key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, 00000002bcde:{value7...}. The key values ​​in the target key-value pairs obtained in this way are globally ordered.

[0085] It should be noted that the storage layer system in the embodiment of the present application is not limited to storing the key-value pairs in the above-mentioned blockchain system, but can also store other key-value pairs as long as the key values ​​in these key-value pairs are globally ordered.

[0086] The following describes the process of constructing the initial target tree for the storage layer system. The process may include the following steps (1) to (5):

[0087] (1) The storage layer system receives a data storage request, which carries a key-value pair to be stored, and the key values ​​in the key-value pair are globally ordered.

[0088] The data storage request is used to instruct the storage layer system to store the key-value pair to be stored. In some embodiments, the data storage request may be sent by an external system, such as a blockchain system or a database system.

[0089] In this case, the key values ​​in all key-value pairs sent by the external system to the storage layer system are in order. After the storage layer system stores these key-value pairs, the key values ​​in all key-value pairs stored in the storage layer system are also in order.

[0090] (2) The storage layer system generates a new file containing one or more key-value pairs according to the order of the key values ​​in the key-value pairs, and stores the new file in a first storage module, which is one of the multiple storage modules; wherein the file identifiers of all files containing the key-value pairs are globally ordered.

[0091] The first storage module is a storage module for storing new files. For example, the first storage module can be a default storage module among the multiple storage modules, or the first storage module can be a storage module randomly selected from the multiple storage modules, which is not limited in the embodiment of the present application.

[0092] The key values ​​in the key-value pairs received by the storage layer system are in order. In this case, the storage layer system generates a new file based on the order of the key values ​​in the key-value pairs, and the key values ​​in the key-value pairs in the new file are also in order.

[0093] After the storage layer system stores the new file in the first storage module, it can obtain the identifier of the first storage module.

[0094] In some embodiments, the operation of step (2) may be: each time the storage layer system receives a data storage request, it saves all key-value pairs carried in the data storage request in the memory according to the order of the key values ​​in all key-value pairs carried in the data storage request. If the difference between the total amount of data of one or more key-value pairs stored in sequence in the memory and the preset amount of data is less than or equal to the preset difference, the storage layer system generates a new file containing the one or more key-value pairs. The storage layer system stores the new file in the first storage module.

[0095] The preset data volume can be pre-set. For example, the preset data volume can be set to 8 megabytes (MB), 9 MB, 10 MB, etc., which is not limited in the embodiment of the present application.

[0096] Optionally, the preset data size can be set based on an integer multiple of the storage layer system's block size configuration. For example, if the block size is 4MB, the preset data size can be set to 4MB, 8MB, 12MB, and so on. The storage layer system's block size refers to the smallest unit used by the storage layer system to organize and manage data storage.

[0097] The preset difference value can be preset. For example, the preset difference value can be set to 100 kilobytes (KB), 200 KB, 300 KB, etc., which is not limited in the embodiment of the present application.

[0098] It should be noted that the data volume of each key-value pair is generally close, and the data volume of a key-value pair is usually within a predetermined range. In the embodiment of the present application, the preset data volume can be much larger than the data volume of a key-value pair. The preset difference can be close to the data volume of a key-value pair, for example, it can be equal to or slightly larger than the data volume of a key-value pair.

[0099] The storage layer system can store all key-value pairs carried by each data storage request received in the memory in order. In this case, if the total data volume of all key-value pairs currently stored in order in the memory is close to the preset data volume (that is, the difference between the preset data volume and the preset data volume is less than or equal to the preset difference), then all key-value pairs currently stored in the memory can be stored in the first storage module in the form of new files. In this way, batch storage of key-value pairs can be achieved, thereby improving data storage efficiency. Moreover, in this case, the file size of each file stored in the first storage module is basically the same, and is close to the preset data volume, which facilitates file management.

[0100] It should be noted that if the total data volume of all key-value pairs currently stored in the memory is less than the preset data volume, and the difference between the total data volume and the preset data volume is greater than the preset difference, it means that the total data volume of the key-value pairs currently stored in the memory is significantly different from the preset data volume, and the conditions for generating a new file are not met. In this case, the storage layer system can continue to receive data storage requests sent by the external system and store the key-value pairs carried in the data storage request in the memory until the difference between the total data volume of multiple key-value pairs stored in sequence in the memory and the preset data volume is less than or equal to the preset difference, and then generate a new file and store it.

[0101] The generated new file is stored in the first storage module, that is, the persistence of the new file is achieved. Moreover, after the new file is stored in the first storage module, one or more key-value pairs in the new file are deleted from the memory.

[0102] It should be noted that the key values ​​in the key-value pairs stored in memory are ordered. In this case, if a new file is generated based on the key-value pairs stored in memory, the key values ​​in all key-value pairs in the new file will also be ordered.

[0103] Another point to note is that since a file can contain one or more key-value pairs, the key-value pairs in a file can cover the data of one block or multiple blocks.

[0104] For example, Figure 4As shown, the storage layer system receives data storage request 1, which carries the key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. The storage layer system saves the key-value pairs carried in data storage request 1 to memory. The key-value pairs currently stored in memory include 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. Because the total data size of these key-value pairs is less than the preset data size, and the difference between the data size and the preset data size is greater than the preset difference, the condition for generating a new file is not met.

[0105] Afterwards, the storage layer system receives data storage request 2, which carries the key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, and 00000002bcde:{value7...}. The storage layer system stores the key-value pairs carried in data storage request 2 in memory in sequence. After storing the key-value pair 00000002bggg:{value6...} in memory, the storage layer system discovers that the difference between the total amount of data currently stored in the memory for the key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} and the preset amount of data is less than or equal to the preset difference, satisfying the condition for generating a new file. Therefore, a new file containing the key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} is generated and stored in the first storage module. In addition, the storage layer system saves the key-value pair 00000002bcde:{value7...} carried in the data storage request 2 in the memory.

[0106] Later, the storage layer system receives Data Storage Request 3, which contains the key-value pairs 00000003bcbb:{value1...}. The storage layer system sequentially stores the key-value pairs in memory. The key-value pairs currently stored in memory are 00000002bcde:{value7...} and 00000003bcbb:{value1...}. Because the total amount of data in these key-value pairs is less than the preset amount, and the difference between the total amount and the preset amount is greater than the preset difference, the conditions for creating a new file are not met. The storage layer system then waits for the next data storage request.

[0107] (3) The storage layer system generates an initial version number.

[0108] The initial version number can be preset. For example, the initial version number can be 0.

[0109] Since the storage layer system stores the file for the first time, the storage layer system can generate an initial version number.

[0110] After the storage layer system stores the new file in the first storage module and generates an initial version number, it can determine the mapping relationship between the initial version number and the identifier of the first storage module, which is used to indicate that the new file is stored in the first storage module under the version corresponding to the initial version number.

[0111] (4) The storage layer system generates a new leaf node and constructs an initial target tree based on the new leaf node. The new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file, and a mapping table. The mapping table includes a mapping relationship between the initial version number and the identifier of the first storage module.

[0112] After generating the new leaf node, the storage layer system can create a new branch node as the parent node of the new leaf node. This branch node stores a list of child nodes, including the minimum and maximum key values ​​of the new leaf node. Since the number of leaf nodes is currently 1, this branch node is the root node, thus completing the initial target tree construction.

[0113] In some embodiments, a branch node in the target tree may include a child node list. The child node list in a branch node stores node information of each child node of the branch node.

[0114] For example, the node information of a node may include a pointer to the node, a minimum key value, and a maximum key value in the node.

[0115] A node pointer is used to point to the memory space where the node is located.

[0116] In this case, the child node list in a branch node includes node information of all child nodes of the branch node, and the node information of all child nodes exists in sequence in the child node list.

[0117] In some cases, after the storage layer system completes the initial construction of the target tree, it can also save the pointer to the root node of the target tree and the pointer to the last leaf node in the target tree in memory. This allows the storage layer system to quickly index to the root node of the target tree based on the pointer to the root node maintained in memory, and quickly index to the last leaf node in the target tree based on the pointer to the leaf node maintained in memory.

[0118] (5) The storage layer system creates a system version number and updates the system version number to the initial version number.

[0119] After building the initial target tree, the storage layer system can create a system version number in memory and update the system version number to the initial version number.

[0120] In some embodiments, for any data storage request, if all key-value pairs contained in the data storage request have been stored in the storage module as files, the target tree has been constructed or updated accordingly, and the system version number has been updated, then the storage layer system may return a data storage success message to the external system that sent the data storage request. For example, the data storage success message may include the file identifier of the file containing the key-value pairs in the data storage request, the identifier of the storage module containing the key-value pairs, etc., although this is not limited in the present embodiments.

[0121] In some embodiments, each time the storage layer system updates the system version number, the version information of the version corresponding to the updated system version number can be stored on disk. The version information includes the version number and version increment information. The version increment information is used to indicate the file changes in the storage module under the version corresponding to the version number compared to the previous version. That is, the version increment information is used to indicate which storage module has reduced which files and which storage module has added which files under this version compared to the previous version. It should be noted that the "files" described in the embodiments of the present application all refer to the files containing key-value pairs involved in the embodiments of the present application.

[0122] It should be noted that the reduction of files in the storage module in the embodiment of the present application is caused by file migration, while the increase of files in the storage module may be caused by file storage or file migration.

[0123] Optionally, the version information may further include the largest file identifier under the version (ie, the file identifier that is ranked the lowest).

[0124] In this case, the version information of the version corresponding to the updated system version number can be stored into the disk every time the system version number is updated, and the storage layer system stores the version information of each version into the disk in sequence, i.e., the one or more version information stored in the disk is in order.

[0125] In some embodiments, the storage layer system can update the file distribution state stored in the disk according to the one or more version information stored in the disk, where the file distribution state contains the correspondence between the version number, the storage module identifier and the file identifier. The file distribution state is used to indicate the files stored in each storage module in the plurality of storage modules under the corresponding version. Wherein, the file identifier corresponding to any one of the storage module identifiers in the correspondence between the storage module identifier and the file identifier is the file identifier of each file stored in the storage module identified by the storage module identifier.

[0126] Since the one or more version information stored in the disk is in order, the storage layer system updates the file distribution state stored in the disk in sequence, i.e., sequentially according to the version order.

[0127] When the storage layer system updates the file distribution state stored in the disk according to a version information, the version number in the file distribution state can be updated to the version number in the version information, and the correspondence between the storage module identifier and the file identifier in the file distribution state can be updated according to the version increment information in the version information. Then, after updating the file distribution state according to the version information of a version, the updated file distribution state is used to indicate the files stored in each storage module in the plurality of storage modules under the version.

[0128] After the storage layer system updates the file distribution state according to a version information, the version information can be deleted from the disk.

[0129] In some cases, the file distribution state can be updated asynchronously by a thread. That is, the storage layer system records version information every time a version update occurs, and then there are multiple version information after multiple version updates. There can be a thread that updates the file distribution state according to the recorded multiple version information in sequence.

[0130] For example, Table 1 below shows multiple version information and the file distribution state updated in sequence according to each version information in the multiple version information.

[0131] Specifically, when the storage layer system stores a new file 3 in the storage module 1 to obtain a version corresponding to version number 3, the version information of the version is "version number: 3, version increment information: the storage module 1 adds the file 3". According to the version information, the file distribution state can be updated to "version number: 3, the corresponding relationship between the storage module identifier and the file identifier: storage module 1: {file 2, file 3}, storage module 2: {file 1}".

[0132] Then, when the storage layer system stores a new file 4 in the storage module 1 to obtain a version corresponding to version number 4, the version information of the version is "version number: 4, version increment information: the storage module 1 adds the file 4". According to the version information, the file distribution state can be updated to "version number: 4, the corresponding relationship between the storage module identifier and the file identifier: storage module 1: {file 2, file 3, file 4}, storage module 2: {file 1}".

[0133] Then, when the storage layer system stores a new file 4 in the storage module 1 to obtain a version corresponding to version number 4, the version information of the version is "version number: 4, version increment information: the storage module 1 adds the file 4". According to the version information, the file distribution state can be updated to "version number: 4, the corresponding relationship between the storage module identifier and the file identifier: storage module 1: {file 2, file 3, file 4}, storage module 2: {file 1}".

[0134] Table 1

[0135]

[0136] The present application only takes Table 1 as an example to exemplarily illustrate the version information and the file distribution state, and Table 1 does not limit the present application.

[0137] As an example, the target tree in the present application does not need to be persisted, that is, does not need to be stored in a disk, but can be saved in a memory, so that storage resources can be saved.

[0138] In this case, if the storage layer system is restarted due to a crash or other reasons, then after the restart, the storage layer system can obtain the version number in the latest version information stored on the disk as the system version number and save the system version number in the memory. In addition, after the restart, the storage layer system updates the file distribution status stored on the disk based on all the version information stored on the disk, and rebuilds the target tree in the memory based on the updated file distribution status. In this case, the updated file distribution status is the file distribution status under the latest version (i.e., the version corresponding to the system version number), and the file distribution status can indicate the files stored in each storage module in the multiple storage modules under the latest version.

[0139] Since the file identifiers are globally ordered and the file identifiers correspond one-to-one to the leaf nodes, all the leaf nodes of the last layer in the target tree can be constructed based on all the file identifiers in the file distribution state and the minimum key value and the maximum key value in the file identified by each file identifier, and a mapping table in each leaf node can be constructed based on the version number in the file distribution state (i.e. the version number of the latest version) and the identifier of the storage module where each file is located. At this time, the mapping table only contains the mapping relationship between the version number and the corresponding storage module identifier. Afterwards, the branch nodes are constructed layer by layer upwards according to the leaf nodes of the last layer until the root node is constructed, and the target tree is reconstructed. After the target tree is reconstructed, the pointer to the root node in the constructed target tree and the pointer to the last leaf node in the target tree are maintained in the memory.

[0140] It should be noted that after the target tree is constructed, the subsequent storage layer system can perform data processing based on the target tree, such as data storage, file migration, data query, etc.

[0141] Next, the process of storing data and updating the target tree in the storage layer system in the data processing method provided in the embodiment of the present application is described. The process may include the following steps A to E:

[0142] Step A: The storage layer system receives a data storage request, which carries a key-value pair to be stored, and the key values ​​in the key-value pair are globally ordered.

[0143] The operation of step A is similar to that of the above step (1), and will not be described in detail in the embodiment of the present application.

[0144] Step B: The storage layer system generates a new file containing one or more key-value pairs according to the order of the key values ​​in the key-value pairs, and stores the new file in a first storage module, which is one of the multiple storage modules; wherein the file identifiers of all files containing the key-value pairs are globally ordered.

[0145] The operation of step B is similar to that of the above step (2), and will not be described in detail in the embodiment of the present application.

[0146] Step C: The storage layer system obtains a first version number, which is the sum of the latest system version number and 1 when the new file is stored.

[0147] The first version number is the version number generated after the storage layer system stores the new file. This number is calculated by adding 1 to the latest system version number at the time the new file is stored. For example, if the latest system version number at the time the new file is stored is 5, then the first version number is 6. In this case, version updates are strictly serialized within the storage layer system.

[0148] After obtaining the first version number, the storage layer system may determine a mapping relationship between the first version number and the identifier of the first storage module, where the mapping relationship indicates that the new file is stored in the first storage module under the version corresponding to the first version number.

[0149] Step D: The storage layer system adds a first leaf node after the last leaf node in the target tree, and updates the target tree based on the first leaf node. The first leaf node contains the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file, and the first mapping table. The first mapping table includes the mapping relationship between the first version number and the identifier of the first storage module.

[0150] The first leaf node is a leaf node newly added after the last leaf node in the target tree, that is, a new leaf node.

[0151] Optionally, the storage layer system may first obtain the pointer of the last leaf node in the target tree in the memory, and then add the first leaf node after the leaf node according to the pointer of the leaf node.

[0152] For example, the last leaf node in the target tree may be located at the rightmost position of the last layer of the target tree. In this case, the leaf nodes are inserted from left to right. Then, the storage layer system adds the first leaf node after the last leaf node in the target tree, that is, inserts the first leaf node at the rightmost position of the last layer of the target tree.

[0153] Alternatively, the last leaf node in the target tree can be located at the leftmost position of the last layer of the target tree. In this case, the leaf nodes are inserted from right to left. The storage layer system then adds the first leaf node after the last leaf node in the target tree, i.e., inserts the first leaf node at the leftmost position of the last layer of the target tree.

[0154] In an embodiment of the present application, when storing data, there is no need to index in the target tree, and a new file can be directly generated for storage. Based on this, a new leaf node can be directly added to the target tree to update the target tree, thereby improving data storage efficiency.

[0155] It should be noted that there is only one update logic for the target tree in the embodiment of the present application, that is, inserting a new leaf node after the last leaf node of the entire tree (such as the rightmost or leftmost side of the last layer). The traditional tree structure adopts the logic of random insertion, which needs to first index downward from the root node to the leaf node, then insert and store data, and then backtrack to update or split the parent node. Compared with the random insertion logic in the traditional tree structure, the update logic of the target tree in the embodiment of the present application can avoid the indexing process from the root node to the leaf node, thereby improving data storage efficiency.

[0156] One thing that needs to be explained is that since the file identifiers are globally ordered and each leaf node stores the file identifier of a file, the file identifiers in all leaf nodes in the last layer of the target tree exist in order from left to right or from right to left.

[0157] Another point to note is that in the embodiments of the present application, the shape of the entire target tree is relatively fixed. Except for the nodes in one path from the root node to the last leaf node (i.e., the rightmost path or the leftmost path), the remaining nodes are static nodes. Static nodes are nodes that will not be modified and have a fixed overall shape.

[0158] In some embodiments, the target tree is an n-ary tree, where n is an integer greater than or equal to 2. When the target tree is an n-ary tree, node splitting is not required during the process of updating the target tree according to the first leaf node, thereby improving update efficiency.

[0159] Optionally, the operation of the storage layer system updating the target tree based on the first leaf node can be: the storage layer system updates the nodes in the target tree layer by layer upward according to the first leaf node until the root node is updated. In the process of updating the nodes layer by layer upward, if an existing node in the target tree needs to be updated, a write lock is added to the existing node, and the write lock is released after the update is completed; in the process of updating the nodes layer by layer upward, if a new node is added to the target tree, there is no need to add a write lock to the new node.

[0160] It should be noted that in this embodiment of the application, when updating an existing node in the target tree, a write lock is first placed on the existing node. In this case, if a query operation on the existing node occurs during the update process, it must wait until the update of the existing node is completed and the write lock is released before the query operation can be performed. This ensures the accuracy of the data query.

[0161] In the embodiment of the present application, new nodes added during the target tree update process do not require a write lock. This is because the new nodes are not visible to the outside world before the target tree update is completed, so the data query process naturally does not pass through the new nodes, and there is no need for a write lock.

[0162] In some embodiments, the operation of the storage layer system updating nodes layer by layer upward in the target tree according to the first leaf node may include the following steps a to f.

[0163] Step a: Set the first leaf node as the current node.

[0164] Step b: Determine whether the current node is a new node or an existing node in the target tree. If the current node is a new node, execute steps c to e below; if the current node is an existing node, execute step f below.

[0165] Step c: If the current node is a new node, determine whether the current node can be a child node of the last node in the upper layer. If the current node can be a child node of the last node in the upper layer, execute step d below; if the current node cannot be a child node of the last node in the upper layer, execute step e below.

[0166] For example, when the target tree is an n-ary tree, if the number of child nodes of the last node in the upper layer is less than n, the current node can be the child node of the last node in the upper layer; if the number of child nodes of the last node in the upper layer is equal to n, the current node cannot be the child node of the last node in the upper layer.

[0167] Step d: If the current node can be the child node of the last node in the upper layer, then the last node in the upper layer is used as the parent node of the current node, a write lock is added to the parent node of the current node, and the parent node of the current node is updated. After the update, the write lock of the parent node of the current node is released, and the parent node of the current node is made the current node, and step b and subsequent steps are re-executed.

[0168] If the current node can be a child node of the last node in the upper layer, then the last node in the upper layer can be updated as the parent node of the current node. The update here is to add the node information of the current node to the child node list of the last node in the upper layer.

[0169] Step e: If the current node cannot be the child node of the last node in the upper layer, then add a new node in the upper layer as the parent node of the current node, make the parent node of the current node the current node, and re-execute step b and subsequent steps.

[0170] If the current node cannot be the child node of the last node in the upper layer, then a new node can be added after the last node in the upper layer as the parent node of the current node. In this case, the parent node of the current node includes a child node list, and the child node list includes the node information of the current node.

[0171] Step f: If the current node is the original node, add a write lock to the parent node of the current node, update the parent node of the current node, release the write lock of the parent node of the current node after the update, make the parent node of the current node the current node, and re-execute step b and subsequent steps.

[0172] If the current node is an existing node, then the parent node of the current node already exists in the target tree, so the parent node of the current node can be directly updated. The update here is to update the node information of the current node included in the child node list of the parent node of the current node.

[0173] It should be noted that after re-determining the current node in step d, step e or step f, before re-executing step b, it may be determined whether the current node is the original root node of the target tree.

[0174] If the current node is not the original root node of the target tree and is not a node at the same level as the original root node, then re-execute step b and subsequent steps.

[0175] If the current node is the original root node of the target tree, it is determined that the update of the target tree is completed.

[0176] If the current node is not the original root node of the target tree and is on the same level as the original root node, a new root node is added to the upper level as the parent node of the original root node and the current node. The new root node contains a child node list containing the node information of the original root node and the current node. This completes the update of the target tree. In this case, the height of the target tree increases by one level.

[0177] It should be noted that after the target tree is updated, the pointer of the root node in the target tree maintained in the memory and the pointer of the last leaf node in the target tree can be updated.

[0178] Updating the root node pointer in memory cannot be performed concurrently with retrieving the root node pointer in memory. That is, if the root node pointer in memory is being updated, the operation to retrieve the root node pointer in memory must wait until the update is complete. If the root node pointer in memory is being retrieved, the operation to update the root node pointer in memory must be performed after the root node pointer is successfully retrieved. This is because updating the root node pointer may cause the root node pointer to be in a transitional state, such as being null. In this case, retrieving the root node pointer will result in an error.

[0179] Step E: The storage layer system updates the system version number to the first version number.

[0180] After the target tree is updated, the storage layer system can update the system version number to the first version number. In this way, the storage layer system updates the system version number after performing the file storage operation.

[0181] In some embodiments, after updating the system version number, the storage layer system may store the version information of the version corresponding to the updated system version number in a disk.

[0182] Next, the process of file migration in the storage layer system in the data processing method provided in the embodiment of the present application is described. The process may include the following steps A to E:

[0183] Step A: The storage layer system migrates the first file in the second storage module to the third storage module, where the second storage module and the third storage module are different storage modules among the multiple storage modules.

[0184] The second storage module is a storage module that stores the first file before migration.

[0185] The first file is a file to be migrated. The number of the first file can be one or more.

[0186] The third storage module is the storage module to which the first file is to be migrated.

[0187] In some cases, if a storage module stores too many files, the storage layer system may migrate at least a portion of the files in this storage module to another storage module (e.g., a storage module that stores fewer files). Of course, the storage layer system may also migrate files from one storage module to another storage module in other situations, and this embodiment of the application is not limited to this.

[0188] Step B: The storage layer system obtains a second version number, which is the sum of the latest system version number when the first file migration is completed and 1.

[0189] The second version number is a version number generated after the first file migration is completed, that is, a version number obtained by adding 1 to the latest system version number when the first file migration is completed.

[0190] After obtaining the second version number, the storage layer system can determine a mapping relationship between the second version number and the identifier of the third storage module, where the mapping relationship indicates that the first file is stored in the third storage module under the version corresponding to the second version number.

[0191] Step C: The storage layer system indexes a second leaf node in the target tree according to the file identifier of the first file, where the second leaf node is a leaf node that includes the file identifier of the first file.

[0192] In some embodiments, the operation of step C may include the following steps a to b:

[0193] Step a: When the target tree is an n-ary tree, the storage layer system determines index information in the target tree according to the file identifier of the first file, n and the tree height of the target tree.

[0194] The index information in the target tree is information required when indexing leaf nodes in the target tree.

[0195] Optionally, assuming that the height of the target tree is m, where m is an integer greater than or equal to 2, the index information in the target tree may include index information of each layer in the target tree, that is, index information of each layer from the 2nd layer to the mth layer.

[0196] The index information of the i-th layer is used to indicate the sorting order of the node of the i-th layer to be indexed among all child nodes of its parent node, where i is greater than or equal to 2 and less than or equal to m.

[0197] That is, after indexing a node at the i-1th level, you need to continue indexing nodes at the i-th level. In this case, you can index the next node from all the child nodes of the node at the i-1th level based on the index information at the i-th level. For example, if the index information at the second level is 0, then after indexing a node at the first level, that is, the root node, you can use the child node with the sorting sequence number of 0 among all the child nodes of the root node as the next node to index. This next node is a node at the second level.

[0198] As an example, the storage layer system can obtain the file identifier in the last leaf node of the target tree maintained in memory based on the pointer to the leaf node, and then add 1 to the target value corresponding to the file identifier to obtain the number of leaf nodes in the target tree. Given the number of leaf nodes in the target tree and the number of forks n in the target tree, the height m of the target tree can be determined.

[0199] The target value corresponding to a file identifier is the sorting sequence number of the file identified by this file identifier among all files. The sorting sequence number starts from 0 and increases by 1. That is, the sorting sequence number can be 0, 1, 2, 3, ...

[0200] There is a corresponding relationship between the file identifier and the target value. Every time a new file is added to the storage layer system, the sorting sequence number of the new file among all files can be used as the target value corresponding to the file identifier of the new file.

[0201] Since the file identifiers are globally ordered, the corresponding target value can be determined for any file identifier.

[0202] For example, if the file identifier starts from 0 and increases by 1, that is, the file identifier is 0, 1, 2, 3, ..., then a file identifier is the same as its corresponding target value.

[0203] For another example, if the file identifier starts at 0 and increases by a fixed value, assuming that the fixed value is j, and j is an integer greater than or equal to 2, then the file identifiers are 0, j, 2j, 3j, .... In this case, the target value corresponding to a file identifier is the value obtained by dividing the file identifier by j.

[0204] Of course, file identifiers can also exist in other orderly forms, such as file identifiers can be a, b, c,... etc. In this case, the file identifiers can be converted into corresponding target values ​​according to certain rules, such as the target value corresponding to file identifier a is 0, the target value corresponding to file identifier b is 1, the target value corresponding to file identifier c is 2, etc.

[0205] Optionally, assume that the height of the target tree is m, where m is an integer greater than or equal to 2. The operation of step a may be: the storage layer system adds a read lock to the root node of the target tree. Determine the target value corresponding to the file identifier of the first file, and the target value corresponding to the file identifier of the first file is the sorting number of the first file among all files. Let i be m, and let the specified parameter be the target value corresponding to the file identifier of the first file. Divide the specified parameter by n to obtain the quotient and remainder, and use the remainder as the index information of the i-th layer. If i is not 2, let i=i-1, let the specified parameter be the quotient, and re-execute the above-mentioned steps of dividing the specified parameter by n to obtain the quotient and remainder and subsequent steps until i is 2; if i is 2, index the root node, and release the read lock of the root node after indexing the root node.

[0206] It should be noted that, in the process of determining the index information in the target tree, the embodiment of the present application needs to add a read lock to the root node. This is because if a new root node is added in the process of determining the index information in the target tree, then the index information determined based on the old root node will be indexed from the new root node. For this reason, before determining the index information in the target tree, the embodiment of the present application first adds a read lock to the root node to prevent other threads from modifying the root node. After determining the index information in the target tree, the root node is indexed first, and then the read lock of the root node is released after the root node is indexed. In this way, it can be ensured that the root node at the beginning of the indexing process is consistent with the root node when the index information is determined, thereby ensuring the correctness of the indexing process.

[0207] In addition, the embodiment of the present application adds a read lock to the root node during the process of determining the index information of the target tree, which will not affect other data indexing processes. In other words, different processes of indexing leaf nodes through file identifiers can be completely concurrent.

[0208] It should be noted that the specified parameter is divided by n to obtain the quotient and remainder. The quotient represents the sorting order of the parent node of the node at level i to be indexed among all nodes at level i-1. The remainder represents the sorting order of the node at level i to be indexed among all the child nodes of its parent node. Based on this, after indexing the node at level i-1, you can continue to index the node at level i.

[0209] In the embodiment of the present application, when determining the index information in the target tree, the index information of the mth layer is determined first, and then the index information of the m-1th layer, the index information of the m-2th layer, and so on are determined in sequence until the index information of the second layer is determined. When indexing, after indexing to the root node, the index information of the second layer, the index information of the third layer, and so on are indexed downward to the leaf nodes layer by layer according to the index information of the mth layer.

[0210] Step b: The storage layer system indexes the second leaf node in the target tree according to the index information in the target tree.

[0211] It should be noted that the embodiment of the present application can directly determine the index path from the root node to the second leaf node based on the index information in the target tree. There is no need to read the node data in this process. Therefore, the process of indexing the second leaf node based on the index information in the target tree does not require a read lock.

[0212] In the embodiment of the present application, only m-1 calculations are required to determine the index information in the target tree, and the index path can be directly determined based on the index information in the target tree. In this way, the indexing process is less complex and the indexing efficiency is higher.

[0213] The following combination Figure 5 The process of determining the index information in the target tree is described with an example.

[0214] like Figure 5 As shown in Figure (b), the target tree is a binary tree, and the height of the target tree is 4.

[0215] Assume that the target value corresponding to the file identifier of the first file is 6, such as Figure 5 As shown in Figure (a), first, determine the index information of the 4th layer. Specifically, let the specified parameter be 6, divide 6 by 2, and the quotient is 3 and the remainder is 0, then determine that the index information of the 4th layer is 0.

[0216] Then, continue to determine the index information of the third layer. Specifically, let the specified parameter be 3, divide 3 by 2, and the quotient is 1 and the remainder is 1, then determine that the index information of the third layer is 1.

[0217] Afterwards, the index information of the second layer is determined. Specifically, let the specified parameter be 1, divide 1 by 2, and the quotient is 0 and the remainder is 1, then the index information of the second layer is determined to be 1.

[0218] At this point, the index information of each layer from the 2nd layer to the 4th layer in the target tree has been determined.

[0219] When indexing is performed based on the index information of each layer from the 2nd to the 4th layer, Figure 5 As shown in Figure (b), after indexing the node of the first layer, that is, the root node, based on the index information of the second layer, it can be determined that the node of the second layer to be indexed is the child node with a sorting sequence number of 1 among all the child nodes of the root node. After indexing a node of the second layer, based on the index information of the third layer, it can be determined that the node of the third layer to be indexed is the child node with a sorting sequence number of 1 among all the child nodes of this node of the second layer. After indexing a node of the third layer, based on the index information of the fourth layer, it can be determined that the node of the fourth layer to be indexed is the child node with a sorting sequence number of 0 among all the child nodes of this node of the third layer. In this way, the corresponding leaf node, that is, the second leaf node, is indexed.

[0220] Step D: The storage layer system adds a mapping relationship between the second version number and the identifier of the third storage module in the mapping table in the second leaf node.

[0221] In some embodiments, the mapping table is a doubly linked list, wherein each node in the doubly linked list includes a version number and a storage module identifier.

[0222] In this case, the operation of step D may be: the storage layer system adds a node to the end of the bidirectional linked list in the second leaf node, and the node includes the second version number and the identifier of the third storage module.

[0223] It should be noted that because the addition of nodes during file migration is based on the tail node of the doubly linked list, the tail node (which is just a single node) requires a write lock to ensure concurrency safety. In other words, when adding a node, first apply a write lock to the tail node, then add a node after the tail node, and then release the write lock.

[0224] Step E: The storage layer system updates the system version number to the second version number.

[0225] After the storage layer system adds the mapping relationship between the second version number and the identifier of the third storage module in the mapping table in the second leaf node, the system version number can be updated to the second version number. In this way, the storage layer system updates the system version number after performing the file migration operation.

[0226] Next, the process of data query by the storage layer system in the data processing method provided in the embodiment of the present application is explained.

[0227] Figure 6 This is a flow chart of a data processing method provided by an embodiment of the present application, specifically a flow chart of a data query process. Figure 6 The method may include steps 601 to 605:

[0228] Step 601: If the storage layer system receives a data query request, it obtains the target version number. The data query request carries a target key value. The target version number is the latest system version number when the data query request is received. The update of the system version number is triggered by any one of the file storage operation and file migration operation.

[0229] The target key value is the key value in the key-value pair to which the value to be queried belongs.

[0230] The data query request is used to query the value in the key-value pair to which the target key value belongs.

[0231] The data query request may be sent by an external system, such as a database system, a blockchain system, etc., and this embodiment of the application does not limit this.

[0232] Step 602: The storage layer system determines the target leaf node indexed by the target key value in the target tree. The leaf node in the target tree contains the file identifier, the minimum key value and the maximum key value in all key-value pairs in the file identified by the file identifier, and a mapping table. The mapping table includes a mapping relationship between the version number and the storage module identifier. The mapping table is used to indicate the storage module where the file identified by the file identifier is located under each version.

[0233] The target key value is greater than or equal to the minimum key value in the target leaf node and less than or equal to the maximum key value in the target leaf node.

[0234] For example, the target leaf node to which the target key value is indexed in the target tree can also be referred to as the target leaf node to which the target key value is distributed in the target tree.

[0235] Optionally, the storage layer system can index the target leaf node in the target tree according to the target key value by means of binary search, and of course, the target leaf node can also be indexed in the target tree by other means, which is not limited by the embodiments of the present application.

[0236] For example, if the target tree is an n-ary tree, the complexity of the indexing process satisfies O(logN) x log2branch. Wherein, the complexity of the tree indexing path is O(logN), and the complexity of specifying a child node in each node by means of binary search is a constant value log2branch, and the branch value is a fixed branch number constant n.

[0237] In some embodiments, the operation of step 603 can be that the storage layer system indexes nodes layer by layer from the root node downward in the target tree according to the target key value until the target leaf node is indexed.

[0238] As an example, in the process of indexing nodes layer by layer downward, a read lock is added to each node after it is indexed, the next node is indexed according to the node and the target key value, and the read lock of the node is released after the next node is indexed; the read lock of the target leaf node is released after the file identifier and the mapping table in the target leaf node are obtained.

[0239] Specifically, after the target version number is obtained, the storage layer system can first obtain the pointer of the root node in the memory, and then index the root node according to the pointer of the root node. A read lock is added to the root node after it is indexed.

[0240] Then, it is determined in which key range of the child node of the root node the target key value is located, and this process is to index the next node. After it is determined that the target key value is located in the key range of a child node of the root node, the child node is the next node indexed, and the read lock of the root node is released and a read lock is added to the latest indexed node.

[0241] If the node is a branch node, it can be determined in which key range of the child node of the node the target key value is located, that is, the next node is continuously indexed. Until the indexed node is a leaf node.

[0242] If this node is a leaf node, then it can be determined that this node is the target leaf node. Subsequently, after obtaining the file identifier and mapping table in the target leaf node, the read lock of the target leaf node is released.

[0243] As another example, in the process of indexing nodes downward layer by layer, each time a node is indexed, if this node is the last child node among all the child nodes of its parent node, a read lock is added to this node, and the next node is indexed based on this node and the target key value. After indexing the next node, the read lock of this node is released; if this node is not the last child node among all the child nodes of its parent node, a read lock is not added to this node, and the next node is directly indexed based on this node and the target key value. In addition, in this case, there is no need to add a read lock to all subsequently indexed nodes, which can save lock resources.

[0244] Among them, if a read lock is added to the target leaf node during the indexing process, the read lock of the target leaf node is released after the file identifier and mapping table in the target leaf node are obtained.

[0245] It should be noted that in the embodiment of the present application, the read lock and the read lock are shared locks, the read lock and the write lock are mutually exclusive locks, and the write lock and the write lock are mutually exclusive locks.

[0246] Optionally, the storage layer system in the embodiment of the present application can be a concurrent model of single-threaded writing and multi-threaded reading.

[0247] In this case, after one thread adds a read lock to a node, it does not affect another thread adding a read lock to the node.

[0248] After one thread has read-locked a node, another thread cannot write-lock the same node. It must wait until the read lock of the node is released before it can write-lock the node and continue operations (such as continuing to update the node). Similarly, after one thread has written-locked a node, another thread cannot read-lock the node. It must wait until the write lock of the node is released before it can read-lock the node and continue operations (such as continuing to index the node).

[0249] In the embodiment of the present application, a tree indexing process and a tree updating process will only cause a maximum of one lock conflict (also known as lock preemption). In other words, the tree structure of the target tree in the embodiment of the present application can avoid path-level lock conflicts to the greatest extent possible, thereby maximizing system concurrency and reducing the latency of the main business process.

[0250] It should be noted that all data query behaviors in the embodiments of this application can be concurrent, which is very friendly to scenarios where data is read concurrently within the system. For example, if the blockchain system supports concurrent transaction execution, then all behaviors that obtain ledger data during concurrent transaction execution can also be fully concurrent within the system.

[0251] In addition, the node insertion behavior in the embodiment of the present application cannot be concurrent, that is, the next update can only be performed after the target tree is updated once.

[0252] Step 603: The storage layer system obtains the target storage module identifier from the mapping table in the target leaf node according to the target version number.

[0253] The storage module identified by the target storage module identifier is the storage module where the file identified by the file identifier in the target leaf node is located under the version corresponding to the target version number.

[0254] Since the target version number is the latest system version number of the storage layer system when it receives the data query request, and the mapping relationship between the version number and the storage module identifier is stored in the mapping table in the target leaf node, it is possible to determine based on the target version number and the mapping table which storage module identifier (i.e., the target storage module identifier) ​​in which the file containing the target key value is stored.

[0255] In some embodiments, the operation of step 603 may be: the storage layer system obtains the storage module identifier mapped to the maximum version number among all version numbers less than or equal to the target version number from the mapping table in the target leaf node as the target storage module identifier.

[0256] That is to say, if the target version number is recorded in the mapping table, then it can be determined that the storage module identifier to which the target version number is mapped is the identifier of the storage module where the file identified by the file identifier in the target leaf node is located under the version corresponding to the target version number, and thus the storage module identifier can be used as the target storage module identifier.

[0257] If the target version number is not recorded in the mapping table, then the maximum version number among all version numbers smaller than the target version number in the mapping table can be determined. The file identified by the file identifier in the target leaf node has not been migrated from the version corresponding to the maximum version number to the version corresponding to the target version number. Therefore, it can be determined that the storage module where the file is located under the version corresponding to the maximum version number is the storage module where the file is located under the version corresponding to the target version number. Therefore, the storage module identifier to which the maximum version number is mapped can be used as the target storage module identifier.

[0258] In some embodiments, when the mapping table is a doubly linked list, the storage layer system can start searching from the end of the doubly linked list when obtaining the target storage module identifier from the doubly linked list. This is because data queries tend to use the latest system version number, so starting the search from the end of the doubly linked list will more quickly find the corresponding node (i.e., the node containing the maximum version number among all version numbers less than or equal to the target version number and the target storage module identifier).

[0259] It should be noted that because the node search behavior during data query is based on the tail node of the doubly linked list, the tail node (just a single node) requires a read lock to ensure its concurrency safety. In other words, when searching for a node, a read lock is first placed on the tail node, and then the search starts from the tail node and moves forward. The read lock is released after the corresponding node is found.

[0260] Step 604: The storage layer system obtains the value in the key-value pair to which the target key value belongs from the file whose file identifier in the storage module identified by the target storage module identifier is the file identifier in the target leaf node.

[0261] Optionally, after obtaining the value, the storage layer system can return a data query result to the external system, and the data query result carries the value.

[0262] In some embodiments, the storage layer system can support concurrent reads, thereby allowing multiple versions to coexist and run within the storage layer system. To this end, multiple versions need to be controlled and released.

[0263] In this case, after obtaining the target version number in step 601, the storage layer system can increase the reference count of the target version number by 1. Thereafter, after obtaining the value in the key-value pair to which the target key value belongs in step 604, the storage layer system can decrease the reference count of the target version number by 1.

[0264] As an example, the reference count has an initial value, and the initial value can be set to 0. That is, each time the storage layer system updates the system version number, the reference count of the latest system version number can be set to 0.

[0265] In this case, the reference count of the target version number is the total number of data query operations in progress under the version corresponding to the target version number.

[0266] In the process of multi-version management, the storage layer system needs to release historical versions (i.e., versions other than the version corresponding to the latest system version number). Therefore, reference counting can be used to determine whether a historical version meets the release conditions.

[0267] Specifically, if the reference count of a historical version number (i.e. any version number except the latest system version number) is 0, it indicates that the storage layer system is not currently performing a data query operation based on the historical version corresponding to the historical version number, i.e. the historical version is not currently in use, thus meeting the release condition, and therefore the historical version can be released.

[0268] If the reference count of a historical version number is greater than 0, it indicates that the storage layer system is currently performing one or more data query operations based on the historical version corresponding to the historical version number, i.e. the historical version is currently in use, thus not meeting the release condition, and therefore the historical version is not released.

[0269] As an example, the reference count has an initial value, and the initial value can be set to 1. That is, after the storage layer system updates the system version number each time, the storage layer system can set the reference count of the latest system version number to 1, and in this case can also decrease the reference count of the previous version number of the latest system version number by 1.

[0270] In this case, the reference count of the target version number can indicate whether the version corresponding to the target version number is in use. For example, if the reference count of the target version number is 1, it indicates that the version corresponding to the target version number can be the current latest version (i.e. the version corresponding to the latest system version number), or the version corresponding to the target version number can be a historical version and is currently performing a data query based on the historical version.

[0271] In the process of multi-version management by the storage layer system, version release is required, and therefore the reference count can be used to determine whether a version meets the release condition.

[0272] Specifically, if the reference count of a version number (whether it is the latest system version number or a historical version number) is 0, it indicates that the storage layer system is not currently using the version corresponding to the version number, and the version meets the release condition, and therefore the version can be released.

[0273] If the reference count of a version number is greater than 0, it indicates that the storage layer system is currently using the version corresponding to the version number, and the version does not meet the release condition, and therefore the version is not released.

[0274] In some embodiments, the operation of the storage layer system to release the version can be: the storage layer system releases the version according to the version information of the next version of the version.

[0275] Specifically, if the version information of the next version indicates that only storage modules increase files, but no storage module decreases files, the release of the version is only to delete the reference count of the version number of the version. If the version information of the next version indicates that storage modules increase files and storage modules (which can be referred to as fourth storage modules) decrease files (which can be referred to as second files), the release of the version is to delete the second files from the fourth storage modules, and delete, from the mapping table in the leaf node of the file identifier of the second files in the target tree, all mapping relationships between version numbers and storage module identifiers, which are smaller than or equal to the version number of the version, and finally delete the reference count of the version number of the version.

[0276] In some embodiments, the mapping table in the leaf node is a double-linked list, and therefore, when it is necessary to delete a mapping relationship in the double-linked list in the leaf node during the release of a version, a node can be directly deleted from the head of the double-linked list. This is because the deletion is usually of the oldest version.

[0277] It should be noted that, in the embodiments of the present application, the deletion scenario can be implemented based on the double-linked list structure to achieve a lock-free deletion logic, because of the following two reasons:

[0278] 1. In the embodiments of the present application, the nodes corresponding to the same version in the double-linked list satisfy the logical clock order of being created first and then deleted. Moreover, for a certain version, the release of the version is triggered again without any subsequent indexing behavior, that is, the deletion of the specified node in the double-linked list in the leaf node is triggered. Therefore, the double-linked list does not need additional concurrent control logic.

[0279] 2. The double-linked list can achieve that the node deletion behavior is completely controlled within the range of "only operating the node corresponding to the head pointer", so that the process can be lock-free.

[0280] Based on the above reasons, the double-linked list in the embodiments of the present application can achieve the maximum degree of safe concurrent read-write logic, thereby ensuring the throughput performance under high access frequency.

[0281] In the embodiments of the present application, the storage layer system stores files through a plurality of storage modules, and the plurality of storage modules can perform file migration, so that the data storage capacity and data storage flexibility can be improved. In this case, the file storage operation and the file migration operation will both trigger version update. Therefore, the mapping table is further saved in the leaf node in the target tree, according to which the storage modules in which the files are located under each version can be determined, so that the required value can be obtained from the corresponding storage module according to the data query request, so that accurate data query can be ensured.

[0282] Figure 7 This is a structural diagram of a data processing device provided by an embodiment of the present application. The device can be implemented as part or all of a computer device by software, hardware, or a combination of both. The computer device can be Figure 8 Computer equipment shown. Figure 7 The device includes: a first acquisition module 701, a determination module 702, a second acquisition module 703, and a third acquisition module 704.

[0283] The first acquisition module 701 is configured to acquire a target version number upon receiving a data query request. The data query request carries a target key value. The target version number is the latest system version number at the time the data query request is received. The update of the system version number is triggered by any one of a file storage operation and a file migration operation.

[0284] Determination module 702, for determining a target leaf node indexed by a target key value in a target tree, where the leaf node in the target tree includes a file identifier, the minimum key value and the maximum key value of all key-value pairs in the file identified by the file identifier, and a mapping table, where the mapping table includes a mapping relationship between a version number and a storage module identifier, and the mapping table is used to indicate the storage module where the file identified by the file identifier is located under each version;

[0285] The second acquisition module 703 is used to obtain the target storage module identifier from the mapping table in the target leaf node according to the target version number;

[0286] The third acquisition module 704 is configured to acquire the value in the key-value pair to which the target key value belongs from the file whose file identifier in the storage module identified by the target storage module identifier is the file identifier in the target leaf node.

[0287] In an embodiment of the present application, the storage layer system stores files through multiple storage modules, and files can be migrated between the multiple storage modules, thereby improving data storage capacity and data storage flexibility. In this case, both file storage operations and file migration operations will trigger version updates. For this reason, the embodiment of the present application also stores a mapping table in the leaf nodes of the target tree. According to the mapping table, the storage module where the file is located under each version can be determined, so that after receiving a data query request, the required value can be obtained from the corresponding storage module, thereby ensuring that data query can be accurately implemented.

[0288] It should be noted that: when processing data, the data processing device provided in the above embodiment only uses the division of the above-mentioned functional modules as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0289] The functional units and modules in the above embodiments may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The above integrated units may be implemented in the form of hardware or software functional units. In addition, the specific names of the functional units and modules are only for the purpose of distinguishing them from each other and are not intended to limit the scope of protection of the embodiments of this application.

[0290] The data processing device and data processing method embodiments provided in the above embodiments belong to the same concept. The specific working processes and technical effects brought about by the units and modules in the above embodiments can be found in the method embodiment part and will not be repeated here.

[0291] Figure 8 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 8 As shown, the computer device 8 includes: a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80. When the processor 80 executes the computer program 82, the steps of the data processing method in the above embodiment are implemented.

[0292] The computer device 8 can be a general-purpose computer device or a dedicated computer device. In a specific implementation, the computer device 8 can be the above-mentioned storage layer system, specifically a desktop computer, a portable computer, a network server, a PDA, a mobile phone, a tablet computer, a wireless terminal device, a communication device or an embedded device. The embodiment of the present application does not limit the type of the computer device 8. Those skilled in the art will understand that Figure 8 This is merely an example of the computer device 8 and does not constitute a limitation on the computer device 8 . The computer device 8 may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0293] The processor 80 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or can also be any conventional processor.

[0294] The memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or a memory of the computer device 8 in some embodiments. The memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 8 in some other embodiments. Further, the memory 81 can include both the internal storage unit and the external storage device of the computer device 8. The memory 81 is used to store an operating system, application programs, a boot loader, data, and other programs, etc. The memory 81 can also be used to temporarily store data that has been output or is to be output.

[0295] If the integrated unit is implemented as a software functional unit and sold or used as a standalone product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to the camera / terminal device, recording medium, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device. The computer-readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0296] It should be understood that all or part of the steps for implementing the above embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the steps may be implemented in the form of a computer program product. The computer program product may include one or more computer instructions. The computer instructions may be stored in the above-mentioned computer-readable storage medium.

[0297] The above are optional embodiments provided for this application and are not intended to limit this application. Any modifications, equivalent replacements, improvements, etc. made within the technical scope disclosed in this application should be included in the scope of protection of this application.

Claims

1. A data processing method, characterized in that: The method is applied to a storage layer system, the storage layer system including multiple storage modules, each of the multiple storage modules is used to store files, and files can be migrated between the multiple storage modules. The method includes: If a data query request is received, the target version number is obtained. The data query request carries a target key value. The target version number is the latest system version number when the data query request is received. The update of the system version number is triggered by any one of the file storage operation and file migration operation; Determine a target leaf node indexed by the target key value in a target tree, where the leaf node in the target tree includes a file identifier, a minimum key value and a maximum key value among all key-value pairs in the file identified by the file identifier, and a mapping table, where the mapping table includes a mapping relationship between a version number and a storage module identifier, and the mapping table is used to indicate the storage module where the file identified by the file identifier is located under each version; Obtaining a target storage module identifier from a mapping table in the target leaf node according to the target version number; Obtain the value in the key-value pair to which the target key value belongs from the file whose file identifier in the storage module identified by the target storage module identifier is the file identifier in the target leaf node.

2. The method according to claim 1, wherein The method further comprises: Receive a data storage request, the data storage request carrying a key-value pair to be stored, wherein the key values ​​in the key-value pair are globally ordered; generating a new file containing one or more key-value pairs according to the order of the key values ​​in the key-value pairs, and storing the new file in a first storage module, where the first storage module is one of the plurality of storage modules; wherein the file identifiers of all files containing the key-value pairs are globally ordered; Obtaining a first version number, where the first version number is the sum of the latest system version number when the new file is stored and 1; Adding a first leaf node after the last leaf node in the target tree, and updating the target tree according to the first leaf node, wherein the first leaf node includes a file identifier of the new file, a minimum key value and a maximum key value in all key-value pairs in the new file, and a first mapping table, wherein the first mapping table includes a mapping relationship between the first version number and the identifier of the first storage module; Update the system version number to the first version number.

3. The method according to claim 1, wherein The method further comprises: Migrating a first file in a second storage module to a third storage module, where the second storage module and the third storage module are different storage modules among the multiple storage modules; Obtain a second version number, where the second version number is the sum of the latest system version number when the first file migration is completed and 1; indexing a second leaf node in the target tree according to the file identifier of the first file, where the second leaf node is a leaf node including the file identifier of the first file; Adding a mapping relationship between the second version number and the identifier of the third storage module in the mapping table in the second leaf node; Update the system version number to the second version number.

4. The method according to claim 3, wherein The step of obtaining a target storage module identifier from a mapping table in the target leaf node according to the target version number includes: From the mapping table in the target leaf node, a storage module identifier to which the maximum version number among all version numbers less than or equal to the target version number is mapped is obtained as the target storage module identifier.

5. The method according to claim 1, wherein The system version number is stored in the memory, and the method further includes: Each time the system version number is updated, the version information of the version corresponding to the updated system version number is stored on the disk. The version information includes the version number and version increment information. The version increment information is used to indicate the file changes in the storage module under the version corresponding to the version number compared to the previous version.

6. The method according to claim 5, wherein The method further comprises: According to one or more version information stored in the disk, the file distribution status stored in the disk is updated, the file distribution status includes the correspondence between the version number, the storage module identifier and the file identifier, and the file distribution status is used to indicate the files stored in each storage module in the multiple storage modules under the corresponding version.

7. The method according to claim 6, wherein The method further comprises: After restarting, obtaining the version number in the latest version information stored in the disk as the system version number; Storing the system version number in the memory; The target tree is located in the memory, and the method further includes: After restarting, updating the file distribution status stored in the disk according to all version information stored in the disk; The target tree is rebuilt in the memory according to the updated file distribution status.

8. The method according to claim 1, wherein After obtaining the target version number, the following steps are also included: Increment the reference count of the target version number by 1; After obtaining the value in the key-value pair to which the target key value belongs, the method further includes: Decrement the reference count of the target version number by 1; The method further comprises: For any version number other than the latest system version number, if the reference count of the version number is 0, the version corresponding to the version number is released.

9. The method according to any one of claims 1 to 8, characterized in that: The mapping table is a bidirectional linked list, and each node in the bidirectional linked list includes a version number and a storage module identifier.

10. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the method according to any one of claims 1 to 9 when executed by the processor.

11. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images