File migration method and apparatus, device, and storage medium

Abstract

The present application belongs to the technical field of computers, and discloses a file migration method and apparatus, a device, and a storage medium. The method comprises: on the basis of a data query request, updating a first correspondence and a second correspondence of one storage module among a plurality of storage modules, the first correspondence comprising a hit count of each file in the storage module, and the second correspondence comprising the number of files in the storage module of which hit counts are within each of a plurality of preset hit count intervals; on the basis of a first correspondence and a second correspondence of a second storage module, determining a file migration list for the second storage module every preset duration; and migrating files indicated by the file migration list from the second storage module to a third storage module. According to the present application, cold and hot data separation is implemented while distributed file storage is implemented, so that hot data is retained in a storage module having high read / write performance as much as possible. Thus, the read / write efficiency of the hot data is improved, and the read / write delay of the hot data is reduced.

Description

File migration method, device, equipment and storage medium

[0001] The present application claims priority from the Chinese patent application No. 202410853231.0, filed on June 27, 2024, and entitled "File migration method, device, equipment and storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of computers, and particularly relates to a file migration method, device, equipment and storage medium. BACKGROUND

[0003] At present, all data in a database equally share operating system resources, which can cause hot data and cold data in a business system layer to be indistinguishable at the operating system layer, and the storage and access of the hot data and the cold data have almost the same delay. In the related art, an additional cache system is added in the business system layer to solve the read-write delay problem of the hot data. However, such a solution can cause an increase in the amount of physical memory, and once the data volume of the hot data far exceeds the size of the physical memory, the cache system itself can also frequently trigger the exchange operation of the memory page and the disk data in the operating system, which can cause the cache effect to decrease significantly, and even the cache itself can also bring additional system processing delay. TECHNICAL PROBLEM

[0004] The present application provides a file migration method, device, equipment and storage medium, which can reduce the read-write delay of hot data. TECHNICAL SOLUTION

[0005] In a first aspect, a file migration method is provided. The method is applied to a first device in a storage layer system. The storage layer system includes the first device, one or more second devices, and a plurality of storage modules. Any of the plurality of storage modules is located in the first device or the second device. The plurality of storage modules is sorted in order of read-write performance from high to low. The method includes: generating, by the first device, a new file according to a received data storage request, and storing the new file to a first storage module. The first storage module is a storage module sorted first in the plurality of storage modules. Updating, by the first device, a first correspondence relationship and a second correspondence relationship of a storage module in the plurality of storage modules according to a received data query request. The first correspondence relationship includes a hit count of each file in the storage module. The second correspondence relationship includes a quantity of files in the storage module that have a hit count in a plurality of preset hit count intervals. Determining, by the first device, a file migration list of a second storage module at a preset time interval according to the first correspondence relationship and the second correspondence relationship of the second storage module. The second storage module is any of the plurality of storage modules except a storage module sorted last. The file migration list is used to indicate files that need to be migrated from the second storage module to a third storage module. The third storage module is a storage module sorted next to the second storage module in the plurality of storage modules. Migrating, by the first device, the files indicated by the file migration list from the second storage module to the third storage module through a device where the second storage module is located and a device where the third storage module is located.

[0006] In a second aspect, a file migration apparatus is provided, which is applied to a first device in a storage layer system, the storage layer system comprising the first device, one or more second devices and a plurality of storage modules, any one of the plurality of storage modules being located in the first device or the second device, the plurality of storage modules being sorted in order of read-write performance from high to low, the apparatus comprising: a generation module configured to generate a new file according to a received data storage request, and store the new file to a first storage module, the first storage module being a storage module sorted at a first position in the plurality of storage modules; an update module configured to update a first correspondence relationship and a second correspondence relationship of a second storage module according to a received data query request, the first correspondence relationship comprising a hit frequency of each file in the second storage module, and the second correspondence relationship comprising a quantity of files in the second storage module with a hit frequency in a plurality of preset hit frequency intervals; a determination module configured to determine a file migration list of the second storage module according to the first correspondence relationship and the second correspondence relationship of the second storage module every preset time length, the second storage module being any one of the plurality of storage modules except a storage module sorted at a last position, the file migration list being used to indicate files that need to be migrated from the second storage module to a third storage module, the third storage module being a storage module sorted at a next position of the second storage module in the plurality of storage modules; and a migration module configured to migrate the files indicated by the file migration list from the second storage module to the third storage module through a device where the second storage module is located and a device where the third storage module is located.

[0007] In a third aspect, a computer device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program, when executed by the processor, implements the file migration method of the first aspect.

[0008] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program, when executed by a processor, implements the file migration method of the first aspect.

[0009] In a fifth aspect, a computer program product containing instructions, which, when executed on a computer, causes the computer to perform the steps of the file migration method of the first aspect. Advantageous effects

[0010] In the present application, the plurality of storage modules are sorted in order of read-write performance from high to low. The first device can continuously migrate files in each storage module in the plurality of storage modules to the next storage module to ensure that the first storage module always has capacity to store new files. Moreover, since the first device performs file migration according to the file migration list of each storage module, and the file migration list of a storage module indicates that the file in the storage module is cold data, cold and hot data separation can be achieved through file migration, so that hot data is kept in storage modules with higher read-write performance, and cold data is migrated to storage modules with lower read-write performance. In this way, while achieving distributed file storage, the read-write efficiency of hot data can be improved, and the read-write delay of hot data can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0011] FIG. 1 is a schematic diagram of a storage layer system according to an embodiment of the present application;

[0012] FIG. 2 is a schematic diagram of a target tree according to an embodiment of the present application;

[0013] FIG. 3 is a schematic diagram of a key-value pair according to an embodiment of the present application;

[0014] FIG. 4 is a schematic diagram of another key-value pair according to an embodiment of the present application;

[0015] FIG. 5 is a schematic diagram of index information according to an embodiment of the present application;

[0016] FIG. 6 is a flowchart of a file migration method according to an embodiment of the present application;

[0017] FIG. 7 is a schematic diagram of the structure of a file migration device according to an embodiment of the present application;

[0018] FIG. 8 is a schematic diagram of the structure of a computer device according to an embodiment of the present application. EMBODIMENTS OF THE INVENTION

[0019] The file migration method provided by the embodiments of the present application can be applied in a system with data processing needs, which is referred to as a storage layer system in the embodiments of the present application. The storage layer system is described below.

[0020] For example, after a blockchain system executes all transactions in a block, a series of ledger data is generated, which can include a plurality of key-value pairs. In this case, the file migration method provided by the embodiments of the present application can be used to store the ledger data in the storage layer system, and the ledger data can be migrated in the storage layer system according to the cold and hot degree of the ledger data.

[0021] The external system connected by the storage layer system is a system that has a data storage requirement. For example, the external system connected by the storage layer system can be a database system, a blockchain system, etc., and the embodiments of the present application do not limit this.

[0022] The storage layer system is used to receive, process and reply to various requests sent by the external system. For example, the request sent by the external system can include a data storage request, a data query request, etc., and the embodiments of the present application do not limit this.

[0023] In some embodiments, the storage layer system can implement a horizontally scalable database internal file storage mechanism for blockchain system features, which can realize hot and cold data statistics and separate storage. For example, this storage mechanism can be adapted to the underlying file storage engine of a continuous database with block granularity, and can also be used for storage of final to-be-fallen disk files of a key value (kv) type database such as a ledger.

[0024] FIG. 1 is a schematic diagram of a storage layer system according to an embodiment of the present application. Referring to FIG. 1, the storage layer system can include a first device 101, one or more second devices 102 (one second device 102 is taken as an example in FIG. 1), and a plurality of storage modules (four storage modules are taken as an example in FIG. 1). Among them, the plurality of storage modules are dispersed in the first device 101 and the plurality of second devices 102, and any one of the plurality of storage modules is located in the first device 101 or the second device 102.

[0025] The first device 101 and the second device 102 can communicate through wired connection or wireless connection. For example, the first device 101 and the second device 102 can establish a communication connection through a google remote procedure call (grpc) protocol.

[0026] The first device 101 is used to receive, process and reply to various requests sent by the external system. The first device 101 can control file migration between the plurality of storage modules. The first device 101 runs management logic for overall management and control of the storage layer system, and the first device 101 can selectively bind physical storage resources (such as disks, etc.) and directly interface with an upper system (such as a blockchain system or a database system). The first device 101 can provide a file read-write interface to the upper system. In this case, the first device 101 can receive requests (such as data query requests, data storage requests, etc.) sent by the upper system and process these requests. The second device 102 is a device other than the first device 101, and the second device 102 can selectively bind physical storage resources (such as disks, etc.).

[0027] In some embodiments, the first device 101 can be referred to as a master device, and the second device 102 can be referred to as a slave device. For example, the first device 101 and the second device 102 can be physical machines. The first device 101 can carry and manage 0, 1 or more storage modules. Any one of the second devices 102 can carry and manage 0, 1 or more storage modules. The first device 101 and the one or more second devices 102 as a whole can carry and manage a plurality of storage modules.

[0028] Each of the plurality of storage modules is configured to store files, and file migration between the plurality of storage modules is enabled. The file migration between the plurality of storage modules can be controlled and executed by the first device 101.

[0029] The file migration refers to that the first device 101 migrates part of files in one storage module to another storage module for storage.

[0030] The file migration operation is periodic. Optionally, the first device 101 can migrate part of files in one storage module to another storage module for storage every preset time length. The preset time length can be preset. For example, the preset time length can be set to 2 minutes, 3 minutes, 4 minutes, etc., which is not limited in the embodiments of the present application.

[0031] The files in the storage module include data, such as key-value pairs, etc., sent by an external system and needing to be stored. For example, the storage module can be a disk, a storage area in the disk, or other devices that can be used for data storage, etc., which is not limited in the embodiments of the present application. For example, the plurality of storage modules can include a solid state drive (SSD), a hybrid hard drive (HHD), a traditional hard disk drive (HDD), etc., which is not limited in the embodiments of the present application.

[0032] Optionally, one storage module can correspond to one instance, and the instance can bind the path of the corresponding storage module, so that the first device 101 can store data to or read data from the corresponding storage module through the instance.

[0033] The storage module has an identifier, i.e., a module identifier. The module identifier of one storage module is used to identify the storage module.

[0034] Optionally, each of the plurality of storage modules has a hot-cold priority score. The hot-cold priority score of each of the plurality of storage modules can be set according to the read-write performance (including but not limited to read-write efficiency) of each of the plurality of storage modules. Specifically, the higher the read-write performance of a storage module, the higher the hot-cold priority score of the storage module. The lower the read-write performance of a storage module, the lower the hot-cold priority score of the storage module.

[0035] The hot-cold priority score of each of the plurality of storage modules can be pre-set. For example, since the read-write efficiency of an SSD is higher than that of an HDD, the hot-cold priority score of the SSD can be set to be higher than that of the HDD.

[0036] In some embodiments, the hot-cold priority score of a storage module can indicate the priority of the storage module in storing hot data. Hot data refers to data with a higher access frequency of the external system, and cold data refers to data with a lower access frequency of the external system.

[0037] The higher the hot-cold priority score of a storage module, the higher the read-write performance of the storage module, and the higher the speed of the first device 101 in querying and storing files in the storage module. Therefore, hot data can be stored in a storage module with a higher hot-cold priority score. Conversely, the lower the hot-cold priority score of a storage module, the lower the read-write performance of the storage module, and the lower the speed of the first device 101 in querying and storing files in the storage module. Therefore, cold data can be stored in a storage module with a lower hot-cold priority score. The hot-cold priority scores of the plurality of storage modules are different, and the hot-cold priority scores of the plurality of storage modules have an order of magnitude. The plurality of storage modules are sorted in descending order of the hot-cold priority scores, i.e., the plurality of storage modules are sorted in descending order of the read-write performance.

[0038] In some embodiments, each of the plurality of storage modules can have a first correspondence relationship, a second correspondence relationship, and a third correspondence relationship. Optionally, the first correspondence relationship, the second correspondence relationship, and the third correspondence relationship of each of the plurality of storage modules can be stored in the memory of the first device 101.

[0039] The first correspondence relationship of a storage module can include the hit count of each file in the storage module. The first correspondence relationship of a storage module can be a correspondence relationship between the file identifier of each file stored in the storage module and the hit count of the file. The hit count of a file can be the number of times the file is queried by the external system.

[0040] The first correspondence can include at least one file identifier and at least one hit count, the at least one file identifier and the at least one hit count corresponding one-to-one, and the hit count corresponding to a file identifier being the hit count of the file identified by the file identifier.

[0041] In some embodiments, the first correspondence can include a file identifier list and a hit count list, the number of elements in the file identifier list being the same as the number of elements in the hit count list, and the elements at the same index position in the file identifier list and the hit count list corresponding to each other. In this way, the elements in the file identifier list and the elements in the hit count list correspond one-to-one.

[0042] The elements in the file identifier list are file identifiers. The file identifier list is a list of file identifiers of each file in all files in the record storage module. The elements in the hit count list are hit counts. The hit count list is a list of hit counts of each file in all files in the record storage module.

[0043] It should be noted that the index of the element in the list is used to identify the element at a specific position in the list, and is usually incremented from 0. For example, in a list containing 5 elements, the index of the first element is 0, the index of the second element is 1, and so on, and the index of the fifth element is 4. Through the index, the element in the list can be quickly located and accessed or modified.

[0044] It should be noted that when the storage layer system has not yet started to store files, the first correspondence of each storage module in the plurality of storage modules is empty, i.e., there is no file identifier and no hit count in the first correspondence.

[0045] After the storage layer system starts to store files, such as after a new file is generated and stored in a storage module, the file identifier of the new file can be added to the first correspondence of the storage module, and the hit count corresponding to the file identifier is set to 0. Then, the first correspondence can be updated according to the received data query request. For example, if a file is found in the storage module according to the data query request, the hit count corresponding to the file identifier of the file in the first correspondence of the storage module can be incremented by 1.

[0046] In addition, if a file in a storage module is migrated to another storage module, the file identifier of the migrated file and the corresponding hit count need to be added to the first correspondence of the other storage module.

[0047] The second correspondence of a storage module includes the number of files in the storage module whose hit times are in each of the plurality of preset hit time intervals. The second correspondence of a storage module can be a correspondence between each of the plurality of preset hit time intervals and the number of files, and the number of files corresponding to a preset hit time interval is the number of all files in the storage module whose hit times are in the preset hit time interval.

[0048] The value range of each of the plurality of preset hit time intervals is different. In addition, the plurality of preset hit time intervals are sorted in ascending order of value range. For example, the value range of the plurality of preset hit time intervals is continuous on integers.

[0049] The plurality of preset hit time intervals can be set in advance. Alternatively, the plurality of preset hit time intervals can be obtained according to elements in a static hit time interval list. The hit time interval list can be a system configuration value, that is, it can be set in advance by a technician and is usually not modified during the running of the storage layer system.

[0050] The hit time interval list can include a plurality of elements, and the plurality of elements in the hit time interval list can be sorted in ascending order. The plurality of elements in the hit time interval list correspond one-to-one to the plurality of preset hit time intervals. For example, for any element in the hit time interval list except the last element, the element can be taken as the minimum value of a preset hit time interval corresponding to the element, and the maximum integer smaller than the next element of the element can be taken as the maximum value of the preset hit time interval corresponding to the element; for the last element in the hit time interval list, the value range of a preset hit time interval corresponding to the element is the element to positive infinity.

[0051] The third correspondence of a storage module can include the total number of hits of files in the storage module that are in each of the plurality of preset hit time intervals. The third correspondence of a storage module can be a correspondence between each of the plurality of preset hit time intervals and the total number of hits, and the total number of hits corresponding to a preset hit time interval is the sum of the hit times of all files in the storage module whose hit times are in the preset hit time interval.

[0052] As an example, the second correspondence relationship and the third correspondence relationship can be implemented in an array form. The hit frequency interval list is an array, and correspondingly, a corresponding file quantity array can be constructed as the second correspondence relationship, and a corresponding hit total frequency array can be constructed as the third correspondence relationship. The three arrays are one-dimensional arrays, and the lengths (i.e., the number of elements) of the three arrays are the same, and the elements at the same index positions in the three arrays correspond to each other.

[0053] In this case, the plurality of elements in the file quantity array correspond to the plurality of elements in the hit frequency interval list one by one, that is, correspond to the plurality of preset hit frequency intervals one by one. The plurality of elements in the hit total frequency array correspond to the plurality of elements in the hit frequency interval list one by one, that is, correspond to the plurality of preset hit frequency intervals one by one.

[0054] The first correspondence relationship, the second correspondence relationship, and the third correspondence relationship of each storage module in the plurality of storage modules can be managed by the first device 101. The first device 101 can manage the first correspondence relationship, the second correspondence relationship, and the third correspondence relationship of each storage module through the module identifier of each storage module in the plurality of storage modules.

[0055] In some embodiments, the first device 101 and each device in the one or more second devices 102 are configured with a globally unique device identifier, and each device needs to assume both the role of a server and the role of a client in the network.

[0056] As a server, different server processing logics can be started according to the device identity (such as a master device or a slave device) of the server. As a module that directly receives messages from other devices, the server will receive processing functions registered by upper modules when starting, so as to call these processing functions to perform corresponding processing when receiving messages sent by other devices, and then return the processing result to the sender.

[0057] As a client, because there is a scenario of data transmission between different storage modules in the same device in the storage layer system, the client needs to have the ability to distinguish the scenario of data transmission in the same device, and in this scenario, the packing and unpacking processing logics of the communication protocol (such as the grpc protocol) itself are omitted, and data transmission in the memory is directly performed.

[0058] In addition, the first device 101 maintains a mapping between the device identifier of the first device 101 and each device in the one or more second devices 102 and the storage module identifier. Each second device 102 maintains a mapping between its own device identifier and the storage module identifier. The device identifier to storage module identifier conversion can be implemented in each device. Therefore, the first device 101 can implement data transmission between any two storage modules.

[0059] In some embodiments, the storage layer system can interface with a database system. The database system can interface with other systems that have data storage needs, such as a blockchain system, and the like. Illustratively, when the blockchain system needs to store data, the data can be first sent to the database system, and then sent to the storage layer system by the database system. The storage layer system can store the data based on the target tree.

[0060] Optionally, the storage layer system and the database system can be independent systems, in which case the storage layer system and the database system can run on independent physical machines, respectively. Alternatively, the database system can include the storage layer system, in which case the database system includes a specific system that interfaces with the blockchain system and runs on the same physical machine as the storage layer system.

[0061] Next, the structure of the target tree provided by the embodiments of the present application is described.

[0062] The target tree is a tree structure. The target tree includes branch nodes and leaf nodes at the lowest level of the target tree, wherein a node with a number of child nodes of 0 is a leaf node, a node with a number of child nodes greater than or equal to 1 is a branch node, and the branch nodes and the leaf nodes are logical nodes. The branch node at the uppermost level of the target tree can be referred to as a root node, which is a branch node without a parent node in the target tree.

[0063] Optionally, the target tree can be an n-ary tree, that is, the number of child nodes of each branch node in the target tree is at most n, where n is an integer greater than or equal to 2. Any branch node in the n-ary tree can have at most n child nodes. Moreover, for branch nodes at the same level in the n-ary tree, a new branch node can be added after the last branch node at the same level only when the number of child nodes of the last branch node at the same level reaches n.

[0064] The leaf node stores a file identifier, a maximum key value, and a minimum key value.

[0065] The file identifier is used to identify a file stored in the storage layer system. For example, the file identifier can be a file identity (ID), a file name, or the like. The minimum key value in the leaf node is the minimum key value in the file identified by the file identifier in the leaf node. The maximum key value in the leaf node is the maximum key value in the file identified by the file identifier in the leaf node.

[0066] In the target tree in the embodiments of the present application, the leaf nodes correspond one-to-one to files. That is, a leaf node contains a file identifier of a file and the minimum key value and the maximum key value in the file.

[0067] In the embodiments of the present application, the storage layer system can store one or more files, and each file can include one or more key-value pairs. The file identifiers of all the files stored in the storage layer system are globally ordered, that is, the file identifier of any file is ordered in all the files stored in the storage layer system. In addition, the key values in all the key-value pairs included in the files are also globally ordered, that is, the key value in any key-value pair included in any file is ordered in all the key-value pairs in all the files stored in the storage layer system.

[0068] Optionally, the global order of the key values in the key-value pairs can be in the order of increasing dictionary order.

[0069] In this case, the minimum key value in a file refers to the key value with the earliest order among the key values in all the key-value pairs in the file. The maximum key value in a file refers to the key value with the latest order among the key values in all the key-value pairs in the file.

[0070] It should be noted that in the computer field, dictionary order can be used to indicate the size relationship between any two strings. The size relationship between two strings depends on the size relationship of the American Standard Code for Information Interchange (ASCII) codes of the characters in the string from left to right.

[0071] Optionally, the global order of the file identifiers of the files can be in the order of increasing dictionary order.

[0072] For example, the file identifier of a file can be increased by a preset value starting from 0.

[0073] The preset value can be set in advance. For example, the preset value can be set to 1, 2, 3, etc., which is not limited in the embodiments of the present application.

[0074] In this case, the file identifiers of all the files are an arithmetic sequence. For example, assuming that the preset value is 1, then the file identifiers of the first file, the second file, the third file, …, the kth file generated can be 0, 1, 2, …, k-1, respectively.

[0075] In some embodiments, a globally continuously increasing counter can be maintained in the storage layer system, and the counter is used to generate the file identifier. In this case, the storage layer system can take the current value of the counter as the file identifier of a new file each time a new file is generated, so that the files stored in the storage layer system are a batch of files with continuously increasing file identifiers.

[0076] The branch node includes a child node list, which includes the minimum key value and the maximum key value in each child node of the branch node. For example, the range from the minimum key value in a child node to the maximum key value in the child node can be referred to as the key range of the child node. The child nodes of a branch node can be branch nodes or leaf nodes in the next layer. The minimum key value in a branch node is the minimum key value in all key values in the child node list in the branch node, that is, the key value with the earliest order in all key values in the child node list; and the maximum key value in the branch node is the maximum key value in all key values in the child node list in the branch node, that is, the key value with the latest order in all key values in the child node list. The minimum key value and the maximum key value in a leaf node are the minimum key value and the maximum key value stored in the leaf node.

[0077] For example, FIG. 2 is a schematic diagram of a target tree provided by an embodiment of the present application. Referring to FIG. 2, the target tree is a 2-ary tree, and the target tree includes a plurality of leaf nodes (for example, node n4, node n5, and node n6 in FIG. 2) and a plurality of branch nodes (for example, node n1, node n2, and node n3 in FIG. 2). The child nodes of node n1 include node n2 and node n3, the child nodes of node n2 include node n4 and node n5, and the child nodes of node n3 include node n6. In the embodiment of the present application, the branch node in the uppermost layer of the target tree can be referred to as a root node, for example, node n1 in FIG. 2 is a root node.

[0078] The leaf node n4 in FIG. 2 stores a file identifier 0, a minimum key value 1, and a maximum key value 3. The leaf node n5 stores a file identifier 1, a minimum key value 4, and a maximum key value 6. The leaf node n6 stores a file identifier 2, a minimum key value 7, and a maximum key value 9.

[0079] The branch node n2 in FIG. 2 stores a child node list, which includes the minimum key value 1 and the maximum key value 3 in the leaf node n4, and includes the minimum key value 4 and the maximum key value 6 in the leaf node n5.

[0080] The branch node n3 in FIG. 2 stores a child node list, which includes the minimum key value 7 and the maximum key value 9 in the leaf node n6.

[0081] The branch node n1 in FIG. 2 stores a child node list, which includes the minimum key value 1 and the maximum key value 6 in the branch node n2, and includes the minimum key value 7 and the maximum key value 9 in the branch node n3.

[0082] In the embodiments of the present application, the storage layer system can store the key-value pairs sent by the external system based on the target tree. For example, the target tree can also be referred to as a solid branch tree.

[0083] In some cases, the key values in the key-value pairs sent by the external system to the storage layer system are globally ordered. The concept of global order has been explained above and will not be repeated here.

[0084] In some embodiments, the storage layer system can be connected to a database system, and the database system can be connected to a blockchain system. In this case, the blockchain system can send the key-value pairs to be stored to the database system, and the database system can send a data storage request carrying the key-value pairs to the storage layer system after receiving the key-value pairs.

[0085] As an example, the key-value pairs to be stored sent by the blockchain system to the database system can be block data. Since the block data is strictly organized according to the block number, that is, the key values in the block data are block numbers, the key values in the block data are naturally globally ordered. In this case, the database system can directly send the block data to the storage layer system for storage in the data storage request.

[0086] As another example, the key-value pairs to be stored sent by the blockchain system to the database system can be account data in a block. The account data in a block includes a plurality of key-value pairs, and the key values in the plurality of key-value pairs in the same block are different, but the key values in the key-value pairs in different blocks can be the same.

[0087] In this case, the database system can first process the account data. Specifically, the database system can introduce a block number in the account data to construct key-value pairs with globally ordered key values, and then send the constructed key-value pairs to the storage layer system for storage in the data storage request. In this way, it can be ensured that the key values in the key-value pairs carried in the data storage request sent by the database system to the storage layer system are globally ordered.

[0088] Optionally, the key-value pairs can be referred to as target key-value pairs in the embodiments of the present application. The key values in the target key-value pairs include a block number and a key value in a block identified by the block number (i.e., a key value in the key-value pairs in the account data), and the value values in the target key-value pairs are value values in the block (i.e., value values in the key-value pairs in the account data).

[0089] In some embodiments, the key value in the target key-value pair can be obtained by splicing the block number and the key value within the block, of course, the key value in the target key-value pair can also be obtained by other ways, which are not limited in the embodiments of the present application.

[0090] Since the block number is naturally globally ordered, and the key values within a block are different, the key values in the target key-value pair containing the block number and the key values within the block identified by the block number are also globally ordered.

[0091] For example, the block with block number 00000001 includes a plurality of key-value pairs, for each key-value pair in the plurality of key-value pairs, the block number 00000001 can be added before the key value in the key-value pair to obtain a new key value.

[0092] For example, as shown in (a) of FIG. 3, the block 1 with block number 00000001 includes key-value pairs bbbb:{value1...}, bggg:{value2...}, caaa:{value3...}, xaab:{value4...}. As shown in (b) of FIG. 3, the block 2 with block number 00000002 includes key-value pairs aabb:{value5...}, bggg:{value6...}, bcde:{value7...}.

[0093] Then, after the blockchain system sends these key-value pairs to the database system, the database system can introduce the block number 00000001 in the key-value pairs in the block 1, specifically, splice 00000001 before the key value in each key-value pair in the block 1 to obtain target key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, 00000001xaab:{value4...}. Introduce the block number 00000002 in the key-value pairs in the block 2, specifically, splice 00000002 before the key value in each key-value pair in the block 2 to obtain target key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, 00000002bcde:{value7...}. The key values in the target key-value pairs obtained in this way are globally ordered.

[0094] It should be noted that the storage layer system in the embodiments of the present application is not limited to storing the key-value pairs in the blockchain system proposed above, but can also store other key-value pairs, as long as the key values in the key-value pairs are globally ordered.

[0095] The following describes a process of constructing an initial target tree for the first device 101, which can include the following steps (1)-(3):

[0096] (1) The first device 101 receives a data storage request carrying a key-value pair to be stored, the key value in the key-value pair being globally ordered. The data storage request is used to instruct the first device 101 to store the key-value pair to be stored. In some embodiments, the data storage request can be sent by an external system, such as a blockchain system or a database system.

[0097] In this case, the key values in all key-value pairs to be stored sent by the external system to the first device 101 are ordered. After the first device 101 stores these key-value pairs, the key values in all key-value pairs stored in the storage layer system are also ordered.

[0098] (2) The first device 101 generates a new file containing one or more key-value pairs according to the order of the key values in the key-value pairs, and stores the new file in the first storage module, which is the first storage module in the plurality of storage modules in terms of order; wherein the file identifiers of all files containing key-value pairs are globally ordered.

[0099] The first storage module is the storage module with the highest read-write performance in the plurality of storage modules.

[0100] The first storage module can be located in the first device 101, and of course, can also be located in the second device 102, and the embodiments of the present application do not limit this. If the first storage module is located in the first device 101, the first device 101 can directly store the new file to the first storage module; if the first storage module is located in the second device 102, the first device 101 can send the new file to the second device 102 where the first storage module is located, and the second device 102 can store the new file to the first storage module after receiving the new file.

[0101] The key values in the key-value pairs to be stored received by the first device 101 are ordered. In this case, the first device 101 generates a new file according to the order of the key values in the key-value pairs, and the key values in the key-value pairs in the new file are also ordered.

[0102] In some embodiments, the operation of step (2) can be that the first device 101 saves all the key-value pairs carried by each received data storage request in the memory of the first device 101 in the order of the key values in the key-value pairs. If the difference between the total data amount of one or more key-value pairs currently saved in the memory in order and the preset data amount is less than or equal to the preset difference, the first device 101 generates a new file containing the one or more key-value pairs. The first device 101 stores the new file in the first storage module.

[0103] The preset data amount can be set in advance. For example, the preset data amount can be set to 8 megabytes (MB), 9 MB, 10 MB, etc., which is not limited in the embodiments of the present application. Alternatively, the preset data amount can be set according to an integer multiple of the block size of the storage layer system. For example, if the block size is 4 MB, the preset data amount can be set to 4 MB, 8 MB, 12 MB, etc. The block size of the storage layer system refers to the smallest unit used by the storage layer system to organize and manage data storage. The preset difference can be set in advance. For example, the preset difference can be set to 100 kilobytes (KB), 200 KB, 300 KB, etc., which is not limited in the embodiments of the present application.

[0104] It should be noted that the data amount of each key-value pair is generally close, and the data amount of one key-value pair is usually within a predetermined range. In the embodiments of the present application, the preset data amount can be much larger than the data amount of one key-value pair. The preset difference can be close to the data amount of one key-value pair, such as equal to or slightly larger than the data amount of one key-value pair.

[0105] The first device 101 can save all the key-value pairs carried by each received data storage request in the memory in order. In this case, if the total data amount of one or more key-value pairs currently saved in the memory in order is close to the preset data amount (i.e., the difference between the total data amount and the preset data amount is less than or equal to the preset difference), all the key-value pairs currently saved in the memory can be stored in the first storage module in the form of a new file. In this way, batch storage of key-value pairs can be implemented, thereby improving data storage efficiency. In this case, the file size of each file stored in the first storage module is basically consistent, and is close to the preset data amount, thereby facilitating file management.

[0106] It should be noted that if the total data amount of all key-value pairs currently saved in the memory is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference, it indicates that the total data amount of the key-value pairs currently saved in the memory is greatly different from the preset data amount, and the condition for generating a new file is not met. In this case, the first device 101 can continue to receive the data storage request sent by the external system, and save the key-value pairs carried by the data storage request in the memory, until the difference between the total data amount of the plurality of key-value pairs saved in sequence in the memory and the preset data amount is less than or equal to the preset difference, and then generate a new file and store it.

[0107] The generated new file is stored to the first storage module, that is, the persistence of the new file is realized. After the new file is stored to the first storage module, one or more key-value pairs in the new file are deleted from the memory.

[0108] It should be noted that the key values in the key-value pairs saved in the memory are in order. In this case, the new file is generated according to the key-value pairs saved in the memory, and the key values in all key-value pairs in the new file are also in order. It should be noted that since one file can contain one or more key-value pairs, the key-value pairs in one file can cover the data of one block or multiple blocks.

[0109] For example, as shown in FIG. 4, the first device 101 receives a data storage request 1, and the data storage request 1 carries key-value pairs 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. The first device 101 saves the key-value pairs carried by the data storage request 1 in sequence to the memory, and then the key-value pairs currently saved in the memory include 00000001bbbb:{value1...}, 00000001bggg:{value2...}, 00000001caaa:{value3...}, and 00000001xaab:{value4...}. Since the total data amount of these key-value pairs is less than the preset data amount, and the difference between the total data amount and the preset data amount is greater than the preset difference, the condition for generating a new file is not met.

[0110] Afterwards, the first device 101 receives a data storage request 2 carrying key-value pairs 00000002aabb:{value5...}, 00000002bggg:{value6...}, 00000002bcde:{value7...}. The first device 101 stores the key-value pairs carried in the data storage request 2 in sequence in the memory. After storing the key-value pair 00000002bggg:{value6...} in the memory, the first device 101 finds that the difference between the total data amount of the key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} currently stored in the memory and the preset data amount is less than or equal to the preset difference, satisfying the condition for generating a new file, and thus generates a new file containing the key-value pairs 00000001bbbb:{value1...} to 00000002bggg:{value6...} and stores the new file in the first storage module. The first device 101 stores the key-value pair 00000002bcde:{value7...} carried in the data storage request 2 in the memory.

[0111] Afterwards, the first device 101 receives a data storage request 3 carrying a key-value pair 00000003bcbb:{value1...}. The first device 101 stores the key-value pair carried in the data storage request 3 in sequence in the memory, and the key-value pairs currently stored in the memory include 00000002bcde:{value7...} and 00000003bcbb:{value1...}. Since the total data amount of these key-value pairs is less than the preset data amount and the difference between the total data amount and the preset data amount is greater than the preset difference, the condition for generating a new file is not satisfied. The first device 101 continues to wait for the next data storage request.

[0112] (3) The first device 101 generates a new leaf node, and constructs an initial target tree according to the new leaf node. The new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value in all key-value pairs in the new file.

[0113] After generating the new leaf node, the first device 101 can newly create a branch node as the parent node of the new leaf node. The branch node contains a child node list, and the child node list includes the minimum key value and the maximum key value in the new leaf node. Since the number of current leaf nodes is 1, the branch node is the root node, and thus the construction of the initial target tree is completed.

[0114] In some embodiments, a branch node in the target tree can include a list of child nodes. The list of child nodes in a branch node stores node information of each child node of the branch node.

[0115] For example, the node information of a node can include a pointer of the node, a minimum key value in the node, and a maximum key value in the node.

[0116] The pointer of a node is used to point to the memory space where the node is located. The list of child nodes in a branch node includes node information of all child nodes of the branch node, and the node information of all child nodes is sequentially arranged in the list of child nodes.

[0117] In some cases, after the first device 101 completes the initial construction of the target tree, the first device 101 can also store a pointer of a root node of the target tree and a pointer of a last leaf node in the target tree in the memory of the first device 101. In this way, the first device 101 can subsequently quickly index to the root node in the target tree according to the pointer of the root node maintained in the memory, and quickly index to the last leaf node in the target tree according to the pointer of the leaf node maintained in the memory.

[0118] In some embodiments, for any data storage request, if all key-value pairs carried by the data storage request have been stored in the storage module in the form of a file, and the target tree has been constructed or updated accordingly, the first device 101 can return a data storage success message to an external system that sends the data storage request. For example, the data storage success message can carry a file identifier of the file where the key-value pairs in the data storage request are located, and the present embodiments are not limited in this regard.

[0119] As an example, the target tree in the present embodiments does not need to be persisted, i.e., does not need to be stored in the storage module, but can be stored in the memory of the first device 101, so that storage resources can be saved.

[0120] In this case, if the first device 101 restarts due to a shutdown or other reasons, after the restart, the first device 101 can reconstruct the target tree in the memory according to all files stored in the plurality of storage modules. It should be noted that the "file" referred to in the present embodiments refers to a file containing key-value pairs involved in the present embodiments.

[0121] Specifically, since the file identifiers are globally ordered, and each leaf node stores a file identifier of a file and the minimum key value and the maximum key value in the file, the first device 101 can construct all the leaf nodes of the last layer of the target tree in the memory according to all the files stored in the plurality of storage modules, and construct the branch nodes layer by layer upwards according to the leaf nodes of the last layer, until the root node is constructed, so that the target tree is reconstructed. After the target tree is reconstructed, the pointer of the root node in the target tree and the pointer of the last leaf node in the target tree are maintained in the memory.

[0122] After the initial target tree is constructed through the steps (1) to (3) described above, the first device 101 can perform data processing based on the target tree, such as data query and data storage.

[0123] Next, the process in which the first device 101 performs data storage and updates the target tree in the file migration method provided by the embodiment of the present application will be described, which can include the following steps A to C:

[0124] Step A: The first device 101 receives a data storage request, and the data storage request carries a key-value pair to be stored, and the key values in the key-value pair are globally ordered. The operation of step A is similar to the operation of step (1) described above, and the embodiment of the present application will not be repeated here.

[0125] Step B: The first device 101 generates a new file containing one or more key-value pairs according to the order of the key values in the key-value pair, and stores the new file to the first storage module; wherein the file identifiers of all the files containing the key-value pairs are globally ordered.

[0126] The operation of step B is similar to the operation of step (2) described above, and the embodiment of the present application will not be repeated here.

[0127] Step C: The first device 101 adds a new leaf node after the last leaf node in the target tree, updates the target tree according to the new leaf node, and the new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value in all the key-value pairs in the new file.

[0128] Optionally, the first device 101 can first obtain the pointer of the last leaf node in the target tree in the memory, and then add a new leaf node after the leaf node according to the pointer of the leaf node.

[0129] For example, the last leaf node in the target tree can be located at the rightmost side of the last layer of the target tree, in which case the insertion order of the leaf nodes is from left to right. Then the first device 101 inserts the new leaf node after the last leaf node in the target tree, that is, inserts the new leaf node at the rightmost side of the last layer of the target tree.

[0130] Alternatively, the last leaf node in the target tree can be located at the leftmost side of the last layer of the target tree, in which case the insertion order of the leaf nodes is from right to left. Then the first device 101 inserts the new leaf node after the last leaf node in the target tree, that is, inserts the new leaf node at the leftmost side of the last layer of the target tree.

[0131] In the embodiments of the present application, when data is stored, no indexing in the target tree is needed, and a new file can be directly generated for storage, and accordingly a new leaf node can be directly added in the target tree to update the target tree, thereby improving the data storage efficiency.

[0132] It should be noted that the update logic of the target tree in the embodiments of the present application is only one, that is, a new leaf node is inserted after the last leaf node of the whole tree (such as the rightmost side or the leftmost side of the last layer). In the traditional tree structure, a random insertion logic is used, which needs to index from the root node to the leaf node first, then insert and store data, and then backtrack to update or split the parent node. Compared with the random insertion logic in the traditional tree structure, the update logic of the target tree in the embodiments of the present application can avoid the indexing process from the root node to the leaf node, thereby improving the data storage efficiency.

[0133] It should be noted that since the file identifiers are globally ordered, and each leaf node stores a file identifier of a file, the file identifiers in all leaf nodes in the last layer of the target tree are ordered from left to right or from right to left.

[0134] It should be noted that the tree shape of the whole target tree in the embodiments of the present application is relatively fixed, and except for the nodes in the path from the root node to the last leaf node (i.e., the rightmost path or the leftmost path), the remaining nodes are static nodes. A static node refers to a node that does not undergo a modification process and has a determined overall shape.

[0135] In some embodiments, the target tree is an n-ary tree, where n is an integer greater than or equal to 2. In the case of an n-ary tree, no node splitting is needed in the process of updating the target tree according to the new leaf node, thereby improving the update efficiency.

[0136] Optionally, the operation of updating the target tree according to the new leaf node by the first device 101 can be that the first device 101 updates the nodes in the target tree upwards layer by layer according to the new leaf node until the root node is updated. In the process of updating the nodes upwards layer by layer, if an original node in the target tree needs to be updated, the original node is added with a write lock, and the write lock is released after the update; in the process of updating the nodes upwards layer by layer, if a new node is added in the target tree, the new node does not need to be added with a write lock.

[0137] It should be noted that the original node in the target tree is added with a write lock before the original node is updated in the embodiments of the present application. In this case, if a query operation on the original node occurs in the process of updating the original node, the query operation needs to wait until the write lock of the original node is released after the update of the original node is completed, so that the accuracy of data query can be ensured.

[0138] The new node added in the process of updating the target tree does not need to be added with a write lock in the embodiments of the present application. This is because the new node is invisible to the outside before the update of the target tree is completed, so the data query process does not pass through the new node, and the new node does not need to be added with a write lock.

[0139] In some embodiments, the operation of updating the nodes in the target tree upwards layer by layer according to the new leaf node by the first device 101 can include the following steps a to f.

[0140] Step a: let the new leaf node be a current node.

[0141] Step b: determine whether the current node is a new node or an original node in the target tree. If the current node is a new node, the following steps c to e are performed; if the current node is an original node, the following step f is performed.

[0142] Step c: if the current node is a new node, determine whether the current node can be a child node of the last node in the upper layer. If the current node can be a child node of the last node in the upper layer, the following step d is performed; if the current node cannot be a child node of the last node in the upper layer, the following step e is performed.

[0143] For example, in the case of an n-ary tree as the target tree, if the number of child nodes of the last node in the upper layer is less than n, the current node can be a child node of the last node in the upper layer; if the number of child nodes of the last node in the upper layer is equal to n, the current node cannot be a child node of the last node in the upper layer.

[0144] Step d: If the current node can be a child node of the last node in the upper layer, the last node in the upper layer is taken as the parent node of the current node, a write lock is added to the parent node of the current node, the parent node of the current node is updated, the write lock of the parent node of the current node is released after the update, the parent node of the current node is taken as the current node, and step b and the subsequent steps are re-executed.

[0145] If the current node can be a child node of the last node in the upper layer, the last node in the upper layer can be taken as the parent node of the current node for updating. The updating here is to add the node information of the current node in the child node list of the last node in the upper layer.

[0146] Step e: If the current node cannot be a child node of the last node in the upper layer, a new node is added in the upper layer as the parent node of the current node, the parent node of the current node is taken as the current node, and step b and the subsequent steps are re-executed.

[0147] If the current node cannot be a child node of the last node in the upper layer, a new node can be added after the last node in the upper layer as the parent node of the current node. In this case, the parent node of the current node includes a child node list, and the child node list includes the node information of the current node.

[0148] Step f: If the current node is an original node, a write lock is added to the parent node of the current node, the parent node of the current node is updated, the write lock of the parent node of the current node is released after the update, the parent node of the current node is taken as the current node, and step b and the subsequent steps are re-executed.

[0149] If the current node is an original node, the parent node of the current node already exists in the target tree, and thus the parent node of the current node can be directly updated at this time. The updating here is to update the node information of the current node included in the child node list of the parent node of the current node.

[0150] It should be noted that after the current node is re-determined in step d, step e or step f, it can be judged whether the current node is an original root node of the target tree before step b is re-executed.

[0151] If the current node is not the original root node of the target tree and is not a node at the same level as the original root node, then step b and the subsequent steps are re-executed. If the current node is the original root node of the target tree, it is determined that the update of the target tree is completed. If the current node is not the original root node of the target tree and is a node at the same level as the original root node, a new root node needs to be added at the upper level as the parent node of the original root node and the current node, and the new root node includes a child node list, which includes the node information of the original root node and the node information of the current node. In this way, the update of the target tree is completed. In this case, the height of the target tree will increase by one level.

[0152] It should be noted that after updating the target tree, the pointer of the root node in the target tree maintained in the memory and the pointer of the last leaf node in the target tree can be updated.

[0153] The behavior of updating the pointer of the root node in the memory and the behavior of obtaining the pointer of the root node in the memory cannot be concurrent. That is, if the pointer of the root node in the memory is being updated, the operation of obtaining the pointer of the root node in the memory needs to be executed after the update is completed; if the pointer of the root node in the memory is being obtained, the operation of updating the pointer of the root node in the memory needs to be executed after the pointer of the root node is successfully obtained. This is because the pointer of the root node may be in a transitional state, such as empty, during the process of updating the pointer of the root node, and in this case, an error will occur if the pointer of the root node is obtained.

[0154] In some cases, an external system needs to query data stored in the storage layer system. In this case, the external system can send a data query request to the first device 101. For example, these data query requests can include two types, the first type is used to query the value in the key-value pair, and the second type is used to query the file containing the key-value pair. The two query methods are described in detail below.

[0155] The first type can include the following step A to the following step C.

[0156] Step A: The first device 101 receives a first data query request, and the first data query request carries a target key value.

[0157] The target key value is the key value in the key-value pair to which the value value to be queried belongs. The first data query request is used to request to query the value value in the key-value pair to which the target key value belongs. The first data query request can be sent by an external system, such as a database system, a blockchain system, etc., which is not limited in the embodiments of the present application.

[0158] Step B: The first device 101 determines the first leaf node indexed by the target key value in the target tree.

[0159] The target key value is greater than or equal to the minimum key value in the first leaf node and less than or equal to the maximum key value in the first leaf node.

[0160] For example, the first leaf node to which the target key value is indexed in the target tree can also be referred to as the first leaf node to which the target key value is distributed in the target tree.

[0161] Optionally, the first device 101 can index the first leaf node in the target tree according to the target key value by means of binary search, and of course, the first leaf node can also be indexed in the target tree by other means, and the embodiments of the present application do not limit this.

[0162] For example, if the target tree is an n-ary tree, the complexity of the indexing process satisfies O (logN) x log2branch. Wherein, the complexity of the tree indexing path is O (logN), and the complexity of specifying a child node in each node by means of binary search is a constant value log2branch, and the branch value is a fixed branch number constant n.

[0163] In some embodiments, the operation of step B can be: the first device 101 indexes the nodes from the root node down layer by layer in the target tree according to the target key value until the first leaf node is indexed.

[0164] As an example, in the process of indexing the nodes down layer by layer, a read lock is added to each node after it is indexed, the next node is indexed according to the node and the target key value, and the read lock of the node is released after the next node is indexed; the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0165] Specifically, after the first device 101 receives the first data query request, it can first obtain the pointer of the root node in the memory, and then index the root node according to the pointer of the root node. A read lock is added to the root node after it is indexed.

[0166] Then, determine which key range of the child node of the root node the target key value is in, which is the process of indexing the next node. After determining that the target key value is in the key range of a child node of the root node, the child node is the next node indexed, and the read lock of the root node is released and a read lock is added to the latest indexed node.

[0167] If the node is a branch node, it can continue to determine which key range of the child node of the node the target key value is in, that is, continue to index the next node. Until the indexed node is a leaf node.

[0168] If the node is a leaf node, it can be determined that the node is the first leaf node, and the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0169] As another example, in the process of indexing nodes layer by layer downwards, each time a node is indexed, if the node is the last child node of all child nodes of its parent node, a read lock is added to the node, the next node is indexed according to the node and the target key value, and the read lock of the node is released after the next node is indexed; if the node is not the last child node of all child nodes of its parent node, no read lock is added to the node, and the next node is directly indexed according to the node and the target key value, and no read lock needs to be added to all nodes indexed subsequently in this case, so that the lock resource can be saved.

[0170] If the read lock is added to the first leaf node in the indexing process, the read lock of the first leaf node is released after the file identifier in the first leaf node is obtained.

[0171] It should be noted that the read lock and the read lock are shared locks, and the read lock and the write lock are mutually exclusive locks in the embodiments of the present application.

[0172] Optionally, the concurrent model of single-threaded writing and multi-threaded reading can be used in the embodiments of the present application.

[0173] In this case, after a thread adds a read lock to a node, another thread can add a read lock to the node.

[0174] After a thread adds a read lock to a node, another thread cannot add a write lock to the node, and needs to wait until the read lock of the node is released, and then add a write lock to the node and continue to operate (such as continuing to update the node). Similarly, after a thread adds a write lock to a node, another thread cannot add a read lock to the node, and needs to wait until the write lock of the node is released, and then add a read lock to the node and continue to operate (such as continuing to index the node).

[0175] In the embodiments of the present application, at most only one lock conflict (also referred to as lock preemption) occurs in one tree indexing process and one tree updating process. That is, the tree structure of the target tree in the embodiments of the present application can avoid path level lock conflict to the greatest extent, so that the system concurrency can be maximized and the main business process delay can be reduced.

[0176] It should be noted that all data query behaviors in the embodiments of the present application can be concurrent, which is very friendly to the scene of concurrent reading of data in the system. For example, if the blockchain system supports concurrent execution of transactions, all behaviors of obtaining ledger data during concurrent execution of transactions can also be completely concurrent in the system. In addition, the node insertion behavior in the embodiments of the present application cannot be concurrent, that is, after one update of the target tree is completed, the next update can be performed.

[0177] Step C: The first device 101 obtains the value in the key-value pair to which the target key value belongs from the identified file (which can be referred to as a target file) identified by the file identifier in the first leaf node.

[0178] The first device 101 can determine a storage module (which can be referred to as a target storage module) in which the target file is located. For example, the first device 101 can determine the target storage module according to the file identifier of the target file and the first correspondence relationship of each storage module in the plurality of storage modules. Specifically, if the file identifier of the target file is included in the first correspondence relationship of a certain storage module, the first device 101 can determine that the storage module is the target storage module. Then, the first device 101 can obtain the value in the key-value pair to which the target key value belongs from the target file in the target storage module.

[0179] The target storage module can be located in the first device 101 or the second device 102. If the target storage module is located in the first device 101, the first device 101 can directly obtain the value in the key-value pair to which the target key value belongs from the target file in the target storage module. If the target storage module is located in the second device 102, the first device 101 can send a data acquisition request to the second device 102 where the target storage module is located. The data acquisition request carries the module identifier of the target storage module, the file identifier of the target file, and the target key value. After receiving the data acquisition request, the second device 102 can obtain the value in the key-value pair to which the target key value belongs from the target file in the target storage module, and send the value to the first device 101.

[0180] Optionally, after obtaining the value, the first device 101 can return a data query result to an external system. The data query result carries the value.

[0181] Secondly, the following step a to the following step c can be included.

[0182] Step a: The first device 101 receives a second data query request. The second data query request carries a target file identifier.

[0183] The target file identifier identifies a file that needs to be queried. The second data query request is used to request to query the file identified by the target file identifier. The second data query request can be sent by an external system. For example, it can be sent by a database system, a blockchain system, etc., and the embodiments of the present application do not limit this.

[0184] Step b: In the case where the target tree is an n-ary tree, the first device 101 determines the index information in the target tree according to the target file identifier, n, and the height of the target tree.

[0185] The index information in the target tree is information needed when indexing the leaf nodes in the target tree.

[0186] Optionally, assuming that the height of the target tree is m, m is an integer greater than or equal to 2. Then the index information in the target tree can include the index information of each layer in the target tree, that is, it can include the index information of each layer in the 2nd layer to the mth layer.

[0187] The index information of the ith layer is used to indicate the ordering sequence number of the node to be indexed in the ith layer among all child nodes of its parent node, where i is greater than or equal to 2 and less than or equal to m.

[0188] That is, after indexing a node in the i-1th layer, it is necessary to continue to index the node in the ith layer. In this case, the next node can be indexed from all child nodes of the node in the i-1th layer according to the index information of the ith layer. For example, the index information of the 2nd layer is 0, then after indexing the node in the 1st layer, that is, the root node, the child node with the ordering sequence number 0 among all child nodes of the root node can be taken as the next node to be indexed, and the next node is the node in the 2nd layer.

[0189] As an example, the first device 101 can obtain the file identifier in the last leaf node in the target tree according to the pointer of the last leaf node in the target tree maintained in the memory, add 1 to the target value corresponding to the file identifier to obtain the number of leaf nodes of the target tree. In the case where the number of leaf nodes of the target tree and the branching number n of the target tree are known, the height m of the target tree can be determined.

[0190] The target value corresponding to a file identifier is the ordering sequence number of the file identified by the file identifier among all files. The ordering sequence number starts from 0 and increases by 1. That is, the ordering sequence number can be 0, 1, 2, 3, ….

[0191] There is a corresponding relationship between the file identifier and the target value. When the storage layer system increases a new file, the ordering sequence number of the new file among all files can be taken as the target value corresponding to the file identifier of the new file.

[0192] Since the file identifiers are globally ordered, for any file identifier, its corresponding target value can be determined.

[0193] For example, if the file identifiers are sequentially numbered starting from 0, i.e., the file identifiers are 0, 1, 2, 3, …, then a file identifier is the same as its corresponding target value. For another example, if the file identifiers are sequentially numbered starting from 0 with a fixed value, and the fixed value is j, where j is an integer greater than or equal to 2, then the file identifiers are 0, j, 2j, 3j, …. In this case, the target value corresponding to a file identifier is the value obtained by dividing the file identifier by j. Of course, the file identifiers can also be ordered in other forms, such as a, b, c, …. In this case, the file identifiers can be converted into corresponding target values according to certain rules, such as the target value corresponding to the file identifier a is 0, the target value corresponding to the file identifier b is 1, the target value corresponding to the file identifier c is 2, and so on.

[0194] Optionally, the height of the target tree is m, where m is an integer greater than or equal to 2. The operation of step b can be: the first device 101 adds a read lock to the root node of the target tree. The target value corresponding to the target file identifier is determined, where the target value corresponding to the target file identifier is the ordering number of the file identified by the target file identifier in all files. Let i be m, and let the specified parameter be the target value corresponding to the target file identifier. The specified parameter is divided by n to obtain a quotient and a remainder, and the remainder is taken as the index information of the i-th layer. If i is not 2, let i = i-1, let the specified parameter be the quotient, and the above steps of dividing the specified parameter by n to obtain a quotient and a remainder and subsequent steps are re-executed until i is 2; if i is 2, the index root node is indexed, and the read lock of the root node is released after the root node is indexed.

[0195] It should be noted that the embodiments of the present application need to add a read lock to the root node in the process of determining the index information in the target tree. This is because if the case of adding a new root node occurs in the process of determining the index information in the target tree, then the index information determined according to the old root node will start from the new root node, which will cause an error. Therefore, before determining the index information in the target tree, the embodiments of the present application first add a read lock to the root node to avoid modification of the root node by other threads. After the index information in the target tree is determined, the root node is indexed, and the read lock of the root node is released after the root node is indexed. In this way, the root node at the beginning of the indexing process is consistent with the root node when the index information is determined, thereby ensuring the correctness of the indexing process.

[0196] In addition, the embodiments of the present application add a read lock to the root node in the process of determining the index information of the target tree, which does not affect other data indexing processes. That is, different processes of indexing leaf nodes through file identifiers can be completely concurrent.

[0197] It should be noted that the specified parameter is divided by n to obtain a quotient and a remainder. The quotient represents the ordering sequence number of the parent node of the node to be indexed in the i-th layer among all nodes in the i-1-th layer. The remainder is the ordering sequence number of the node to be indexed in the i-th layer among all child nodes of the parent node. Accordingly, after the node in the i-1-th layer is indexed, the node in the i-th layer can be continuously indexed.

[0198] In the embodiment of the present application, the index information in the target tree is determined from the index information in the m-th layer, and then the index information in the m-1-th layer, the index information in the m-2-th layer, and so on, until the index information in the second layer is determined. When indexing, after the root node is indexed, the nodes in the second layer, the third layer, and so on, the m-th layer are sequentially indexed downward layer by layer to the leaf node according to the index information.

[0199] The process of determining the index information in the target tree will be described below with reference to FIG. 5.

[0200] As shown in (b) of FIG. 5, the target tree is a binary tree, and the height of the target tree is 4.

[0201] Suppose the target file identifier corresponds to a target value of 6, as shown in (a) of FIG. 5, first, the index information in the fourth layer is determined. Specifically, the specified parameter is 6, 6 is divided by 2 to obtain a quotient of 3 and a remainder of 0, and then it is determined that the index information in the fourth layer is 0. Then, the index information in the third layer is continuously determined. Specifically, the specified parameter is 3, 3 is divided by 2 to obtain a quotient of 1 and a remainder of 1, and then it is determined that the index information in the third layer is 1. Then, the index information in the second layer is continuously determined. Specifically, the specified parameter is 1, 1 is divided by 2 to obtain a quotient of 0 and a remainder of 1, and then it is determined that the index information in the second layer is 1. Thus, the index information in each layer from the second layer to the fourth layer in the target tree is determined. When indexing according to the index information in each layer from the second layer to the fourth layer, as shown in (b) of FIG. 5, after the node in the first layer, i.e., the root node, is indexed, according to the index information in the second layer, it can be determined that the node to be indexed in the second layer is the child node with an ordering sequence number of 1 among all child nodes of the root node. After the node in the second layer is indexed, according to the index information in the third layer, it can be determined that the node to be indexed in the third layer is the child node with an ordering sequence number of 1 among all child nodes of the node in the second layer. After the node in the third layer is indexed, according to the index information in the fourth layer, it can be determined that the node to be indexed in the fourth layer is the child node with an ordering sequence number of 0 among all child nodes of the node in the third layer. In this way, the corresponding leaf node is indexed.

[0202] Step c: The first device 101 indexes the second leaf node in the target tree according to the index information in the target tree, and the second leaf node is the leaf node containing the target file identifier.

[0203] It should be noted that the index information in the target tree can be used to directly determine the index path from the root node to the second leaf node, and no node data needs to be read in the process, so the process of indexing the second leaf node according to the index information in the target tree does not need to add a read lock.

[0204] In the embodiments of the present application, the index information in the target tree can be determined by calculating m-1 times, and the index path can be directly determined according to the index information in the target tree. In this way, the complexity of the indexing process is low, and the indexing efficiency is high. In this case, after the first device 101 indexes the second leaf node, the first device 101 can obtain the file identified by the target file identifier in the second leaf node (which can be referred to as a target file).

[0205] The first device 101 can determine the storage module where the target file is located (which can be referred to as a target storage module), for example, the first device 101 can determine the target storage module according to the target file identifier and the first correspondence relationship of each storage module in the plurality of storage modules. Specifically, if the target file identifier is included in the first correspondence relationship of a certain storage module, the storage module can be determined as the target storage module. Then, the first device 101 can obtain the target file from the target storage module.

[0206] The target storage module can be located in the first device 101 or the second device 102. If the target storage module is located in the first device 101, the first device 101 can directly obtain the target file from the target storage module; if the target storage module is located in the second device 102, the first device 101 can send a data acquisition request to the second device 102 where the target storage module is located. The data acquisition request carries the module identifier of the target storage module and the file identifier of the target file. After receiving the data acquisition request, the second device 102 can obtain the target file from the target storage module and send the target file to the first device 101.

[0207] Optionally, after obtaining the target file, the first device 101 can return a data query result to an external system, and the data query result carries the target file.

[0208] The process of the first device 101 performing file migration between the plurality of storage modules in the file migration method provided by the embodiments of the present application will be explained in detail below.

[0209] FIG. 6 is a flowchart of a file migration method according to an embodiment of the present application. Referring to FIG. 6, the method can include the following steps:

[0210] Step 601: The first device 101 generates a new file according to the received data storage request, and stores the new file into a first storage module, which is a storage module ranked first among the plurality of storage modules.

[0211] The operation of step 601 has been described in detail above, and will not be repeated here.

[0212] After storing the new file into the first storage module, the first device 101 can add the file identifier of the new file into the first correspondence relationship of the first storage module, and set the corresponding hit count of the file identifier of the new file in the first correspondence relationship to 0.

[0213] For example, the first device 101 can add the file identifier of the new file after the last element in the file identifier list in the first correspondence relationship of the first storage module. In addition, the first device 101 can add an element after the last element in the hit count list in the first correspondence relationship of the first storage module, and set the added element to 0.

[0214] It should be noted that in the case where the first correspondence relationship adopts the form of the file identifier list and the hit count list, the read-write efficiency of the first correspondence relationship is relatively high. In addition, since the file identifier list and the hit count list are static within a version range, the reading and modification of the internal elements thereof can use atomic operations, without using lock structures, so as to reduce the overhead.

[0215] Step 602: The first device 101 updates the first correspondence relationship and the second correspondence relationship of one storage module among the plurality of storage modules according to the received data query request.

[0216] The first correspondence relationship includes the hit count of each file in the storage module, and the second correspondence relationship includes the number of files in the storage module whose hit count is in each preset hit count interval among a plurality of preset hit count intervals.

[0217] The data query request can be the first data query request or the second data query request described above. The process of the first device 101 performing data query after receiving the data query request has been described in detail above, and will not be repeated here.

[0218] In some embodiments, the operation of step 602 can be that the first device 101 receives the data query request, and if a target file is found in the target storage module according to the data query request, determines a first hit count corresponding to the file identifier of the target file in the first correspondence relationship of the target storage module as a first hit count, adds 1 to the hit count corresponding to the file identifier of the target file in the first correspondence relationship of the target storage module to obtain a second hit count; and the first device 101 updates the second correspondence relationship of the target storage module according to the first hit count and the second hit count.

[0219] For example, in the case where the data query request is the first data query request, the target file is a file containing a key-value pair to which the target key value carried by the first data query request belongs, and the target storage module is a storage module storing the target file.

[0220] In the case where the data query request is the second data query request, the target file is a file identified by the target file identifier carried by the second data query request, and the target storage module is a storage module storing the target file.

[0221] The first hit count is the hit count corresponding to the file identifier of the target file in the first correspondence relationship of the target storage module before the target storage module is updated.

[0222] The second hit count is the hit count corresponding to the file identifier of the target file in the first correspondence relationship of the target storage module after the target storage module is updated.

[0223] For example, if the first device 101 finds the target file in the target storage module according to the data query request, it can first find the file identifier of the target file in the file identifier list in the first correspondence relationship of the target storage module by means of binary search according to the file identifier of the target file, then obtains the subscript of the file identifier in the file identifier list, and then obtains the hit count at the same subscript position in the hit count list in the first correspondence relationship of the target storage module as the first hit count, and then adds 1 to the hit count at the same subscript position in the hit count list to obtain the second hit count.

[0224] In some embodiments, the first device 101 can update the second correspondence relationship of the target storage module according to the first hit number and the second hit number. If the preset hit number interval where the first hit number is located is different from the preset hit number interval where the second hit number is located, the first device 101 can decrease the file number corresponding to the preset hit number interval where the first hit number is located in the second correspondence relationship of the target storage module by 1, and increase the file number corresponding to the preset hit number interval where the second hit number is located in the second correspondence relationship of the target storage module by 1. If the preset hit number interval where the first hit number is located is the same as the preset hit number interval where the second hit number is located, the first device 101 does not update the second correspondence relationship of the target storage module.

[0225] In some embodiments, the first device 101 can also update the third correspondence relationship of the target storage module according to the first hit number and the second hit number. The third correspondence relationship includes the total hit number of the files in the target storage module that are in each of the preset hit number intervals.

[0226] Specifically, if the preset hit number interval where the first hit number is located is different from the preset hit number interval where the second hit number is located, the first device 101 can decrease the total hit number corresponding to the preset hit number interval where the first hit number is located in the third correspondence relationship of the target storage module by the first hit number, and increase the total hit number corresponding to the preset hit number interval where the second hit number is located in the third correspondence relationship of the target storage module by the second hit number. If the preset hit number interval where the first hit number is located is the same as the preset hit number interval where the second hit number is located, the first device 101 can increase the total hit number corresponding to the preset hit number interval where the first hit number is located in the third correspondence relationship of the target storage module by 1.

[0227] For example, it is assumed that the preset hit number intervals are [1, 4], [5, 24], [25, 124], and [125, +∞) respectively. The file identifier list in the first correspondence relationship of the storage module 1 is {000001, 000002, 000003, 000004}, and the hit number list is {4, 6, 25, 11}. The second correspondence relationship of the storage module 1 is {1, 2, 1, 0}. The third correspondence relationship of the storage module 1 is {4, 17, 25, 0}.

[0228] If the first device 101 queries the file 1 in the storage module 1 according to the received data query request, and the file identifier of the file 1 is 000001, the first device 101 can update the first correspondence relationship, the second correspondence relationship, and the third correspondence relationship of the storage module 1 accordingly.

[0229] Since the index of the file identifier 000001 in the file identifier list {000001, 000002, 000003, 000004} is 1, the first device 101 can determine the hit number 4 at the index 1 position in the hit number list {4, 6, 25, 11} as the first hit number, and then add 1 to the hit number 4 at the index 1 position in the hit number list {4, 6, 25, 11} to obtain the updated hit number list {5, 6, 25, 11}, that is, the second hit number is 5.

[0230] Since the preset hit number interval in which the first hit number 4 is located is [1, 4] and the preset hit number interval in which the second hit number 5 is located is [5, 24], the two are different, so the first device 101 can reduce 1 from the file quantity 1 in the second correspondence relation {1, 2, 1, 0} corresponding to the preset hit number interval [1, 4], and increase 1 from the file quantity 2 in the second correspondence relation {1, 2, 1, 0} corresponding to the preset hit number interval [5, 24], to obtain the updated second correspondence relation {0, 3, 1, 0}. Moreover, the first device 101 can reduce the total hit number 4 in the third correspondence relation {4, 17, 25, 0} corresponding to the preset hit number interval [1, 4] by the first hit number 4, and increase the total hit number 17 in the third correspondence relation {4, 17, 25, 0} corresponding to the preset hit number interval [5, 24] by the second hit number 5, to obtain the updated third correspondence relation {0, 22, 25, 0}.

[0231] Step 603: The first device 101 determines, according to the first correspondence relation and the second correspondence relation of the second storage module, a file migration list of the second storage module every preset time length, the second storage module being any one of the plurality of storage modules except the storage module ranked last, and the file migration list being used to indicate files that need to be migrated by the second storage module to a third storage module, the third storage module being the storage module ranked next to the second storage module in the plurality of storage modules.

[0232] The preset time length can be set in advance. For example, the preset time length can be set to 2 minutes, 3 minutes, 4 minutes, etc., which is not limited in the embodiments of the present application.

[0233] The file migration list of the second storage module is used to record file identifiers of files that need to be migrated by the second storage module.

[0234] Since the first correspondence relation of the second storage module includes the hit times of each file in the second storage module, and the second correspondence relation of the second storage module includes the file quantity in the second storage module with the hit times in each preset hit time interval, the first correspondence relation and the second correspondence relation can reflect the file distribution of the second storage module in each preset hit time interval, and thus it can be analyzed that which file with the hit time in the second storage module belongs to hot data compared with other files, and which file with the hit time in the second storage module belongs to cold data compared with other files, so that the file migration list of the second storage module can be determined, and the file migration list is used to indicate the cold data that needs to be migrated to the next storage module.

[0235] In some embodiments, the operation of step 603 can include steps 6031 to 6033 as follows:

[0236] Step 6031: The first device 101 determines the first file quantity and the second file quantity according to the first correspondence relation and the second correspondence relation of the second storage module, the first file quantity is the quantity of files with the hit time of 0 in the second storage module, and the second file quantity is the maximum file quantity in the second correspondence relation of the second storage module.

[0237] The first device 101 can obtain the total file quantity of the second storage module according to the first correspondence relation of the second storage module, and then subtract all the file quantities in the second correspondence relation of the second storage module from the total file quantity of the second storage module, so as to obtain the quantity of files with the hit time of 0 in the second storage module, that is, the first file quantity.

[0238] Step 6032: If the first file quantity is greater than or equal to the second file quantity, the first device 101 adds the file identifier corresponding to the file with the hit time of 0 in the first correspondence relation of the second storage module to the file migration list of the second storage module.

[0239] If the first file quantity is greater than or equal to the second file quantity, it indicates that there are more files with the hit time of 0 in the second storage module, and the files with the hit time of 0 belong to cold data, so the file identifiers of these files are added to the file migration list. So that these cold data in the second storage module can be migrated to the next storage module according to the file migration list in the subsequent process.

[0240] The first device 101 can determine the files that need to be migrated by traversing the first correspondence relation of the second storage module, that is, determine the file identifiers that need to be added to the file migration list of the second storage module.

[0241] In some embodiments, in the case that the current file migration is the first file migration of the second storage module, the first device 101 can start the traversal from the first element in the hit frequency list in the first correspondence relationship, or can randomly select an element in the hit frequency list to start the traversal. After starting the traversal, if the first device 101 determines that a certain element in the hit frequency list is 0, the index of the element can be determined, and the element at the same index position in the file identifier list in the first correspondence relationship of the second storage module is added to the file migration list. In this case, the first device 101 can complete the traversal of all elements in the hit frequency list to add the file identifiers of all files in the second storage module with a hit frequency of 0 to the file migration list.

[0242] In some embodiments, in the case that the current file migration is not the first file migration of the second storage module, the first device 101 can estimate the number of files to be migrated this time (which can be referred to as a target migration number) according to the number of newly added files of the second storage module during the last file migration to the current file migration and / or the historical migration number of the second storage module (i.e., the number of files migrated by the second storage module to the third storage module each time before). For example, the second storage module can obtain the target migration number by weighted averaging the number of newly added files and the historical migration number. Of course, the second storage module can also determine the target migration number by other ways according to the number of newly added files and / or the historical migration number, which is not limited in the embodiments of the present application.

[0243] The target migration number is the number of files to maintain the migration balance of the second storage module, so that the second storage module can maintain good storage performance.

[0244] Optionally, if the first file number is greater than or equal to the target migration number, the first device 101 can start the traversal from the first element in the hit frequency list in the first correspondence relationship of the second storage module, or can randomly select an element in the hit frequency list to start the traversal, or can start the traversal from the next element of the position where the last traversal ends in the hit frequency list. After starting the traversal, if the first device 101 determines that a certain element in the hit frequency list is 0, the index of the element can be determined, and the element at the same index position in the file identifier list in the first correspondence relationship of the second storage module is added to the file migration list of the second storage module. At the same time, if the first device 101 determines that the number of file identifiers added to the file migration list reaches the target migration number, the traversal is ended.

[0245] If the first file quantity is less than the target migration quantity, the first device 101 can start traversing from the first element in the hit frequency list in the first correspondence relationship of the second storage module, or can randomly select an element in the hit frequency list to start traversing, or can start traversing from the next element of the position where the last traversal ends in the hit frequency list. After starting the traversal, if the first device 101 determines that a certain element traversed in the hit frequency list is 0, it can determine the index of the element, and add the element at the same index position in the file identification list in the first correspondence relationship of the second storage module to the file migration list of the second storage module. In this case, the first device 101 needs to complete the traversal of all elements in the hit frequency list to add the file identification of all files with a hit frequency of 0 in the second storage module to the file migration list.

[0246] Step 6033: If the first file quantity is less than the second file quantity, the first device 101 determines a first hit frequency interval, which is the preset hit frequency interval corresponding to the largest file quantity in the second correspondence relationship of the second storage module. The first device 101 determines at least one second hit frequency interval according to the first hit frequency interval, which is all preset hit frequency intervals before the first hit frequency interval in the second correspondence relationship of the second storage module. The first device 101 adds the file identification in the first correspondence relationship of the second storage module with a corresponding hit frequency within the at least one second hit frequency interval to the file migration list of the second storage module, and adds the file identification in the first correspondence relationship of the second storage module with a hit frequency of 0 to the file migration list.

[0247] The number of files in the second storage module with a hit frequency within the first hit frequency interval is relatively large, so for the second storage module, it can be determined that the files with a hit frequency within the first hit frequency interval belong to hot data. Moreover, the hit frequencies in the preset hit frequency intervals after the first hit frequency interval in the second correspondence relationship of the second storage module are relatively large, so it can be determined that the files with a hit frequency within these preset hit frequency intervals also belong to hot data. The hit frequencies within the at least one second hit frequency interval are relatively small, and the number of files with a hit frequency within each second hit frequency interval is also relatively small, so it can be determined that these files belong to cold data. Therefore, the file identification of the files in the second storage module with a hit frequency within the at least one second hit frequency interval can be added to the file migration list. At the same time, the files with a hit frequency of 0 also belong to cold data, so the file identification of these files is also added to the file migration list. In order to subsequently migrate these cold data in the second storage module to the next storage module according to the file migration list.

[0248] In some embodiments, after the first device 101 determines the at least one second hit frequency interval, if this file migration is the first file migration of the second storage module, the first device 101 can directly determine the file migration list of the second storage module according to the at least one second hit frequency interval.

[0249] For example, after the first device 101 determines the at least one second hit frequency interval, the first device 101 can determine the files that need to be migrated, i.e., the file identifiers that need to be added to the file migration list of the second storage module, by traversing the first correspondence relationship of the second storage module. The first device 101 can start traversing from the first element in the hit frequency list in the first correspondence relationship of the second storage module, or can randomly select an element in the hit frequency list to start traversing. After starting the traversal, if the first device 101 determines that a certain element traversed in the hit frequency list is 0 or the element is in the at least one second hit frequency interval, the first device 101 can determine the index of the element, and add the element at the same index position in the file identifier list in the first correspondence relationship of the second storage module to the file migration list. In this case, the first device 101 can complete the traversal of all elements in the hit frequency list to add the file identifiers of all files with a hit frequency of 0 in the second storage module to the file migration list.

[0250] In some embodiments, after the first device 101 determines the at least one second hit frequency interval, if this file migration is not the first file migration of the second storage module, and if the sum of the number of files corresponding to each second hit frequency interval in the at least one second hit frequency interval is greater than or equal to the target migration number, the first device 101 can start traversing from the first element in the hit frequency list in the first correspondence relationship of the second storage module, or can randomly select an element in the hit frequency list to start traversing, or can start traversing from the next element of the position where the last traversal ended in the hit frequency list. After starting the traversal, if the first device 101 determines that a certain element traversed in the hit frequency list is 0 or the element is in the at least one second hit frequency interval, the first device 101 can determine the index of the element, and add the element at the same index position in the file identifier list in the first correspondence relationship of the second storage module to the file migration list of the second storage module. Meanwhile, if the first device 101 determines that the number of file identifiers added to the file migration list reaches the target migration number, the traversal is ended.

[0251] Optionally, after the first device 101 determines the at least one second hit frequency interval, if the sum of the file quantities corresponding to each second hit frequency interval in the at least one second hit frequency interval is less than the target migration quantity, the first device 101 can divide the total hit frequency corresponding to the first hit frequency interval in the third correspondence relationship of the second storage module by the file quantity corresponding to the first hit frequency interval in the second correspondence relationship of the second storage module to obtain a target hit frequency, and add the file identifiers corresponding to the hit frequencies in the first correspondence relationship of the second storage module that are within the first hit frequency interval and less than the target hit frequency to the file migration list of the second storage module.

[0252] The target hit frequency can reflect the average level of the hit frequencies of the files in the second storage module that have hit frequencies within the first hit frequency interval, and the files with hit frequencies less than the average level are also relatively cold data to some extent, so the file identifiers of these files can also be added to the file migration list.

[0253] For example, after the first device 101 determines the at least one second hit frequency interval and the target hit frequency, the first device 101 can start traversing from the first element in the hit frequency list in the first correspondence relationship of the second storage module, or can start traversing randomly in the hit frequency list, or can start traversing from the next element of the position where the traversal ends last time in the hit frequency list. After starting the traversal, if the first device 101 determines that a certain element in the hit frequency list is 0 or the element is within the at least one second hit frequency interval or the element is within the first hit frequency interval and less than the target hit frequency, the first device 101 can determine the index of the element, add the element at the same index position in the file identifier list in the first correspondence relationship of the second storage module to the file migration list of the second storage module. Meanwhile, if the first device 101 determines that the number of file identifiers added to the file migration list reaches the target migration quantity, the traversal ends.

[0254] In some embodiments, the first device 101 needs to determine the file migration list of the second storage module every preset time length in step 603, and can first intercept the hit frequency updating behavior through state switching, and then cancel the interception after determining the file migration list, so as to ensure that the management data in the first correspondence relationship, the second correspondence relationship and the third correspondence relationship of the second storage module do not change during the process of determining the file migration list according to the first correspondence relationship, the second correspondence relationship and the third correspondence relationship of the second storage module, so that the concurrent security problem of the read-write behavior does not need to be considered during the process.

[0255] In some embodiments, after determining the file migration list of the second storage module, the first device 101 can further update the first correspondence relationship, the second correspondence relationship and the third correspondence relationship of the second storage module.

[0256] For the second correspondence relationship and the third correspondence relationship of the second storage module, after determining the file migration list of the second storage module, the first device 101 can set the number of files corresponding to each of the plurality of preset hit number intervals in the second correspondence relationship of the second storage module to 0, and set the total number of hits corresponding to each of the plurality of preset hit number intervals in the third correspondence relationship of the second storage module to 0.

[0257] As an example, in the case where the second storage module is the storage module ranked first among the plurality of storage modules, the first device 101 can delete the file identifier (which can be referred to as a first specified file identifier) in the file migration list of the second storage module and the hit number corresponding to the first specified file identifier from the first correspondence relationship of the second storage module.

[0258] For example, for the file identifier list in the first correspondence relationship, since the file identifiers in the file identifier list are ordered, a new file identifier list can be quickly obtained by performing incremental modification based on the existing file identifier list. Specifically, the first device 101 can first apply for a memory space sufficient for the new file identifier list, and then for the old file identifier list, the position of the first specified file identifier to be deleted can be located by binary search, and the remaining file identifiers are copied into the new file identifier list by segmenting. In this process, only one memory space needs to be applied for, so the memory overhead can be saved. Moreover, since the new file identifier list is completely ordered throughout the entire processing process, the complexity of the deletion operation is log(N), which is relatively low.

[0259] After obtaining the new file identifier list described above, a new hit number list is constructed, in which the hit numbers are all 0. The number of elements in the new hit number list is the same as the number of elements in the new file identifier list, and the elements at the same index position in the new hit number list and the new file identifier list correspond.

[0260] As another example, in the case that the second storage module is not the first one in the ranking of the plurality of storage modules, the second storage module can delete the file identifier (which can be referred to as the first specified file identifier) in the file migration list of the second storage module and the hit count corresponding to the first specified file identifier from the first correspondence relation of the second storage module, and add the file identifier (which can be referred to as the second specified file identifier) in the file migration list of the fourth storage module and the hit count corresponding to the second specified file identifier in the first correspondence relation, the fourth storage module being the one next to the second storage module in the ranking of the plurality of storage modules.

[0261] For example, for the file identifier list in the first correspondence relation, the first device 101 can first apply a memory space sufficient for the new file identifier list, and then for the old file identifier list, locate the position of the first specified file identifier to be deleted by means of binary search, skip the first specified file identifier, and segment and copy the remaining file identifiers to the new file identifier list, and then insert the second specified file identifier into the new file identifier list in sequence. In this process, only one memory space needs to be applied, so the memory overhead can be saved. Moreover, since the new file identifier list is completely ordered in the entire processing process, the complexity of the deletion and insertion operations is log(N), which is relatively low.

[0262] After obtaining the new file identifier list, a new hit count list is constructed, the hit counts in the new hit count list being all 0. The number of elements in the new hit count list is the same as the number of elements in the new file identifier list, and the elements at the same index position in the new hit count list and the new file identifier list correspond to each other.

[0263] Step 604: The first device 101 migrates the files indicated by the file migration list from the second storage module to the third storage module through the device where the second storage module is located and the device where the third storage module is located.

[0264] In the embodiments of the present application, the plurality of storage modules are sorted in the order from high to low of read-write performance, the first device 101 can constantly migrate the files in each of the plurality of storage modules to the next storage module, so as to ensure that the first storage module always has capacity to store new files. Moreover, since the first device 101 performs file migration according to the file migration list of each storage module, and the file migration list of a storage module indicates that the file in the storage module is cold data, the hot and cold data separation can be achieved through file migration, so that the hot data is kept in the storage module with higher read-write performance as much as possible, and the cold data is migrated to the storage module with lower read-write performance as much as possible. In this way, the read-write efficiency of the hot data can be improved and the read-write delay of the hot data can be reduced while achieving distributed storage of files.

[0265] For example, if the device where the second storage module is located and the device where the third storage module is located are both the first device 101, the first device 101 can directly migrate the files indicated by the file migration list from the second storage module to the third storage module.

[0266] For example, if the device where the second storage module is located and the device where the third storage module is located are both the second device 102, the first device 101 can send a file migration request to the second device 102, the file migration request carrying the module identifier of the second storage module, the file migration list, and the module identifier of the third storage module, to indicate that the files indicated by the file migration list are migrated from the second storage module to the third storage module. After receiving the file migration request, the second device 102 can migrate the files indicated by the file migration list from the second storage module to the third storage module.

[0267] For example, if the device where the second storage module is located is the first device 101 and the device where the third storage module is located is the second device 102, the first device 101 can obtain the files indicated by the file migration list from the second storage module, and send the files and the module identifier of the third storage module in a file migration message to the second device 102. After receiving the file migration message, the second device 102 can store the files carried in the file migration message to the third storage module, and then send a migration completion message to the first device 101. After receiving the migration completion message, the first device 101 can delete the files indicated by the file migration list from the second storage module.

[0268] For example, if the device where the second storage module is located is the second device 102 and the device where the third storage module is located is the first device 101, the first device 101 can send a file migration request to the second device 102, where the file migration request carries the module identifier of the second storage module, the file migration list, and the device identifier of the first device 101 and the module identifier of the third storage module, to instruct the second device 102 to migrate the files indicated by the file migration list from the second storage module to the third storage module in the first device 101. After receiving the file migration request, the second device 102 can obtain the files indicated by the file migration list from the second storage module, and send the files and the module identifier of the third storage module to the first device 101 in a file migration message. After receiving the file migration message, the first device 101 can store the files carried by the file migration message to the third storage module, and then send a migration completion message to the second device 102. After receiving the migration completion message, the second device 102 can delete the files indicated by the file migration list from the second storage module.

[0269] For example, if the device where the second storage module is located and the device where the third storage module is located are different second devices 102, assuming that the device where the second storage module is located is the second device A and the device where the third storage module is located is the second device B, the first device 101 can send a file migration request to the second device A, where the file migration request carries the module identifier of the second storage module, the file migration list, and the device identifier of the second device B and the module identifier of the third storage module, to instruct the second device A to migrate the files indicated by the file migration list from the second storage module to the third storage module in the second device B. After receiving the file migration request, the second device A can obtain the files indicated by the file migration list from the second storage module, and send the files and the module identifier of the third storage module to the second device B in a file migration message. After receiving the file migration message, the second device B can store the files carried by the file migration message to the third storage module, and then send a migration completion message to the second device A. After receiving the migration completion message, the second device A can delete the files indicated by the file migration list from the second storage module.

[0270] In the embodiments of the present application, the first device 101 generates a new file according to the received data storage request, and stores the new file to a first storage module, the first storage module being a storage module ranked first in a plurality of storage modules, the plurality of storage modules being ranked in order from high to low in terms of read-write performance. In addition, the first device 101 updates a first correspondence relationship and a second correspondence relationship of a storage module in the plurality of storage modules according to the received data query request, the first correspondence relationship including a hit frequency of each file in the storage module, and the second correspondence relationship including a number of files in the storage module with a hit frequency in each of a plurality of preset hit frequency intervals. The first device 101 determines a file migration list of a second storage module according to the first correspondence relationship and the second correspondence relationship of the second storage module every preset time length, and then migrates files in the file migration list from the second storage module to a third storage module through a device where the second storage module is located and a device where the third storage module is located. In this way, the first device 101 can continuously migrate files in each storage module in the plurality of storage modules to a next storage module, so as to ensure that the first storage module always has capacity to store new files. In addition, since the first device 101 migrates files according to the file migration list of each storage module, and the file migration list of a storage module indicates that the files in the storage module belong to cold data, the hot and cold data separation can be achieved through file migration, so that the hot data is kept in the storage module with higher read-write performance as much as possible, and the cold data is migrated to the storage module with lower read-write performance as much as possible. In this way, the read-write efficiency of the hot data can be improved and the read-write delay of the hot data can be reduced while achieving distributed storage of files.

[0271] FIG. 7 is a structural schematic diagram of a file migration apparatus provided in an embodiment of the present application. The apparatus can be realized by software, hardware or a combination of both as part or all of a computer device, and the computer device can be the computer device shown in FIG. 8. Referring to FIG. 7, the apparatus includes a generating module 701, an updating module 702, a determining module 703 and a migrating module 704.

[0272] The generating module 701 is configured to generate a new file according to the received data storage request, and store the new file to a first storage module, the first storage module being a storage module ranked first among the plurality of storage modules; the updating module 702 is configured to update a first correspondence relationship and a second correspondence relationship of one storage module among the plurality of storage modules according to the received data query request, the first correspondence relationship including a hit frequency of each file in the storage module, and the second correspondence relationship including a quantity of files in the storage module with a hit frequency in each preset hit frequency interval among a plurality of preset hit frequency intervals; the determining module 703 is configured to determine a file migration list of a second storage module according to the first correspondence relationship and the second correspondence relationship of the second storage module every preset time length, the second storage module being any one of the plurality of storage modules except a storage module ranked last, the file migration list being used to indicate files that need to be migrated from the second storage module to a third storage module, the third storage module being a storage module ranked next to the second storage module among the plurality of storage modules; and the migration module 704 is configured to migrate the files indicated by the file migration list from the second storage module to the third storage module through a device where the second storage module is located and a device where the third storage module is located.

[0273] In the embodiments of the present application, the plurality of storage modules are ranked in order of read-write performance from high to low. The first device can constantly migrate files in each storage module among the plurality of storage modules to the next storage module, so as to ensure that the first storage module always has capacity to store new files. Moreover, since the first device performs file migration according to the file migration list of each storage module, and the files indicated by the file migration list of a storage module belong to cold data in the storage module, the cold and hot data separation can be achieved through file migration, so that the hot data is kept in the storage module with higher read-write performance as much as possible, and the cold data is migrated to the storage module with lower read-write performance as much as possible. In this way, the read-write efficiency of the hot data can be improved and the read-write delay of the hot data can be reduced while achieving distributed storage of files.

[0274] FIG. 8 is a structural schematic diagram of a computer device according to an embodiment of the present application. As shown in FIG. 8, the computer device 8 includes a processor 80, a memory 81, and a computer program 82 stored in the memory 81 and executable on the processor 80, and the processor 80 implements the operations performed by the first device in the file migration method in the above embodiments when executing the computer program 82.

[0275] The computer device 8 can be a general purpose computer device or a special purpose computer device. In specific implementations, the computer device 8 can be a desktop computer, a laptop computer, a network server, a palmtop computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device, and the embodiments of the present application do not limit the type of the computer device 8. Those skilled in the art can understand that FIG. 8 is merely an example of the computer device 8, and does not limit the computer device 8, which can include more or less components than shown, or combine some components, or different components, such as an input / output device, a network access device, and the like.

[0276] The processor 80 can be a central processing unit (CPU), and can also be other general purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc. The general purpose processor can be a microprocessor or any conventional processor.

[0277] The memory 81 can be an internal storage unit of the computer device 8, such as a hard disk or a memory of the computer device 8 in some embodiments. The memory 81 can also be an external storage device of the computer device 8, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 8 in other embodiments. Further, the memory 81 can include both the internal storage unit and the external storage device of the computer device 8. The memory 81 is used to store an operating system, application programs, a boot loader, data, and other programs, etc. The memory 81 can also be used to temporarily store data that has been output or will be output.

[0278] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above method embodiments can be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc. The computer readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, a non-transitory storage medium.

[0279] It should be understood that all or part of the steps of the above-mentioned embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. The computer instructions can be stored in the computer readable storage medium described above.

Claims

1. A file migration method, characterized in that, The method is applied to a first device in a storage layer system, the storage layer system including the first device, one or more second devices, and multiple storage modules, wherein any one of the multiple storage modules is located in the first device or a second device, and the multiple storage modules are ordered in descending order of read / write performance. The method includes: The first device generates a new file based on the received data storage request and stores the new file in the first storage module, which is the first storage module in the order of the plurality of storage modules; The first device updates the first and second correspondence relationships of one of the plurality of storage modules according to the received data query request. The first correspondence relationship includes the number of hits for each file in the storage module, and the second correspondence relationship includes the number of files in the storage module whose hit counts fall within each of the plurality of preset hit count intervals. Every preset time interval, the first device determines the file migration list of the second storage module according to the first correspondence and the second correspondence of the second storage module. The second storage module is any one of the plurality of storage modules except the last one in the order. The file migration list is used to indicate the files that the second storage module needs to migrate to the third storage module. The third storage module is the storage module that is next in the order of the second storage module in the plurality of storage modules. The first device migrates the files indicated by the file migration list from the second storage module to the third storage module through the devices where the second storage module is located and the third storage module is located.

2. The method as described in claim 1, characterized in that, The data storage request carries key-value pairs to be stored, wherein the key values ​​in the key-value pairs are globally ordered. The first device generates a new file based on the received data storage request, including: After receiving the data storage request, the first device generates a new file containing one or more key-value pairs according to the order of the key values ​​in the key-value pairs; wherein, the file identifiers of all files containing the key-value pairs are globally ordered.

3. The method as described in claim 2, characterized in that, After the first device stores the new file to the first storage module, the method further includes: the first device adding a new leaf node after the last leaf node in the target tree, and updating the target tree according to the new leaf node, wherein the new leaf node contains the file identifier of the new file, the minimum key value and the maximum key value among all key-value pairs in the new file.

4. The method as described in claim 1, characterized in that, The first device updates the first and second correspondence relationships of one of the plurality of storage modules according to the received data query request, including: After receiving the data query request, if the first device finds a target file in the target storage module among the multiple storage modules according to the data query request, it determines the number of hits corresponding to the file identifier of the target file in the first correspondence of the target storage module as the first hit count, and increments the number of hits corresponding to the file identifier of the target file in the first correspondence of the target storage module by 1 to obtain the second hit count; the first device updates the second correspondence of the target storage module according to the first hit count and the second hit count.

5. The method as described in claim 4, characterized in that, The first correspondence includes a file identifier list and a hit count list. The number of elements in the file identifier list is the same as the number of elements in the hit count list. The file identifier list corresponds to the elements at the same index position in the hit count list.

6. The method as described in claim 4, characterized in that, The first device updates the second correspondence of the target storage module based on the first hit count and the second hit count, including: if the preset hit count interval in which the first hit count is located is different from the preset hit count interval in which the second hit count is located, then the first device decrements the number of files in the second correspondence of the target storage module corresponding to the preset hit count interval in which the first hit count is located by 1, and increments the number of files in the second correspondence of the target storage module corresponding to the preset hit count interval in which the second hit count is located by 1.

7. The method as described in any one of claims 1 to 6, characterized in that, The first device determines the file migration list of the second storage module based on the first and second correspondences of the second storage module, including: the first device determines the number of first files and the number of second files based on the first and second correspondences of the second storage module, wherein the number of first files is the number of files in the second storage module with a hit count of 0, and the number of second files is the largest number of files in the second correspondence of the second storage module; if the number of first files is greater than or equal to the number of second files, the first device adds the file identifiers with a hit count of 0 corresponding to the first correspondence of the second storage module to the file migration list.

8. The method as described in claim 7, characterized in that, After the first device determines the number of the first file and the number of the second file based on the first correspondence and the second correspondence of the second storage module, it further includes: If the number of the first files is less than the number of the second files, the first device determines a first hit count interval, which is a preset hit count interval corresponding to the largest number of files in the second correspondence of the second storage module; the first device determines at least one second hit count interval based on the first hit count interval, which is all preset hit count intervals in the second correspondence of the second storage module that are located before the first hit count interval; the first device adds the file identifiers in the first correspondence of the second storage module whose hit count is within the at least one second hit count interval to the file migration list, and adds the file identifiers in the first correspondence of the second storage module whose hit count is 0 to the file migration list.

9. The method as described in claim 8, characterized in that, The method further includes: The first device updates the third correspondence of one of the plurality of storage modules according to the received data query request. The third correspondence includes the total number of hits of files in the storage module that are in each of the plurality of preset hit count intervals. After the first device determines at least one second hit interval based on the first hit interval, it further includes: If the sum of the number of files corresponding to each of the at least one second hit count intervals is less than the target migration number, then the first device divides the total number of hits corresponding to the first hit count interval in the third correspondence of the second storage module by the number of files corresponding to the first hit count interval in the second correspondence of the second storage module to obtain the target hit count, and adds the file identifiers whose hit counts in the first correspondence of the second storage module are within the first hit count interval and less than the target hit count to the file migration list.

10. The method as described in claim 9, characterized in that, The first device updates the third correspondence relationship of one of the plurality of storage modules according to the received data query request, including: After receiving the data query request, if the first device finds the target file in the target storage module among the multiple storage modules according to the data query request, then the number of hits corresponding to the file identifier of the target file in the first correspondence of the target storage module is determined as the first hit count, and the number of hits corresponding to the file identifier of the target file in the first correspondence of the target storage module is incremented by 1 to obtain the second hit count; If the preset hit interval where the first hit count is located is different from the preset hit interval where the second hit count is located, then the first device reduces the total number of hits in the third correspondence of the target storage module corresponding to the preset hit interval where the first hit count is located by the first hit count, and increases the total number of hits in the third correspondence of the target storage module corresponding to the preset hit interval where the second hit count is located by the second hit count. If the preset hit count interval where the first hit count is located is the same as the preset hit count interval where the second hit count is located, then the first device will increment the total number of hits in the third correspondence of the target storage module corresponding to the preset hit count interval where the first hit count is located by 1.

11. A file migration device, characterized in that, The device is applied to a first device in a storage layer system, the storage layer system including the first device, one or more second devices, and multiple storage modules, any one of the multiple storage modules being located in the first device or a second device, the multiple storage modules being ordered in descending order of read / write performance, the device comprising: The generation module is used to generate a new file according to the received data storage request and store the new file in the first storage module, which is the first storage module in the order of the plurality of storage modules; The update module is used to update the first correspondence and the second correspondence of one of the plurality of storage modules according to the received data query request. The first correspondence includes the number of hits for each file in the storage module, and the second correspondence includes the number of files in the storage module whose hit count falls within each of the plurality of preset hit count intervals. A determining module is used to determine a file migration list for the second storage module at preset intervals based on the first correspondence and the second correspondence of the second storage module. The second storage module is any one of the plurality of storage modules except the last one in the order. The file migration list is used to indicate the files that the second storage module needs to migrate to the third storage module. The third storage module is the storage module that is next in the order of the second storage module in the plurality of storage modules. The migration module is used to migrate the files indicated by the file migration list from the second storage module to the third storage module via the devices where the second storage module is located and the third storage module is located.

12. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 10.

Images