Data dump method and device, storage medium and electronic device

Abstract

The present application discloses a data dump method and device, a storage medium and an electronic device. Relating to the field of data processing technology, the method comprises: if the current state of the target system meets the preset conditions for full data dump, scanning multiple first adaptive radix trees to obtain first data information; scanning multiple first associated containers to obtain second data information, the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; dumping the first data information and the second data information to the target storage medium. Through the present application, the problem that the layered storage technology based on LSM tree in the related technology stores the metadata information of the data block, and when dumping the data, the metadata information in each layer is searched from top to bottom in turn, resulting in a relatively low efficiency of data dump is solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data dump method and device, a storage medium, and an electronic device. Background Art

[0002] A distributed block storage system is a storage system that stores data in multiple storage servers and forms a virtual storage device with these distributed storage resources, providing block (read and write by location) semantics. The metadata information of a block is the conversion of the logical address of the volume to the underlying physical storage address, which needs to be queried during each read and write io (Input / Output, data input / output operation) process and updated during the write io process. In the prior art, metadata information is stored and managed through the layered storage technology of the LSM tree. However, the data read by the LSM tree may be in any layer, so it may be necessary to search from top to bottom to see whether the target data is contained in each layer, which may cause a very large delay in the read operation, and the two layers of data information will be read for inspection and comparison before de-duplication and merging. Generally, it is set to be processed in other threads independent of the business io thread, which will bring about a very large read and write amplification, and introduce additional thread resources and CPU (Central Processing Unit) consumption.

[0003] Regarding the problem that the metadata information of data blocks is stored in the layered storage technology based on LSM tree in the related technology, when dumping data, the metadata information in each layer is searched from top to bottom, resulting in low efficiency of data dumping. No effective solution has been proposed yet. Summary of the invention

[0004] The main purpose of the present application is to provide a data dump method and device, a storage medium and an electronic device to solve the problem that in the related technology, the metadata information of the data block is stored based on the layered storage technology of the LSM tree. When performing data dump, the metadata information in each layer is searched from top to bottom in sequence, resulting in a relatively low efficiency of data dump.

[0005] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a data dump method is provided. The method comprises: if the current state of the target system meets the preset conditions for full data dump, multiple first adaptive radix trees are scanned to obtain first data information, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in the multiple first adaptive radix trees; multiple first associated containers are scanned to obtain second data information, wherein each first associated container stores second metadata information of a data shard, the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; the first data information and the second data information are dumped to the target storage medium.

[0006] Furthermore, scanning multiple first adaptive radix trees to obtain first data information includes: scanning the multiple first adaptive radix trees to obtain metadata information corresponding to leaf nodes in the multiple first adaptive radix trees, node types of the leaf nodes, and common prefix information in upper nodes corresponding to the leaf nodes; determining attribute information of the metadata information corresponding to the leaf nodes based on the node type of the leaf nodes, the number of leaves, and the common prefix information; and obtaining the first data information based on the attribute information and the metadata information corresponding to the leaf nodes.

[0007] Furthermore, before dumping the first data information and the second data information to the target storage medium, the method also includes: scanning the coding table in the target system and the bitmap corresponding to the coding table to obtain third data information, wherein the coding table is used to compress and encode metadata information when constructing the multiple first adaptive radix trees, and the bitmap is used to express the usage of the array in the coding table; dumping the third data information to the target storage medium.

[0008] Furthermore, after dumping the first data information and the second data information to the target storage medium, the method also includes: judging whether the full data dump has been completed by using a completion identifier corresponding to the full data dump; if the full data dump has been completed, reading the first data information in the target storage medium, and constructing multiple second adaptive radix trees based on the first data information; reading the second data information in the target storage medium, and constructing multiple second associated containers based on the first data information.

[0009] Furthermore, after constructing multiple second associated containers based on the first data information, the method also includes: reading the newly added metadata information in the incremental block to obtain first newly added metadata information; obtaining a first version number corresponding to the first newly added metadata information, wherein the first version number is generated when the first newly added metadata information is added; determining whether the first version number is a first expected version number based on a target version number, wherein the target version number is a version number corresponding to this full data dump; if the first version number is the first expected version number, updating the multiple second adaptive radix trees based on the first newly added metadata information to obtain multiple updated second adaptive radix trees, or updating the multiple second associated containers based on the first newly added metadata information to obtain multiple updated second associated containers.

[0010] Furthermore, after determining whether the first version number is the first expected version number based on the target version number, the method also includes: if the target version number is not the first expected version number, storing the first newly added metadata information in the target queue; reading the newly added metadata information in the incremental block again to obtain second newly added metadata information; if the second version number corresponding to the second newly added metadata information is the first expected version number, updating the multiple second adaptive radix trees based on the second newly added metadata information to obtain multiple updated second adaptive radix trees, or updating the multiple second associated containers based on the second newly added metadata information to obtain multiple updated second associated containers.

[0011] Further, after updating the multiple second adaptive radix trees according to the second new metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers according to the second new metadata information to obtain the updated multiple second associated containers, the method also includes: judging whether there is third new metadata information in the target queue, wherein the version number corresponding to the third new metadata information is the second expected version number; if the third new metadata information does not exist, repeating the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read; if the third new metadata information exists, updating the updated multiple second adaptive radix trees according to the third new metadata information, or updating the updated multiple second associated containers according to the third new metadata information, and repeating the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read.

[0012] Furthermore, before scanning multiple first adaptive radix trees to obtain first data information, the method also includes: obtaining historical time information of the last time the target system completed a full data dump; determining a time difference based on the historical time information and current time information; and judging whether the current state of the target system meets the preset conditions for the full data dump based on the time difference and a time threshold condition in the preset conditions for the full data dump.

[0013] Further, judging whether the current state of the target system satisfies the preset conditions of the full data dump based on the time difference and the time threshold condition in the preset conditions of the full data dump includes: judging whether the time difference is greater than or equal to the time threshold in the time threshold condition; if the time difference is greater than the time threshold, calculating a first total value of the metadata information in the incremental block within a target time period, wherein the target time period is from the time point corresponding to the historical time information to the time point of the current time information; obtaining a second total value of the metadata information counted by the metadata management module in the target system; judging whether the current state of the target system satisfies the preset conditions of the full data dump based on the data relationship between the first total value and the second total value and the data relationship condition in the preset conditions of the full data dump.

[0014] Furthermore, before scanning multiple first adaptive radix trees to obtain first data information, the method also includes: obtaining the number of requests received by the target system to process metadata information; and judging whether the current state of the target system meets the preset conditions for the full data dump based on the number of requests and the request threshold condition in the preset conditions for the full data dump.

[0015] In order to achieve the above-mentioned purpose, according to another aspect of the present application, a data dump device is provided. The device includes: a first scanning unit, which is used to scan multiple first adaptive radix trees to obtain first data information if the current state of the target system meets the preset conditions for full data dump, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in the multiple first adaptive radix trees; a second scanning unit, which is used to scan multiple first associated containers to obtain second data information, wherein each first associated container stores second metadata information of a data shard, the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; a first dump unit, which is used to dump the first data information and the second data information to a target storage medium.

[0016] Furthermore, the first scanning unit includes: a scanning subunit, used to scan the multiple first adaptive radix trees to obtain metadata information corresponding to leaf nodes in the multiple first adaptive radix trees, node types of the leaf nodes, and common prefix information in upper nodes corresponding to the leaf nodes; a determination subunit, used to determine attribute information of the metadata information corresponding to the leaf nodes based on the node type of the leaf nodes, the number of leaves and the common prefix information; and a processing subunit, used to obtain the first data information based on the attribute information and the metadata information corresponding to the leaf nodes.

[0017] Furthermore, the device also includes: a third scanning unit, used to scan the coding table and the bitmap corresponding to the coding table in the target system before dumping the first data information and the second data information to the target storage medium to obtain third data information, wherein the coding table is used to compress and encode metadata information when constructing the multiple first adaptive radix trees, and the bitmap is used to express the usage of the array in the coding table; a second dump unit, used to dump the third data information to the target storage medium.

[0018] Furthermore, the device also includes: a first judgment unit, which is used to judge whether the full data dump has been completed through a completion identifier corresponding to the full data dump after dumping the first data information and the second data information to the target storage medium; a first reading unit, which is used to read the first data information in the target storage medium if the full data dump has been completed, and construct multiple second adaptive radix trees based on the first data information; a second reading unit, which is used to read the second data information in the target storage medium, and construct multiple second associated containers based on the first data information.

[0019] Furthermore, the device also includes: a third reading unit, used to read the newly added metadata information in the incremental block to obtain the first newly added metadata information; a first acquisition unit, used to obtain the first version number corresponding to the first newly added metadata information, wherein the first version number is generated when the first newly added metadata information is added; a first determination unit, used to determine whether the first version number is the first expected version number based on the target version number, wherein the target version number is the version number corresponding to this full data dump; a first update unit, used to update the multiple second adaptive radix trees based on the first newly added metadata information if the first version number is the first expected version number, to obtain the updated multiple second adaptive radix trees, or to update the multiple second associated containers based on the first newly added metadata information to obtain the updated multiple second associated containers.

[0020] Furthermore, the device also includes: a storage unit, which is used to determine whether the first version number is the first expected version number based on the target version number, and if the target version number is not the first expected version number, store the first new metadata information in the target queue; a fourth reading unit, which is used to read the new metadata information in the incremental block again to obtain second new metadata information; and a second updating unit, which is used to update the multiple second adaptive radix trees based on the second new metadata information if the second version number corresponding to the second new metadata information is the first expected version number, to obtain multiple updated second adaptive radix trees, or to update the multiple second associated containers based on the second new metadata information to obtain multiple updated second associated containers.

[0021] Furthermore, the device also includes: a second judgment unit, which is used to determine whether there is third new metadata information in the target queue after updating the multiple second adaptive radix trees according to the second new metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers according to the second new metadata information to obtain the updated multiple second associated containers, wherein the version number corresponding to the third new metadata information is the second expected version number; a first execution unit, which is used to repeat the step of reading the new metadata information in the incremental block if the third new metadata information does not exist, until all the new metadata information in the incremental block has been read; a third updating unit, which is used to update the updated multiple second adaptive radix trees according to the third new metadata information if the third new metadata information exists, or update the updated multiple second associated containers according to the third new metadata information, and repeat the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read.

[0022] Furthermore, the device also includes: a second acquisition unit, used to obtain the historical time information of the last time the target system completed a full data dump before scanning multiple first adaptive radix trees to obtain the first data information; a second determination unit, used to determine the time difference based on the historical time information and the current time information; and a third judgment unit, used to judge whether the current state of the target system meets the preset conditions for the full data dump based on the time difference and the time threshold condition in the preset conditions for the full data dump.

[0023] Furthermore, the third judgment unit includes: a first judgment subunit, used to judge whether the time difference is greater than or equal to the time threshold in the time threshold condition; a calculation subunit, used to calculate a first total value of the metadata information in the incremental block within the target time period if the time difference is greater than the time threshold, wherein the target time period is from the time point corresponding to the historical time information to the time point of the current time information; an acquisition subunit, used to acquire a second total value of the metadata information counted by the metadata management module in the target system; and a second judgment subunit, used to judge whether the current state of the target system meets the preset conditions for the full data dump based on the data relationship between the first total value and the second total value and the data relationship condition in the preset conditions for the full data dump.

[0024] Furthermore, the device also includes: a third acquisition unit, used to obtain the number of requests received by the target system for processing metadata information before scanning multiple first adaptive radix trees to obtain first data information; a fourth judgment unit, used to judge whether the current state of the target system meets the preset conditions for full data dump based on the number of requests and the request threshold condition in the preset conditions for full data dump.

[0025] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the above-mentioned data dump methods.

[0026] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the above-mentioned data dump methods when running.

[0027] Through the present application, the following steps are adopted: if the current state of the target system meets the preset conditions for full data dump, multiple first adaptive radix trees are scanned to obtain first data information, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in multiple first adaptive radix trees; multiple first associated containers are scanned to obtain second data information, wherein each first associated container stores second metadata information of a data shard, the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; the first data information and the second data information are dumped to the target storage medium, which solves the problem that the layered storage technology based on LSM tree in the related technology stores metadata information of data blocks, and when dumping data, the metadata information in each layer is searched from top to bottom in turn, resulting in a relatively low efficiency of data dump. In this solution, an adaptive radix tree is used as the memory structure of fixed-length metadata information, and an associative container is used as the memory structure of variable-length metadata information. When performing a full data dump, the adaptive radix tree and the associative container can quickly scan the full data, and there is no need to search for metadata information in each layer from top to bottom, thereby achieving the effect of improving the efficiency of data dump. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 is a flow chart of a data dump method provided according to an embodiment of the present application;

[0030] Figure 2 is a schematic diagram of the data layout of a meta chunk provided according to an embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of data layout of variable-length kv during dumping according to an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of an adaptive radix tree provided according to an embodiment of the present application;

[0033] Figure 5 It is a schematic diagram of the data layout of the fixed-length kv when dumping according to the embodiment of the present application;

[0034] Figure 6 is a schematic diagram of a coding table provided according to an embodiment of the present application;

[0035] Figure 7 is a schematic diagram of index cache information provided according to an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of a writing process of index cache information provided according to an embodiment of the present application;

[0037] Fig. 9 is a schematic diagram of a data dump device provided according to an embodiment of the present application;

[0038] Fig.10 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0040] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0043] Distributed block storage system: a storage system that stores data in a distributed manner across multiple storage servers and combines these distributed storage resources into a virtual storage device, providing block (read and write by location) semantics to the outside world.

[0044] LSM tree (Log-Structured Merge Tree): is a data structure used to store and manage data, usually used to implement key-value storage or similar database systems. The characteristic of LSM tree is that data is written to disk in the form of logs, and merged and compressed in the background to improve write performance and reduce read costs. LSM tree usually consists of multiple layers, including memory layer, disk layer and possible compression layer, each layer has different performance characteristics and uses.

[0045] Metadata information: the mapping relationship between the logical location of the block and the physical location of the underlying storage server, which generally includes two forms: fixed-length key-value pairs and variable-length key-value pairs.

[0046] Adaptive radix tree: Compressed prefix tree is a more space-saving prefix tree and multi-branch search tree. The middle nodes of the tree do not save leaves but point to child nodes, and leaves are only saved in the leaf nodes of the last layer. Each node includes a fixed number of slots, which is 2 to the power of n, called the base of the adaptive radix tree. The main feature is to store a set of strings in the order of characters and save space by sharing common prefixes. Its structure is a multi-branch tree, each node represents a character, starting from the root node, each node has several child nodes, each child node represents the next character.

[0047] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0048] The present invention is described below in conjunction with preferred implementation steps. Figure 1 is a flow chart of a data dump method provided according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0049] Step S101, if the current state of the target system meets the preset conditions for full data dump, multiple first adaptive radix trees are scanned to obtain first data information, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information includes at least metadata information corresponding to leaf nodes in multiple first adaptive radix trees.

[0050] Optionally, the target system can be a distributed block storage system. For the fixed-length metadata information used to represent the mapping relationship between the logical location of the block and the physical location of the underlying storage server, an adaptive radix tree (ART tree) is used as the memory structure. The adaptive radix tree (Radix Tree) is a Trie (prefix tree). Trie splits the key (i.e., the logical address in the metadata information above) and stores each subsequence in order from root to leaf. The order from root to leaf is called a path. The complete path constitutes the key of the leaf. If multiple keys have the same prefix, the prefix part is stored only once. The last layer node where the leaf is located does not use a child pointer but directly embeds the value (i.e., the physical address mentioned above) in the ART tree, thereby reducing memory usage. Finally, the adaptive radix tree corresponding to the data shard is constructed according to the node leaf.

[0051] It should be noted that the concept of shard is introduced to split the data blocks displayed by the distributed block storage system into smaller units within the system, which is called shard. That is, each data block is sharded and finally an adaptive radix tree corresponding to each data shard is constructed. It should be noted that one data shard corresponds to one adaptive radix tree.

[0052] It should be noted that in order to reduce the total IO write latency, the index information (i.e., the metadata information of the block) and the metadata information of the index information can be combined into one IO and written to the persistent medium (meta chunk). The types of meta chunks are divided into full or incremental chunks. The data layout of the index information and the metadata information of the index information in the persistent medium is as follows: Figure 2 As shown, the layout of the meta chunk consists of three parts: header part: occupies a total of 64 bytes including reserved space, and the data structure is as follows:

[0053] Table 1 Explanation of the meaning of each field in the header

[0054]

[0055] Each entry consists of two parts: meta and elements set. The elements set is the specific index information, and meta is the metadata information of the index information. It is of fixed length and includes the following data structure:

[0056] Table 2 Explanation of the meaning of each entry’s data structure

[0057]

[0058] Elements set: Index data after compression (if any). If there are multiple records, they are stored in the following intervals: op_type+key_len (key length)+key+value_len (value length)+value, next<key,value> ....

[0059] Footer part: It occupies a total of 64 bytes including the reserved space. The data structure is as follows:

[0060] Table 3 Explanation of the meaning of footer data structure

[0061]

[0062] By constructing the above meta chunk layout, multiple index data and their metadata information, such as crc, elements_count, version, etc., can be encapsulated into one IO transaction operation, thereby avoiding the introduction of complex scenarios and logic such as increased latency caused by multiple IO operations and rollback of operation failures.

[0063] The index cache module (used to manage metadata information of blocks) includes two types of index data: fixed-length KV: mapping of logical offsets within storage shards to underlying physical storage space offsets; variable-length KV: KV used by other modules within storage shards, both key and value are variable length.

[0064] Since the chunk (underlying physical storage space) used by the index cache module only provides the append-only function, multiple write operations for the same logical offset in the same shard will generate multiple copies of data and the same amount of index data (i.e., the metadata information mentioned above). Only the last copy of these index data is valid data, and the others are invalid data, also known as garbage data. At the same time, these index data are stored in the incremental chunks of the index cache. As time goes by, the previously invalid index data will increase, which not only occupies the underlying physical storage space and cannot be released, but also affects the time to restore the index cache ART tree and the variable-length kv memory structure after the process is restarted (because all incremental chunks need to be traversed). Therefore, it is necessary to delete these invalid index data to free up storage space and reduce the number of incremental chunks that need to be read during data recovery.

[0065] By performing a full data dump, that is, writing all the latest fixed-length and variable-length data to the full chunk (fullchunk), it is equivalent to marking all index data in the incremental chunk as invalid data, thereby deleting chunks to release storage space and reducing the number of chunks that need to be read during data recovery.

[0066] Therefore, under the premise of using an adaptive radix tree as the memory structure of fixed-length key-value pairs, if the current state of the target system meets the preset conditions for full data dump, the existing multiple first adaptive radix trees are directly scanned to obtain first data information that includes at least metadata information corresponding to leaf nodes in multiple first adaptive radix trees.

[0067] It should be noted that when performing a full data dump, a version number for this full data dump needs to be assigned: the epoch value. This version number is monotonically increasing, and each dump operation (dump op) in this full data dump needs to carry this version number as specific identity information, and also as verification information for data recovery.

[0068] The current committed version of the index cache module is used as the above epoch. The index cache module maintains two versions internally. One is the submitted version, which is used to assign serial numbers to each fixed-length or variable-length kv update operation that the front-end user writes to the incremental chunk. The other is the committed version, which is used to record the maximum continuous version that has been completed in the fixed-length or variable-length kv update operation that the front-end user writes to the incremental chunk. For example, submitted version = 100, indicating that the serial number of the last fixed-length or variable-length update operation that has been submitted is 100, committed version = 98, indicating that the serial number of the maximum and continuous update operation that has been completed is 98, and the #99 operation must not have been returned from the underlying physical storage, otherwise the committed version will be equal to 99; the #100 operation may or may not be returned. Even if #100 returns before #99, the committed version will not be updated because the #99 of the previous operation did not return. When #99 returns, the index cache module scans a queue called operation completion, and then finds the largest continuous completed operation number as the new committed version. The version number can accurately realize the dump of the entire data.

[0069] Step S102, scanning multiple first associated containers to obtain second data information, wherein each first associated container stores second metadata information of a data slice, and the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair.

[0070] Optionally, the variable-length kv (i.e., the second metadata information mentioned above) is mainly a kv for organizing information for a data slice, and there are only a limited number of variable-length kvs for a data slice. Therefore, the memory structure of the variable-length kv uses the std::map container (i.e., the first associative container mentioned above) provided by the C++ standard STL (Standard Template Library) based on the red-black tree data structure. The elements in the std::map are sorted in the order of the keys, the keys are unique, and each key can only correspond to one value. It should be noted that one data slice corresponds to one first associative container, that is, the variable-length kv of each data slice is stored separately.

[0071] When dumping the full data, use the traversal method provided by the std::map container, start traversing from the smallest key of the std::map, and encode each dump with the kv as the granularity, such as Figure 3 The data layout after variable-length kv encoding shown, that is, the data layout of the second data information obtained by scanning multiple first associated containers, includes metadata information and meta of the metadata information (for example, kv1 includes meta, key1 and value1; kv2 includes meta, key2 and value2), and meta consists of the key length (key length) and the value length (value length).

[0072] It should be noted that the amount of metadata information obtained in one scan is limited. For example, a dump op can dump a maximum of 4 MiB of kv information in a std::map. The next dump op continues to traverse the std::map container from the end of the previous dump op until the entire container is traversed. Each dump op needs to be assigned an op operation number within the epoch, encoded starting from 0, and persisted in the version of the entry meta of the written full chunk. At the same time, the last dump op needs to set completion to 1 in the entry meta to determine whether the full data dump is complete when the process is restarted for data recovery.

[0073] Step S103: dump the first data information and the second data information to a target storage medium.

[0074] Optionally, the first data information and the second data information obtained by scanning are transferred to a target storage medium.

[0075] In summary, by using an adaptive radix tree as the memory structure of fixed-length metadata information and an associative container as the memory structure of variable-length metadata information, when performing a full data dump, the adaptive radix tree and the associative container can quickly scan the full data, and there is no need to search for metadata information in each layer from top to bottom, thereby achieving the effect of improving the efficiency of data dump.

[0076] Optionally, in the data dump method provided in the embodiment of the present application, scanning multiple first adaptive radix trees to obtain first data information includes: scanning multiple first adaptive radix trees to obtain metadata information corresponding to leaf nodes in the multiple first adaptive radix trees, node types of leaf nodes, and common prefix information in upper nodes corresponding to leaf nodes; determining attribute information of the metadata information corresponding to the leaf nodes based on the node type of the leaf nodes, the number of leaves, and the common prefix information; and obtaining the first data information based on the attribute information and the metadata information corresponding to the leaf nodes.

[0077] In an optional embodiment, scanning multiple first adaptive radix trees includes: for example, the data shard size is calculated as 32GiB, and the 32GiB logical space uses a maximum of 32GiB / 4KiB = 8MiB leafs, and the value of a leaf occupies 5 bytes, which is a total memory usage of 40MiB. At the same time, the total memory used by the intermediate node (inner node) of the ART tree will not exceed 1MiB, because the ART tree of the 32GiB shard has only 3 layers, the first layer is the root node, which has only one inner Node256 node, and the second layer has a maximum of 256 inner Node256 nodes, each inner Node256 stores 256 pointers to the lower layer nodes, each pointer occupies 8 bytes, so the first layer and the second layer occupy a total of (1+256) x 256 x 8 = 514KiB (1KiB = 1024 bytes) of memory.

[0078] When dumping, only the leaf node information is encoded, and the prefix information of all previous layers of the leaf node (that is, the common prefix information mentioned above) is recorded. For example, if the key of a leaf is 0x123400, when dumping the leaf node, only the common prefix 0x1234 of each leaf is recorded. Each dump is encoded at the leaf node granularity, and a dump op operation can dump up to 4 MiB of ART tree leaf nodes and their prefix encoding information. Figure 4The ART tree shown in the figure includes root, inner node1, inner node2, and leaf node1 to leaf node5. Scan the last layer of leaf nodes (leaf node1) from the leftmost leaf node of the ART tree. Figure 4 The leaf node in the tree, the order direction is indicated by a dotted line). The prefix of inner node1 is 0x12; the prefix of inner node2 is 0x77; the prefix of leaf node1 is 0x1234, and there are two leaves inside. The key of the first leaf is 0x123400. Figure 4 The ... in the table indicates the key of other subsequent leaves, and the value range is (0x123400, 0x1234FF]; the prefix of leaf node2 is 0x1256, and the prefix of leaf node3 is 0x1278.

[0079] The data structure of the first data information is as follows Figure 5 As shown, it includes: leaf node1 and leaf node2. A leaf node (leaf node1 and leaf node2) consists of meta and leaves. Meta includes node type, leaf count, and common prefix information (key prefix). The ART tree may exceed 4 MiB and cannot complete the encoding of all leaf nodes in one dump operation. Therefore, multiple ART tree dump operations may be required.

[0080] By scanning the ART tree, the corresponding fixed-length KV can be quickly obtained, thereby improving the efficiency of data dump.

[0081] Optionally, in the data dump method provided in an embodiment of the present application, before dumping the first data information and the second data information to the target storage medium, the method also includes: scanning the coding table and the bitmap corresponding to the coding table in the target system to obtain third data information, wherein the coding table is used to compress and encode the metadata information when constructing multiple first adaptive radix trees, and the bitmap is used to describe the usage of the array in the coding table; dumping the third data information to the target storage medium.

[0082] In an optional embodiment, when constructing multiple first adaptive radix trees, the metadata information may be compressed and encoded to reduce memory usage, such as Figure 6As shown in the figure, the values ​​2065, 3089 and 1010 in the box represent the metadata information before encoding, and the index number of the encoding array (numbering starting from 0, for example, 0, 1, 2, 3, 4, 5, 6) is used as the encoded value; in addition, black indicates that the array item is occupied, and white indicates that the array item is not occupied. The color information is stored in the bitmap information and is represented by 1 bit. The information that needs to be persisted in the encoding table includes the encoding array and the bitmap. They are both continuous logical memory, and the total occupied space is less than 200KiB. The encoding can be completed in one dump op operation. It should be noted that a first adaptive radix tree corresponds to a bitmap and a coding table.

[0083] The memory usage of metadata information can be effectively reduced through bitmaps and encoding tables.

[0084] Optionally, in the data dump method provided in an embodiment of the present application, after dumping the first data information and the second data information to the target storage medium, the method further includes: judging whether the full data dump has been completed through a completion identifier corresponding to the full data dump; if the full data dump has been completed, reading the first data information in the target storage medium, and constructing multiple second adaptive radix trees based on the first data information; reading the second data information in the target storage medium, and constructing multiple second associated containers based on the first data information.

[0085] In an optional embodiment, after the first data information and the second data information as well as the encoding table and the bitmap corresponding to the encoding table are dumped to the target storage medium, the following steps are also included: each time the full data dump starts, the index cache module will record the starting position and length information of each dump op written in the full chunk, and this information is called a full chunk check point. When the process restarts the data recovery, it starts searching forward from the latest full data dump to determine whether the full data dump is complete, that is, to determine whether the completion (i.e., the above-mentioned completion identifier) ​​in the entry marked by the last checkpoint is true. If the full data dump is completed, the ART tree (i.e., the above-mentioned second adaptive radix tree) and the memory data structure of the variable-length kv (i.e., the above-mentioned second associative container) are reconstructed according to the first data information and the second data information in the target storage medium.

[0086] The first data information and the second data information can be used to quickly rebuild the memory data structure of the ART tree and the variable-length kv.

[0087] Optionally, in the data dump method provided in the embodiment of the present application, after constructing multiple second associated containers based on the first data information, the method also includes: reading the newly added metadata information in the incremental block to obtain the first newly added metadata information; obtaining the first version number corresponding to the first newly added metadata information, wherein the first version number is generated when the first newly added metadata information is added; determining whether the first version number is the first expected version number based on the target version number, wherein the target version number is the version number corresponding to this full data dump; if the first version number is the first expected version number, updating the multiple second adaptive radix trees based on the first newly added metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers based on the first newly added metadata information to obtain the updated multiple second associated containers.

[0088] In an optional embodiment, when performing a full data dump, new index data may be written into the incremental chunk. The incremental chunk where these new index data are located will not be released. At the same time, these new index data also need to be read during data recovery and updated to the ART tree and the memory structure of the variable-length kv, specifically including: reading the newly added metadata information in the incremental chunk (i.e., the aforementioned incremental chunk). It should be noted that, assuming that the full data dump with epoch 99 is the last full data dump, it is necessary to read each entry with submit version > 99 in the incremental chunk in turn (i.e., the aforementioned newly added metadata information) and then update it to the memory data structure of the ART tree and the variable-length kv. The reason why it is not necessary to read the entry with submit version <= 99 is that the results of these entries have been completely written into the full chunk corresponding to the full data dump with epoch 99.

[0089] It should be noted that it is necessary to determine whether there are holes and disorder based on the version in the incremental chunk, that is, it is necessary to determine whether the first newly added metadata information read is the metadata information corresponding to the expected version, specifically including: judging whether the first version number is the first expected version number based on the epoch of the full data dump (that is, the above-mentioned target version number). It should be noted that the first expected version number is epoch+1. If the first version number is the first expected version number, multiple second adaptive radix trees or multiple second associated containers are directly updated according to the first newly added metadata information.

[0090] The accuracy of metadata information updates can be guaranteed by judging the version number, avoiding the situation where data errors exist in the updated adaptive radix tree or associated container due to sequence problems.

[0091] Optionally, in the data dump method provided in the embodiment of the present application, after determining whether the first version number is the first expected version number based on the target version number, the method also includes: if the target version number is not the first expected version number, storing the first newly added metadata information in the target queue; reading the newly added metadata information in the incremental block again to obtain the second newly added metadata information; if the second version number corresponding to the second newly added metadata information is the first expected version number, updating the multiple second adaptive radix trees based on the second newly added metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers based on the second newly added metadata information to obtain the updated multiple second associated containers.

[0092] In an optional embodiment, if the target version number is not the first expected version number, an order-preserving queue (i.e., the target queue mentioned above) is constructed, and the order-preserving queue needs to have the ability to search according to the entry version. Start reading the incremental chunk entry, and if the read entry version is greater than the expected version (i.e., the target version number is not the first expected version number), the entry data is stored in the order-preserving queue, and then the incremental chunk is continued to be read. If an entry that meets the expected version is read later (i.e., the second newly added metadata information mentioned above), multiple second adaptive radix trees are updated according to the second newly added metadata information to obtain multiple updated second adaptive radix trees, or multiple second associated containers are updated according to the second newly added metadata information to obtain multiple updated second associated containers.

[0093] If the second version number corresponding to the second newly added metadata information is not the first expected version number, it is necessary to read the incremental chunk again until an entry that meets the expected version is read.

[0094] Accurate updates of metadata information can be achieved by saving the queue.

[0095] Optionally, in the data dump method provided in the embodiment of the present application, after updating the multiple second adaptive radix trees according to the second new metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers according to the second new metadata information to obtain the updated multiple second associated containers, the method also includes: determining whether there is third new metadata information in the target queue, wherein the version number corresponding to the third new metadata information is the second expected version number; if the third new metadata information does not exist, repeating the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read; if the third new metadata information exists, updating the updated multiple second adaptive radix trees according to the third new metadata information, or updating the updated multiple second associated containers according to the third new metadata information, and repeating the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read.

[0096] In an optional embodiment, after the update process is performed, it is determined whether there is an entry that meets the expected version in the order-preserving queue, that is, whether there is a third newly added metadata information in the above-mentioned target queue. If so, the entry is taken out of the order-preserving queue, and the memory structure of the ART tree and the variable-length kv is updated after decoding. It should be noted that if the epoch of the full data dump is 99, and the corresponding second newly added metadata information with version number 100 has been written, then the second expected version number should be 101. At the same time, continue to determine whether there is an entry that meets the expected version in the order-preserving queue until the order-preserving queue is empty, or no entry that meets the expected version is found. If not, continue to read the incremental chunk.

[0097] It should be noted that after all incremental chunks are scanned and read, there may still be entries in the order-preserving queue. These entries are discontinuous with the entryversion that has been decoded and updated to the memory structure of the ART tree and variable-length kv, that is, there are holes, and they cannot be used to restore index data. Due to the existence of holes, even if subsequent entries are written to the incremental chunk, they will not be returned to the front-end user, so there is no need to worry about data inconsistency caused by not using these read entries here. For example, entry #98 completes writing the incremental chunk and returns success to the front-end user, and entry #100 completes writing the incremental chunk, but does not return to the front-end user, because it is necessary to ensure that entry #99 returns first. At this time, entry #99 may not be successfully written to the incremental chunk due to network and underlying storage reasons. When the entire process is restarted later, it will appear that only entry #98 and entry #100 can be read, and there will be holes. Since entry #99 and entry #100 did not return a success message to the front-end user, entry #100 is not decoded and updated to the ART tree and the memory structure of the variable-length kv, so there will be no situation where the write is successful but the read cannot be performed.

[0098] Optionally, in the data dump method provided in the embodiment of the present application, before scanning multiple first adaptive radix trees to obtain the first data information, the method also includes: obtaining historical time information of the last time the target system completed a full data dump; determining a time difference based on the historical time information and the current time information; and judging whether the current state of the target system meets the preset conditions for the full data dump based on the time difference and the time threshold condition in the preset conditions for the full data dump.

[0099] In an optional embodiment, the following steps are used to determine whether the current state of the target system meets the preset conditions for a full data dump, set a time threshold threshold for a timed trigger (i.e., the above-mentioned time threshold condition), with the default value set to 1 day, and then record the time point T1 when the last full data dump was completed (i.e., the above-mentioned historical time information) and the current time information, and determine whether the current state of the target system meets the preset conditions for a full data dump based on the time difference between the historical time information and the current time information and the time threshold threshold.

[0100] By using the time threshold to determine whether a full data dump is needed, you can effectively avoid the problem of a huge number of incremental chunks consuming the underlying physical storage space.

[0101] Optionally, in the data dump method provided in the embodiment of the present application, judging whether the current state of the target system meets the preset conditions for the full data dump based on the time difference and the time threshold condition in the preset conditions for the full data dump includes: judging whether the time difference is greater than or equal to the time threshold in the time threshold condition; if the time difference is greater than the time threshold, calculating a first total value of the metadata information in the incremental block within the target time period, wherein the target time period is from the time point corresponding to the historical time information to the time point of the current time information; obtaining a second total value of the metadata information counted by the metadata management module in the target system; judging whether the current state of the target system meets the preset conditions for the full data dump based on the data relationship between the first total value and the second total value and the data relationship condition in the preset conditions for the full data dump.

[0102] In an optional embodiment, judging whether the current state of the target system meets the preset conditions for the full data dump based on the time difference and the time threshold condition in the preset conditions for the full data dump includes: if the time difference exceeds the set timing trigger time threshold threshold, then calculating the total amount S1 of index data written to the incrementalchunk since the last full data dump (i.e., the first total amount value mentioned above) and the second total amount S2 of metadata information counted by the metadata management module (i.e., the index cache module mentioned above), and judging whether the current state of the target system meets the preset conditions for the full data dump based on the data relationship between the first total amount value and the second total amount value and the data relationship condition in the preset conditions for the full data dump.

[0103] For example, if S1 < 2 x S2, that is, the amount of changed data in the incremental chunk is less than twice the amount of data that requires a full data dump, a full data dump is not triggered (the reason why it is set to twice is that a full data dump consumes additional space and CPU, so a certain space recovery rate is required to trigger it); otherwise, a full data dump is triggered and the above-mentioned related statistical information is reset.

[0104] The above judgment process can more reasonably perform full data dump and avoid unnecessary waste of resources.

[0105] Optionally, in the data dump method provided in the embodiment of the present application, before scanning multiple first adaptive radix trees to obtain first data information, the method also includes: obtaining the number of requests received by the target system for processing metadata information; and judging whether the current state of the target system meets the preset conditions for the full data dump based on the number of requests and the request threshold condition in the preset conditions for the full data dump.

[0106] In an optional embodiment, the following steps can also be used to determine whether the current state of the target system meets the preset conditions for full data dump: incremental chunk entry is used to record the fixed-length and variable-length kv submitted by the front-end user. The more entries there are, the more modifications the user has made. When these modifications accumulate to a certain number of times, if the full data dump is not performed in time, a lot of incremental chunks need to be read when the process is restarted for data recovery, which affects the data recovery speed and increases the waiting time for external services. In addition, a huge number of incremental chunks also consumes the underlying physical storage space. When the cluster size is not large, it is easy to cause cluster capacity overload. However, if the full data dump operation is performed too frequently, it will occupy too many business threads, reduce the iops of the front-end user, and increase the processing delay of io. Therefore, it is necessary to weigh the triggering mechanism among the various factors of overload consumption of physical storage capacity, long data recovery time, and impact on cluster business. In the embodiment of the present application, the request threshold condition is set to 128,000 times. Setting this value is based on the following considerations:

[0107] According to actual running test statistics, the average size of incremental chunk entry is about 25KiB, and 128,000 times takes up about 128,000 times 25KiB = 3.2GiB of data. The total data includes fixed-length and variable-length parts, and the total average size is about 0.5GiB. In the worst case, 3.2GiB of data needs to be read from the incremental chunk to recover the index data. Taking the nvme disk P4610 as an example, the bandwidth of large block sequential reading is 3200MiB per second, so it takes about 1 second to read 3.2GiB of data. Considering that multiple incremental chunks may be distributed on different nvme disks, and network transmission after reading, entry disorder recognition and order preservation after local decoding, and updating of ART tree and variable-length kv memory data, the overall time consumption is estimated to be 2 seconds. According to the average reading of 1.6GiB of data, the average expected recovery time is 1 second. At the same time, 3.2GiB / 0.5GiB = 6.4 times the space recovery rate is also acceptable.

[0108] In an optional embodiment, once a full data dump is completed, the index data stored in the incremental chunk can be cleaned up. Since the index data stores metadata information of the user data, the proportion of the underlying physical storage space occupied by it is relatively low compared to the user data. Therefore, the strategy of cleaning up the user data, that is, moving some valid data to other chunks to free up the physical space of the entire chunk, is not adopted. Instead, a strategy of deleting the chunk only when all the index data in the entire chunk can be cleaned up is adopted.

[0109] like Figure 7 As shown, including incremental chunk1 (including header, entry_1 (entry 1), entry_2, ...entry_n and footer) and incremental chunk2 (including header, entry_n+1 (entry n+1), entry_ n+2, ...entry_n+m and footer), when the full data dump of epoch 99 is completed, that is, the results of the entries in incremental chunk1 (including entry_1 (entry 1), entry_2, ...entry_n and footer) and incremental chunk2 (including entry_n+1 (entry n+1, corresponding to submit version = 98), entry_ n+2 (entry n+2, corresponding to submit version = 99)) with submit version <= 99 have been dumped to full chunk1 (that is, Figure 7 At this point, the incremental chunk entry with submitted version <= 99 can be deleted, that is, from Figure 7 As can be seen, the entry with submit version = 99 is stored in incremental chunk2, and the older incremental chunk1 can be completely deleted.

[0110] At the same time, the full chunk also needs to be cleaned up. Figure 7As shown in the figure, including full chunk1 (including header, entry_99_1, entry_99_2…entry_99_n and footer) and full chunk2 (including header, entry_377_2, entry_666_0…), after the full data dump with epoch 666 is completed in full chunk2 (that is, completion = true, epoch = 666), the index data of the previous full data dump can be deleted, from Figure 7 It can be seen that the full data dump with epoch 666 is written starting from full chunk2, so the previous full chunk1 can be completely deleted.

[0111] In an optional embodiment, various statistical information is recorded, and the information to be counted includes:

[0112] 1) Memory usage:

[0113] a. The number of various nodes of the ART tree, including inner and leaf nodes, so that the overall memory capacity used by the ART tree can be calculated.

[0114] b. The memory capacity used by the memory structure of the variable-length kv.

[0115] 2) Storage media usage space

[0116] a. The number of incremental chunks used and the chunk info of each chunk. The chunk info includes the chunk id, the total chunk size, and the actual written capacity.

[0117] b.Full chunk information is the same as incremental chunk.

[0118] 3) Other information during runtime

[0119] a.submit version and committed version.

[0120] b. Whether a full data dump is currently in progress, and information about the full data dump in progress, including epochs and the number of completed dump operations.

[0121] c. Information about full data dumps that have been triggered in the past, including epoch, number of dumped ops, and whether it is a complete full data dump.

[0122] Recording the above statistical information will help with subsequent abnormality location and analysis.

[0123] In an optional embodiment, the accurate writing of index cache information (i.e., metadata information of a block) can be achieved based on the above fixed-length kv memory structure, such as Figure 8 As shown, the specific process description:

[0124] If the front end submits three write ios in succession, these three write ios will be assigned a unique version number version within the shard when they are submitted to the index cache module, and the version will be persisted to the storage medium. In the incremental chunk, the three write ios correspond to entry_1, entry_2, and entry_n.

[0125] When the incremental chunk is persisted, the corresponding write io will enter the second step, which is to update the ART. Figure 8 The ART tree shown in the figure includes root, inner node1, inner node2, and leaf node1 to leaf node5. Figure 8 As shown, entry_1, entry_2 and entry_n correspond to leaf_1, leaf_2 and leaf_3 in ART. The update operation will be strictly carried out in order according to the assigned version, so as to ensure that the index data in the ART tree will be updated strictly in the order of operations submitted by the user, and prevent the new data submitted later from being overwritten by the old data submitted earlier.

[0126] The data dump method provided by the embodiment of the present application is as follows: if the current state of the target system meets the preset conditions for full data dump, multiple first adaptive radix trees are scanned to obtain first data information, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in multiple first adaptive radix trees; multiple first associated containers are scanned to obtain second data information, wherein each first associated container stores second metadata information of a data shard, the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; the first data information and the second data information are dumped to the target storage medium, which solves the problem that the layered storage technology based on LSMtree in the related technology stores metadata information of data blocks, and when performing data dump, the metadata information in each layer is searched from top to bottom in sequence, resulting in a relatively low efficiency of data dump. In this solution, an adaptive radix tree is used as the memory structure of fixed-length metadata information, and an associative container is used as the memory structure of variable-length metadata information. When performing a full data dump, the adaptive radix tree and the associative container can quickly scan the full data, and there is no need to search for metadata information in each layer from top to bottom, thereby achieving the effect of improving the efficiency of data dump.

[0127] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0128] The embodiment of the present application also provides a data dump device. It should be noted that the data dump device of the embodiment of the present application can be used to execute the data dump method provided in the embodiment of the present application. The data dump device provided in the embodiment of the present application is introduced below.

[0129] Fig. 9 Schematic diagram of a data dump device according to an embodiment of the present application. Fig. 9 As shown, the device includes: a first scanning unit 901, a second scanning unit 902 and a first dump unit 903.

[0130] The first scanning unit 901 is used to scan multiple first adaptive radix trees to obtain first data information if the current state of the target system meets the preset conditions for full data dump, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in the multiple first adaptive radix trees;

[0131] A second scanning unit 902 is used to scan multiple first associated containers to obtain second data information, wherein each first associated container stores second metadata information of a data slice, and the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair;

[0132] The first dump unit 903 is used to dump the first data information and the second data information to a target storage medium.

[0133] The data dump device provided by the embodiment of the present application, through the first scanning unit 901, if the current state of the target system meets the preset conditions for full data dump, scans multiple first adaptive radix trees to obtain first data information, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in multiple first adaptive radix trees; the second scanning unit 902 scans multiple first associated containers to obtain second data information, wherein each first associated container stores second metadata information of a data shard, the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; the first dump unit 903 dumps the first data information and the second data information to the target storage medium, which solves the problem that the layered storage technology based on LSM tree in the related technology stores metadata information of data blocks, and when dumping data, the metadata information in each layer is searched from top to bottom in sequence, resulting in a relatively low efficiency of data dump. In this solution, an adaptive radix tree is used as the memory structure of fixed-length metadata information, and an associative container is used as the memory structure of variable-length metadata information. When performing a full data dump, the adaptive radix tree and the associative container can quickly scan the full data, and there is no need to search for metadata information in each layer from top to bottom, thereby achieving the effect of improving the efficiency of data dump.

[0134] Optionally, in the data dump device provided in the embodiment of the present application, the first scanning unit includes: a scanning subunit, used to scan multiple first adaptive radix trees to obtain metadata information corresponding to leaf nodes in the multiple first adaptive radix trees, node types of leaf nodes, and common prefix information in upper nodes corresponding to leaf nodes; a determination subunit, used to determine attribute information of the metadata information corresponding to the leaf node based on the node type of the leaf node, the number of leaves and the common prefix information; a processing subunit, used to obtain first data information based on the attribute information and the metadata information corresponding to the leaf node.

[0135] Optionally, in the data dump device provided in the embodiment of the present application, the device also includes: a third scanning unit, used to scan the coding table and the bitmap corresponding to the coding table in the target system before dumping the first data information and the second data information to the target storage medium to obtain third data information, wherein the coding table is used to compress and encode the metadata information when constructing multiple first adaptive radix trees, and the bitmap is used to describe the usage of the array in the coding table; a second dump unit, used to dump the third data information to the target storage medium.

[0136] Optionally, in the data dump device provided in the embodiment of the present application, the device also includes: a first judgment unit, used to judge whether the full data dump has been completed through a completion identifier corresponding to the full data dump after dumping the first data information and the second data information to the target storage medium; a first reading unit, used to read the first data information in the target storage medium if the full data dump has been completed, and construct multiple second adaptive radix trees based on the first data information; a second reading unit, used to read the second data information in the target storage medium, and construct multiple second associated containers based on the first data information.

[0137] Optionally, in the data dump device provided in the embodiment of the present application, the device also includes: a third reading unit, used to read the newly added metadata information in the incremental block to obtain the first newly added metadata information; a first acquisition unit, used to obtain the first version number corresponding to the first newly added metadata information, wherein the first version number is generated when the first newly added metadata information is added; a first determination unit, used to determine whether the first version number is the first expected version number based on the target version number, wherein the target version number is the version number corresponding to this full data dump; a first update unit, used to update the multiple second adaptive radix trees based on the first newly added metadata information if the first version number is the first expected version number, to obtain the updated multiple second adaptive radix trees, or to update the multiple second associated containers based on the first newly added metadata information to obtain the updated multiple second associated containers.

[0138] Optionally, in the data dump device provided in the embodiment of the present application, the device also includes: a storage unit, which is used to determine whether the first version number is the first expected version number based on the target version number, and if the target version number is not the first expected version number, store the first newly added metadata information in the target queue; a fourth reading unit, which is used to read the newly added metadata information in the incremental block again to obtain the second newly added metadata information; and a second updating unit, which is used to update the multiple second adaptive radix trees based on the second newly added metadata information if the second version number corresponding to the second newly added metadata information is the first expected version number, to obtain the updated multiple second adaptive radix trees, or to update the multiple second associated containers based on the second newly added metadata information to obtain the updated multiple second associated containers.

[0139] Optionally, in the data dump device provided in the embodiment of the present application, the device also includes: a second judgment unit, which is used to update the multiple second adaptive radix trees according to the second new metadata information to obtain the updated multiple second adaptive radix trees, or, after updating the multiple second associated containers according to the second new metadata information to obtain the updated multiple second associated containers, determine whether there is third new metadata information in the target queue, wherein the version number corresponding to the third new metadata information is the second expected version number; a first execution unit, which is used to repeat the step of reading the new metadata information in the incremental block if the third new metadata information does not exist, until all the new metadata information in the incremental block has been read; a third update unit, which is used to update the updated multiple second adaptive radix trees according to the third new metadata information if the third new metadata information exists, or, update the updated multiple second associated containers according to the third new metadata information, and repeat the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read.

[0140] Optionally, in the data dump device provided in the embodiment of the present application, the device also includes: a second acquisition unit, used to obtain the historical time information of the last time the target system completed a full data dump before scanning multiple first adaptive radix trees to obtain the first data information; a second determination unit, used to determine the time difference based on the historical time information and the current time information; a third judgment unit, used to judge whether the current state of the target system meets the preset conditions for the full data dump based on the time difference and the time threshold condition in the preset conditions for the full data dump.

[0141] Optionally, in the data dump device provided in the embodiment of the present application, the third judgment unit includes: a first judgment subunit, used to judge whether the time difference is greater than or equal to the time threshold in the time threshold condition; a calculation subunit, used to calculate the first total value of the metadata information in the incremental block within the target time period if the time difference is greater than the time threshold, wherein the target time period is from the time point corresponding to the historical time information to the time point of the current time information; an acquisition subunit, used to obtain the second total value of the metadata information counted by the metadata management module in the target system; and a second judgment subunit, used to judge whether the current state of the target system meets the preset conditions for the full data dump based on the data relationship between the first total value and the second total value and the data relationship condition in the preset conditions for the full data dump.

[0142] Optionally, in the data dump device provided in the embodiment of the present application, the device also includes: a third acquisition unit, used to obtain the number of requests received by the target system for processing metadata information before scanning multiple first adaptive radix trees to obtain first data information; a fourth judgment unit, used to judge whether the current state of the target system meets the preset conditions for full data dump based on the number of requests and the request threshold condition in the preset conditions for full data dump.

[0143] The data dump device includes a processor and a memory. The first scanning unit 901, the second scanning unit 902 and the first dump unit 903 are all stored in the memory as program units. The processor executes the program units stored in the memory to implement corresponding functions.

[0144] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set to achieve fast full data dump of metadata information by adjusting kernel parameters.

[0145] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0146] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, a data dump method is implemented.

[0147] An embodiment of the present invention provides a processor, which is used to run a program, wherein a data dump method is executed when the program is running.

[0148] like Fig.10As shown, an embodiment of the present invention provides an electronic device, the device includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the following steps when executing the program: if the current state of the target system meets the preset conditions for full data dump, multiple first adaptive radix trees are scanned to obtain first data information, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in multiple first adaptive radix trees; multiple first associated containers are scanned to obtain second data information, wherein each first associated container stores second metadata information of a data shard, the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; the first data information and the second data information are dumped to the target storage medium.

[0149] Optionally, scanning multiple first adaptive radix trees to obtain first data information includes: scanning multiple first adaptive radix trees to obtain metadata information corresponding to leaf nodes in the multiple first adaptive radix trees, node types of leaf nodes, and common prefix information in upper nodes corresponding to the leaf nodes; determining attribute information of the metadata information corresponding to the leaf nodes based on the node type of the leaf nodes, the number of leaves, and the common prefix information; and obtaining the first data information based on the attribute information and the metadata information corresponding to the leaf nodes.

[0150] Optionally, before dumping the first data information and the second data information to the target storage medium, the method also includes: scanning the coding table and the bitmap corresponding to the coding table in the target system to obtain third data information, wherein the coding table is used to compress and encode the metadata information when constructing multiple first adaptive radix trees, and the bitmap is used to describe the usage of the array in the coding table; dumping the third data information to the target storage medium.

[0151] Optionally, after dumping the first data information and the second data information to the target storage medium, the method further includes: determining whether the full data dump has been completed by using a completion identifier corresponding to the full data dump; if the full data dump has been completed, reading the first data information in the target storage medium, and constructing multiple second adaptive radix trees based on the first data information; reading the second data information in the target storage medium, and constructing multiple second associated containers based on the first data information.

[0152] Optionally, after constructing multiple second associated containers based on the first data information, the method also includes: reading the newly added metadata information in the incremental block to obtain the first newly added metadata information; obtaining a first version number corresponding to the first newly added metadata information, wherein the first version number is generated when the first newly added metadata information is added; determining whether the first version number is the first expected version number based on the target version number, wherein the target version number is the version number corresponding to this full data dump; if the first version number is the first expected version number, updating the multiple second adaptive radix trees based on the first newly added metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers based on the first newly added metadata information to obtain the updated multiple second associated containers.

[0153] Optionally, after determining whether the first version number is the first expected version number based on the target version number, the method also includes: if the target version number is not the first expected version number, storing the first newly added metadata information in the target queue; reading the newly added metadata information in the incremental block again to obtain the second newly added metadata information; if the second version number corresponding to the second newly added metadata information is the first expected version number, updating the multiple second adaptive radix trees based on the second newly added metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers based on the second newly added metadata information to obtain the updated multiple second associated containers.

[0154] Optionally, after updating the multiple second adaptive radix trees according to the second new metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers according to the second new metadata information to obtain the updated multiple second associated containers, the method also includes: determining whether there is third new metadata information in the target queue, wherein the version number corresponding to the third new metadata information is the second expected version number; if the third new metadata information does not exist, repeating the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read; if the third new metadata information exists, updating the updated multiple second adaptive radix trees according to the third new metadata information, or updating the updated multiple second associated containers according to the third new metadata information, and repeating the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read.

[0155] Optionally, before scanning multiple first adaptive radix trees to obtain the first data information, the method also includes: obtaining historical time information of the last time the target system completed a full data dump; determining a time difference based on the historical time information and the current time information; and judging whether the current state of the target system meets the preset conditions for the full data dump based on the time difference and a time threshold condition in the preset conditions for the full data dump.

[0156] Optionally, judging whether the current state of the target system satisfies the preset conditions for the full data dump based on the time difference and the time threshold condition in the preset conditions for the full data dump includes: judging whether the time difference is greater than or equal to the time threshold in the time threshold condition; if the time difference is greater than the time threshold, calculating a first total value of the metadata information in the incremental block within the target time period, wherein the target time period is from the time point corresponding to the historical time information to the time point of the current time information; obtaining a second total value of the metadata information counted by the metadata management module in the target system; judging whether the current state of the target system satisfies the preset conditions for the full data dump based on the data relationship between the first total value and the second total value and the data relationship condition in the preset conditions for the full data dump.

[0157] Optionally, before scanning multiple first adaptive radix trees to obtain the first data information, the method also includes: obtaining the number of requests received by the target system to process metadata information; and judging whether the current state of the target system meets the preset conditions for the full data dump based on the number of requests and the request threshold condition in the preset conditions for the full data dump.

[0158] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0159] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that is initialized with the following method steps: if the current state of the target system meets the preset conditions for full data dump, multiple first adaptive radix trees are scanned to obtain first data information, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in multiple first adaptive radix trees; multiple first associated containers are scanned to obtain second data information, wherein each first associated container stores second metadata information of a data shard, the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; the first data information and the second data information are dumped to a target storage medium.

[0160] Optionally, scanning multiple first adaptive radix trees to obtain first data information includes: scanning multiple first adaptive radix trees to obtain metadata information corresponding to leaf nodes in the multiple first adaptive radix trees, node types of leaf nodes, and common prefix information in upper nodes corresponding to the leaf nodes; determining attribute information of the metadata information corresponding to the leaf nodes based on the node type of the leaf nodes, the number of leaves, and the common prefix information; and obtaining the first data information based on the attribute information and the metadata information corresponding to the leaf nodes.

[0161] Optionally, before dumping the first data information and the second data information to the target storage medium, the method also includes: scanning the coding table and the bitmap corresponding to the coding table in the target system to obtain third data information, wherein the coding table is used to compress and encode the metadata information when constructing multiple first adaptive radix trees, and the bitmap is used to describe the usage of the array in the coding table; dumping the third data information to the target storage medium.

[0162] Optionally, after dumping the first data information and the second data information to the target storage medium, the method further includes: determining whether the full data dump has been completed by using a completion identifier corresponding to the full data dump; if the full data dump has been completed, reading the first data information in the target storage medium, and constructing multiple second adaptive radix trees based on the first data information; reading the second data information in the target storage medium, and constructing multiple second associated containers based on the first data information.

[0163] Optionally, after constructing multiple second associated containers based on the first data information, the method also includes: reading the newly added metadata information in the incremental block to obtain the first newly added metadata information; obtaining a first version number corresponding to the first newly added metadata information, wherein the first version number is generated when the first newly added metadata information is added; determining whether the first version number is the first expected version number based on the target version number, wherein the target version number is the version number corresponding to this full data dump; if the first version number is the first expected version number, updating the multiple second adaptive radix trees based on the first newly added metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers based on the first newly added metadata information to obtain the updated multiple second associated containers.

[0164] Optionally, after determining whether the first version number is the first expected version number based on the target version number, the method also includes: if the target version number is not the first expected version number, storing the first newly added metadata information in the target queue; reading the newly added metadata information in the incremental block again to obtain the second newly added metadata information; if the second version number corresponding to the second newly added metadata information is the first expected version number, updating the multiple second adaptive radix trees based on the second newly added metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers based on the second newly added metadata information to obtain the updated multiple second associated containers.

[0165] Optionally, after updating the multiple second adaptive radix trees according to the second new metadata information to obtain the updated multiple second adaptive radix trees, or updating the multiple second associated containers according to the second new metadata information to obtain the updated multiple second associated containers, the method also includes: determining whether there is third new metadata information in the target queue, wherein the version number corresponding to the third new metadata information is the second expected version number; if the third new metadata information does not exist, repeating the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read; if the third new metadata information exists, updating the updated multiple second adaptive radix trees according to the third new metadata information, or updating the updated multiple second associated containers according to the third new metadata information, and repeating the step of reading the new metadata information in the incremental block until all the new metadata information in the incremental block has been read.

[0166] Optionally, before scanning multiple first adaptive radix trees to obtain the first data information, the method also includes: obtaining historical time information of the last time the target system completed a full data dump; determining a time difference based on the historical time information and the current time information; and judging whether the current state of the target system meets the preset conditions for the full data dump based on the time difference and a time threshold condition in the preset conditions for the full data dump.

[0167] Optionally, judging whether the current state of the target system satisfies the preset conditions for the full data dump based on the time difference and the time threshold condition in the preset conditions for the full data dump includes: judging whether the time difference is greater than or equal to the time threshold in the time threshold condition; if the time difference is greater than the time threshold, calculating a first total value of the metadata information in the incremental block within the target time period, wherein the target time period is from the time point corresponding to the historical time information to the time point of the current time information; obtaining a second total value of the metadata information counted by the metadata management module in the target system; judging whether the current state of the target system satisfies the preset conditions for the full data dump based on the data relationship between the first total value and the second total value and the data relationship condition in the preset conditions for the full data dump.

[0168] Optionally, before scanning multiple first adaptive radix trees to obtain the first data information, the method also includes: obtaining the number of requests received by the target system to process metadata information; and judging whether the current state of the target system meets the preset conditions for the full data dump based on the number of requests and the request threshold condition in the preset conditions for the full data dump.

[0169] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0170] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0171] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0172] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0173] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0174] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0175] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0176] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0177] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0178] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data dump method, characterized in that: include: If the current state of the target system meets the preset conditions for full data dump, multiple first adaptive radix trees are scanned to obtain first data information, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in the multiple first adaptive radix trees; Scanning a plurality of first associated containers to obtain second data information, wherein each first associated container stores second metadata information of a data slice, and the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; Dumping the first data information and the second data information to a target storage medium; After dumping the first data information and the second data information to the target storage medium, the method further includes: If the full data dump has been completed, reconstructing the second adaptive radix tree and the second associative container according to the first data information and the second data information in the target storage medium; The method further includes: reading the newly added metadata information in the incremental block to obtain first newly added metadata information; Obtaining a first version number corresponding to the first newly added metadata information, wherein the first version number is generated when the first newly added metadata information is added; Determine whether the first version number is the first expected version number according to the target version number, wherein the target version number is the version number corresponding to the current full data dump; If the first version number is the first expected version number, the multiple second adaptive radix trees are updated according to the first new metadata information to obtain multiple updated second adaptive radix trees, or the multiple second associated containers are updated according to the first new metadata information to obtain multiple updated second associated containers.

2. The method according to claim 1, characterized in that Scanning a plurality of first adaptive radix trees to obtain first data information includes: Scan the multiple first adaptive radix trees to obtain metadata information corresponding to leaf nodes in the multiple first adaptive radix trees, node types of the leaf nodes, and common prefix information in upper-level nodes corresponding to the leaf nodes; Determine attribute information of metadata information corresponding to the leaf node according to the node type, the number of leaves and the common prefix information of the leaf node; The first data information is obtained according to the attribute information and the metadata information corresponding to the leaf node.

3. The method according to claim 1, characterized in that Before dumping the first data information and the second data information to the target storage medium, the method further includes: Scanning a coding table in the target system and a bitmap corresponding to the coding table to obtain third data information, wherein the coding table is used to compress and encode metadata information when constructing a first adaptive radix tree, and the bitmap is used to describe the usage of an array in the coding table; The third data information is dumped to the target storage medium.

4. The method according to claim 1, characterized in that After dumping the first data information and the second data information to the target storage medium, the method further includes: Determine whether the full data dump has been completed by using the completion identifier corresponding to the full data dump; If the full data dump has been completed, reading the first data information in the target storage medium, and constructing a plurality of second adaptive radix trees according to the first data information; The second data information in the target storage medium is read, and a plurality of second associated containers are constructed according to the first data information.

5. The method according to claim 1, characterized in that: After determining whether the first version number is a first expected version number according to the target version number, the method further includes: If the target version number is not the first expected version number, storing the first newly added metadata information in the target queue; Reading the newly added metadata information in the incremental block again to obtain second newly added metadata information; If the second version number corresponding to the second newly added metadata information is the first expected version number, the multiple second adaptive radix trees are updated according to the second newly added metadata information to obtain multiple updated second adaptive radix trees, or the multiple second associated containers are updated according to the second newly added metadata information to obtain multiple updated second associated containers.

6. The method according to claim 5, characterized in that After updating the plurality of second adaptive radix trees according to the second newly added metadata information to obtain the updated plurality of second adaptive radix trees, or updating the plurality of second associated containers according to the second newly added metadata information to obtain the updated plurality of second associated containers, the method further includes: Determine whether there is third newly added metadata information in the target queue, wherein the version number corresponding to the third newly added metadata information is the second expected version number; If the third newly added metadata information does not exist, repeating the step of reading the newly added metadata information in the incremental block until all the newly added metadata information in the incremental block has been read; If the third newly added metadata information exists, the updated multiple second adaptive radix trees are updated according to the third newly added metadata information, or the updated multiple second associated containers are updated according to the third newly added metadata information, and the step of reading the newly added metadata information in the incremental block is repeated until all the newly added metadata information in the incremental block has been read.

7. The method according to claim 1, characterized in that Before scanning the plurality of first adaptive radix trees to obtain the first data information, the method further includes: Obtain historical time information of the last time the target system completed a full data dump; Determine a time difference based on the historical time information and the current time information; According to the time difference and the time threshold condition in the preset condition for the full data dump, it is determined whether the current state of the target system meets the preset condition for the full data dump.

8. The method according to claim 7, characterized in that Judging whether the current state of the target system meets the preset condition for full data dump according to the time difference and the time threshold condition in the preset condition for full data dump includes: Determine whether the time difference is greater than or equal to the time threshold in the time threshold condition; If the time difference is greater than the time threshold, then calculating a first total value of the metadata information in the incremental block within a target time period, wherein the target time period is from the time point corresponding to the historical time information to the time point of the current time information; Obtaining a second total value of metadata information counted by the metadata management module in the target system; According to the data relationship between the first total value and the second total value and the data relationship condition in the preset condition for the full data dump, it is determined whether the current state of the target system meets the preset condition for the full data dump.

9. The method according to claim 1, characterized in that: Before scanning the plurality of first adaptive radix trees to obtain the first data information, the method further includes: Obtaining the number of requests received by the target system to process metadata information; According to the number of requests and a request threshold condition in the preset condition for the full data dump, it is determined whether the current state of the target system meets the preset condition for the full data dump.

10. A data dump device, characterized in that: include: A first scanning unit is used to scan a plurality of first adaptive radix trees to obtain first data information if the current state of the target system meets the preset condition of full data dump, wherein each first adaptive radix tree is constructed by first metadata information corresponding to a data shard, the first metadata information is a fixed-length key-value pair, and the first data information at least includes metadata information corresponding to leaf nodes in the plurality of first adaptive radix trees; A second scanning unit is used to scan the plurality of first associated containers to obtain second data information, wherein each first associated container stores second metadata information of a data slice, and the second data information at least includes second metadata information, and the second metadata information is a variable-length key-value pair; A first dump unit, used for dumping the first data information and the second data information to a target storage medium; The device is further configured to, after dumping the first data information and the second data information to a target storage medium, reconstruct a second adaptive radix tree and a second associative container according to the first data information and the second data information in the target storage medium if the full data dump has been completed; The device also includes: a third reading unit, used to read the newly added metadata information in the incremental block to obtain the first newly added metadata information; a first acquisition unit, used to obtain the first version number corresponding to the first newly added metadata information, wherein the first version number is generated when the first newly added metadata information is added; a first determination unit, used to determine whether the first version number is the first expected version number based on the target version number, wherein the target version number is the version number corresponding to this full data dump; a first update unit, used to update the multiple second adaptive radix trees based on the first newly added metadata information if the first version number is the first expected version number to obtain the updated multiple second adaptive radix trees, or to update the multiple second associated containers based on the first newly added metadata information to obtain the updated multiple second associated containers.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the data dump method according to any one of claims 1 to 9.

12. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program, when running, executes the data dump method according to any one of claims 1 to 9.

Images