Data migration method and device and electronic equipment

Abstract

The invention relates to the technical field of databases, and particularly provides a data migration method and device and electronic equipment. The method comprises the following steps: in response to a data emigration event, finding metadata corresponding to prefixes associated with a fragment identifier to be emigrated according to a sequence of data layers from high to low, and migrating the found metadata to a target node; in the data migration process, if a metadata transaction which needs to be written into the to-be-migrated fragment corresponding to the to-be-migrated fragment identifier is received, writing the received metadata transaction into a specified storage space in a memory table MemTable of a database; and after metadata migration of the to-be-migrated fragment in the data layer is completed, migrating each metadata transaction recorded in the specified storage space to a target node. Therefore, batch query can be performed through the to-be-migrated fragment identifier, and newly received metadata transactions can be aggregated and cached, so that the data migration efficiency is improved, and consumed resources are reduced.

Description

Technical Field

[0001] This application relates to the field of database technology, specifically to a data migration method, apparatus, and electronic device. Background Technology

[0002] With the widespread application of distributed systems (such as distributed file systems and distributed storage systems) in big data storage and cloud computing, the scale of data is growing exponentially, and the need for migration of metadata shards between different nodes in distributed systems is becoming increasingly frequent.

[0003] Metadata storage in current distributed systems typically adopts a layered architecture (i.e., containing multiple data layers such as L0, L1, and L2). The metadata of each shard is stored in the corresponding data layer of the node in the form of key-value pairs (KV), and the metadata key is associated with the shard identifier to achieve shard-level management.

[0004] In related technologies, when migrating a shard between different nodes, the metadata of each shard is typically read one by one from the originating node, then aggregated and packaged, and sent to the target node over the network. This results in a long shard migration time and consumes excessive network and computing resources. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method, apparatus, and electronic device for data migration.

[0006] On one hand, this application provides a data migration method applied to a migration node in a distributed system. The migration node deploys a database containing multiple data layers in ascending order. Each data layer stores metadata contained in at least one corresponding shard using key-value pairs. The metadata in each data layer is arranged according to the order of the prefixes of the corresponding keys, so that multiple metadata corresponding to the same prefix are aggregated together. The prefix corresponding to any metadata is associated with the shard identifier corresponding to that metadata. The method includes:

[0007] In response to a data migration event, the metadata corresponding to the prefix associated with the shard identifier to be migrated is found in descending order of data layer, and the found metadata is migrated to the target node.

[0008] During the data migration process, if a metadata transaction that needs to be written to the shard to be migrated out corresponding to the shard identifier is received, the received metadata transaction will be written to the specified storage space in the database's memory table MemTable; the metadata transaction carries metadata; the specified storage space is only used to store the metadata transaction corresponding to the shard to be migrated out.

[0009] After the metadata migration of the shard to be migrated out in the data layer is completed, the metadata transactions recorded in the specified storage space will be migrated to the target node.

[0010] In one implementation, the database includes multiple data layers from the first layer to the nth layer; n is a positive integer;

[0011] Following the data layer order from high to low, locate the metadata corresponding to the prefix associated with the shard identifier to be migrated out, and migrate the found metadata to the target node, including:

[0012] Create snapshots of the data in each data layer of the database and lock the metadata associated with the snapshots to prevent the metadata associated with the snapshots from being deleted;

[0013] Based on the snapshot, traverse each data layer in descending order of data layer and query the metadata corresponding to the prefix associated with the shard identifier to be migrated out;

[0014] When reading metadata from the first data layer, deduplication is performed on the metadata read from the first data layer;

[0015] The filtered metadata is migrated to the target node via remote direct memory access.

[0016] In one implementation, during the data migration process, if a metadata transaction that needs to be written to the shard to be migrated out corresponding to the shard identifier is received, the received metadata transaction is written to a designated storage space in the database's in-memory table MemTable, including:

[0017] Request an initial memory aggregation cache;

[0018] During the data migration process, the following steps are executed repeatedly:

[0019] When a metadata request for a shard to be migrated out is received, the corresponding metadata transaction is obtained based on the metadata request;

[0020] Write metadata transactions to the write-ahead log;

[0021] After the write-ahead log is successfully written, determine the current state of the memory aggregation cache:

[0022] If the number of metadata transactions in the current memory aggregation cache does not reach the set threshold and the total data volume does not reach the set size threshold, then the metadata transactions will be copied to the current memory aggregation cache.

[0023] If the number of metadata transactions in the memory aggregation cache reaches a set threshold or the total data volume reaches a set size threshold, a new memory aggregation cache will be requested, and the metadata transactions will be copied to the new memory aggregation cache. The specified storage space includes the initial memory aggregation cache and the new memory aggregation cache.

[0024] In one embodiment, the method further includes:

[0025] When the amount of data in the metadata transaction to be migrated in the specified storage space is lower than a preset threshold, the metadata request for the fragment to be migrated out is suspended.

[0026] Migrate the metadata transactions to be migrated, along with the end markers, to the target node;

[0027] After receiving the end response from the target node in response to the end marker, the migration end indication file containing the identifier of the shard to be migrated out is stored in the database; the migration end indication file is used to indicate that the migration of the shard to be migrated out is complete.

[0028] Remove the metadata corresponding to the prefix associated with the shard identifier to be migrated from the database;

[0029] Send a mapping update request to the control module of the distributed system so that the control module updates the mapping between shards and nodes, thereby updating the mapping between the shard to be migrated out and the node to be migrated out to the mapping between the shard to be migrated out and the target node.

[0030] In one embodiment, the method further includes:

[0031] Use any one of the following:

[0032] If it is determined that the node to be migrated out is abnormal and the metadata transaction migration is not completed, the shard data migration operation will be re-executed for the shard to be migrated out.

[0033] If it is determined that the outgoing node is abnormal and the end marker was sent successfully, a corresponding relationship update request is sent to the control module of the distributed system.

[0034] If an anomaly is detected in the target node during the migration of metadata in the data layer, the shard data migration operation will be re-executed for the shard to be migrated out.

[0035] If an anomaly is detected in the target node during the metadata transaction migration, the migration message is resent.

[0036] On one hand, this application provides a data migration apparatus applied to a migration node in a distributed system. The migration node is equipped with a database containing multiple data layers in ascending order. Each data layer stores metadata contained in at least one corresponding shard using key-value pairs. The metadata in each data layer is arranged according to the order of the prefixes of the corresponding keys, so that multiple metadata corresponding to the same prefix are aggregated together. The prefix corresponding to any metadata is associated with the shard identifier corresponding to that metadata. The apparatus includes:

[0037] The lookup unit is used to respond to a data migration event, find the metadata corresponding to the prefix associated with the identifier of the shard to be migrated, and migrate the found metadata to the target node in descending order of data layers.

[0038] The receiving unit is used to write the received metadata transaction into the specified storage space of the MemTable in the database if it receives a metadata transaction that needs to be written into the shard to be migrated out corresponding to the shard identifier during the data migration process; the metadata transaction carries metadata; the specified storage space is only used to store the metadata transaction corresponding to the shard to be migrated out.

[0039] The migration unit is used to migrate the metadata transactions recorded in the specified storage space to the target node after the metadata migration of the shard to be migrated out in the data layer is completed.

[0040] In one implementation, the database includes multiple data layers from the first layer to the nth layer; n is a positive integer;

[0041] The lookup unit is used for:

[0042] Create snapshots of the data in each data layer of the database and lock the metadata associated with the snapshots to prevent the metadata associated with the snapshots from being deleted;

[0043] Based on the snapshot, traverse each data layer in descending order of data layer and query the metadata corresponding to the prefix associated with the shard identifier to be migrated out;

[0044] When reading metadata from the first data layer, deduplication is performed on the metadata read from the first data layer;

[0045] The filtered metadata is migrated to the target node via remote direct memory access.

[0046] In one embodiment, the receiving unit is used to:

[0047] Request an initial memory aggregation cache;

[0048] During the data migration process, the following steps are executed repeatedly:

[0049] When a metadata request for a shard to be migrated out is received, the corresponding metadata transaction is obtained based on the metadata request;

[0050] Write metadata transactions to the write-ahead log;

[0051] After the write-ahead log is successfully written, determine the current state of the memory aggregation cache:

[0052] If the number of metadata transactions in the current memory aggregation cache does not reach the set threshold and the total data volume does not reach the set size threshold, then the metadata transactions will be copied to the current memory aggregation cache.

[0053] If the number of metadata transactions in the memory aggregation cache reaches a set threshold or the total data volume reaches a set size threshold, a new memory aggregation cache will be requested, and the metadata transactions will be copied to the new memory aggregation cache. The specified storage space includes the initial memory aggregation cache and the new memory aggregation cache.

[0054] In one embodiment, the migration unit is further configured to:

[0055] When the amount of data in the metadata transaction to be migrated in the specified storage space is lower than a preset threshold, the metadata request for the fragment to be migrated out is suspended.

[0056] Migrate the metadata transactions to be migrated, along with the end markers, to the target node;

[0057] After receiving the end response from the target node in response to the end marker, the migration end indication file containing the identifier of the shard to be migrated out is stored in the database; the migration end indication file is used to indicate that the migration of the shard to be migrated out is complete.

[0058] Remove the metadata corresponding to the prefix associated with the shard identifier to be migrated from the database;

[0059] Send a mapping update request to the control module of the distributed system so that the control module updates the mapping between shards and nodes, thereby updating the mapping between the shard to be migrated out and the node to be migrated out to the mapping between the shard to be migrated out and the target node.

[0060] In one embodiment, the migration unit is further configured to:

[0061] Use any one of the following:

[0062] If it is determined that the node to be migrated out is abnormal and the metadata transaction migration is not completed, the shard data migration operation will be re-executed for the shard to be migrated out.

[0063] If it is determined that the outgoing node is abnormal and the end marker was sent successfully, a corresponding relationship update request is sent to the control module of the distributed system.

[0064] If an anomaly is detected in the target node during the migration of metadata in the data layer, the shard data migration operation will be re-executed for the shard to be migrated out.

[0065] If an anomaly is detected in the target node during the metadata transaction migration, the migration message is resent.

[0066] On one hand, this application provides an electronic device, including:

[0067] Processor; and

[0068] The memory stores computer instructions that cause the processor to perform the steps of the methods provided in the various alternative implementations of any of the data migrations described above.

[0069] On one hand, embodiments of this application provide a computer-readable storage medium storing computer instructions for causing a computer to perform the steps of the methods provided in various alternative implementations of any of the above-described data migration methods.

[0070] On one hand, this application provides a computer program product including computer-readable code or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device performs the steps of the method provided in any of the above-described alternative implementations of data migration.

[0071] The data migration method in this embodiment includes, in response to a data migration event, finding the metadata corresponding to the prefix associated with the shard identifier to be migrated out, in descending order of the data layer, and migrating the found metadata to the target node; during the data migration process, if a metadata transaction that needs to be written to the shard corresponding to the shard identifier to be migrated out is received, the received metadata transaction is written to a designated storage space in the MemTable of the database; the metadata transaction carries metadata; the designated storage space is only used to store the metadata transaction corresponding to the shard to be migrated out; after the metadata migration of the shard to be migrated out in the data layer is completed, each metadata transaction recorded in the designated storage space is migrated to the target node. In this way, batch queries can be performed using the shard identifier to be migrated out, and newly received metadata transactions can be aggregated and cached, improving data migration efficiency and reducing resource consumption. Attached Figure Description

[0072] Figure 1 This is an example diagram of a metadata distribution in an embodiment of this application.

[0073] Figure 2This is a flowchart of a data migration method according to an embodiment of this application.

[0074] Figure 3 This is a schematic diagram of a fragment migration in an embodiment of this application.

[0075] Figure 4 This is a schematic diagram of a metadata key in an embodiment of this application.

[0076] Figure 5 This is an example diagram of a segment ownership switching in an embodiment of this application.

[0077] Figure 6 This is a structural block diagram of a data migration apparatus according to an embodiment of this application.

[0078] Figure 7 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0079] The technical solution of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, the technical features involved in the different embodiments of this application described below can be combined with each other as long as they do not conflict with each other.

[0080] The following explains some terms used in the embodiments of this application:

[0081] A MemTable is an in-memory data structure in a database system used to store and manage recent write operations. It is typically an ordered collection of key-value pairs. When data is written to the database, it is first written to the MemTable. When the MemTable reaches a certain size or meets other conditions, it is persisted to a Sorted String Table (SSTable) on disk, and the MemTable is then cleared. This mechanism improves write performance while maintaining data persistence and consistency. MemTables are widely used in storage engines such as Level Database (LevelDB) and Rocks Database (RocksDB).

[0082] Log-Structured Merge Tree (LSM-Tree): This is a data structure widely used in databases and file systems for efficiently handling large volumes of write operations. LSM-Tree first writes data to a MemTable in memory. When the MemTable reaches a certain size, it is persisted to an SSTable on disk. The SSTable on disk is read-only; new data is always appended, avoiding frequent random write operations and thus improving write performance. LSM-Tree optimizes read performance by periodically compacting the SSTable on disk, reducing the number of disk accesses during reads. This data structure is widely used in storage engines such as LevelDB and RocksDB.

[0083] SSTable is a file format used in LSM-Tree data structures for storing persistent data. When the MemTable in memory reaches a certain size, it is persisted to an SSTable on disk. Data in an SSTable is sorted by key, making range queries and key-value lookups highly efficient. Because SSTables are read-only, new data is always appended to a new SSTable, avoiding frequent random write operations and thus improving write performance. SSTables are widely used in storage engines such as LevelDB and RocksDB.

[0084] A Metadata Server (MDS) is a server in a Distributed File System (DFS) used to manage metadata. The MDS is typically responsible for maintaining file and directory attribute information, such as filename, permissions, location, and the distribution of file blocks. When a client needs to access a file, it first requests the file's metadata from the MDS, and then reads or writes data directly from the data nodes based on the information returned by the MDS. The MDS plays a crucial role in distributed systems such as Ceph and Lustre, improving system scalability and performance through centralized metadata management.

[0085] RocksDB is a high-performance, persistent key-value store engine suitable for applications that require efficient processing of large datasets, especially excelling in data persistence and high-speed read / write operations.

[0086] Write-Ahead Log (WAL) is a database technique used to ensure data consistency and durability. Its core mechanism involves recording all changes to a log file before actual data updates, ensuring that the database can recover data from the log in the event of system failures, preventing data loss or corruption. This technique is widely used in database systems such as PostgreSQL and MySQL, and is crucial for improving database reliability and performance.

[0087] Scaling Up: In distributed systems, scaling up typically refers to increasing the system's storage capacity and processing power. Scaling up can be achieved in several ways: 1. Horizontal Scaling (Scale Out): This is the most common scaling method in distributed systems. It expands the system's storage capacity and processing power by adding more nodes (such as servers or storage devices). The newly added nodes can immediately begin receiving data and processing requests, thereby distributing the load on existing nodes and improving overall performance. 2. Vertical Scaling (Scale Up): Although less common than horizontal scaling in distributed systems, vertical scaling is also possible. It involves upgrading the hardware of existing nodes, such as increasing memory, using a faster Central Processing Unit (CPU), or a larger hard drive, to improve the performance and storage capacity of a single node. 3. Data Redistribution: During horizontal scaling, the system may need to redistribute data to maintain a uniform distribution and load balance. This usually involves data replication and migration to ensure that the load on all nodes is roughly equal, avoiding hotspots. 4. Elastic Scaling: Some modern distributed systems support automatic or manual elastic scaling, which dynamically adjusts resources based on system load. When the load increases, the system automatically adds nodes; when the load decreases, the system automatically reduces nodes to optimize resource usage and cost. During scaling, distributed systems typically utilize techniques such as consistent hashing, partitioning, replication, and data migration to ensure data integrity and high system availability.

[0088] In the context of the rapid development of Artificial Intelligence (AI) and big data technologies, distributed systems, as key infrastructure supporting massive data storage and processing, have become particularly important in terms of performance, scalability, and reliability. The widespread adoption of AI applications, such as deep learning, natural language processing, and image recognition, has generated unprecedented volumes of data, placing higher demands on storage systems. Distributed systems not only need to handle the storage of large-scale data but also support high-concurrency read and write operations, ensuring data consistency and integrity.

[0089] However, faced with the data challenges brought by AI, traditional distributed systems encounter numerous difficulties when scaling up:

[0090] High resource consumption: AI applications generate massive amounts of data, and the synchronization and updating of full metadata during the expansion process requires a large amount of computing, storage, and network resources, leading to resource shortages.

[0091] Significant business impact: Updating metadata may conflict with critical business processes such as AI model training and data preprocessing, affecting business continuity and performance.

[0092] Low expansion efficiency: Due to the need to process a large amount of metadata, the expansion process takes a long time, affecting the system's response speed and overall performance, thus restricting the efficiency and innovation of AI applications.

[0093] In view of the deficiencies of the above-mentioned related technologies, this application provides a data migration method, apparatus and electronic device, which aims to improve data migration efficiency and reduce data migration cost.

[0094] This application provides a data migration method that can be applied to electronic devices. This application does not limit the type of electronic device; it can be any suitable type of device, such as terminal devices and servers. This application will not elaborate further on this.

[0095] The embodiments of this application can be applied to distributed systems, which can be distributed file systems or distributed storage systems. A distributed system includes multiple nodes, and metadata from a shard of any node (the outgoing node) can be migrated to another node (the target node). The outgoing node and the target node are different nodes in the distributed system.

[0096] Each node includes a metadata service and a database. Optionally, the metadata service manages metadata and can be an MDS (Metadata Storage Service). The database stores the metadata for the corresponding shard and can be RocksDB. RocksDB itself is a high-performance, single-machine key-value (KV) system and does not have the ability to scale. Therefore, metadata migration is mainly implemented at the MDS layer of the distributed system.

[0097] The database contains MemTable and multiple data layers that increase in hierarchy. The database can adopt an LSM-Tree architecture. The data aggregation degree of higher-level data layers (such as L2 layer) in the database is higher than that of lower-level data layers (such as L1 layer). Each data layer stores metadata in key-value pair format and sorts the metadata according to the order of the keys, such as lexicographical order of the key prefixes.

[0098] Lexicographical order is a method of arranging character sequences (such as letters, strings, and number strings) in an ordered manner, mimicking the sorting rules humans use when looking up words in a dictionary. Its core logic is: starting from the "first position" of the sequence, compare characters one by one, prioritizing the differences between characters at earlier positions to determine the overall order; if the preceding characters are identical, continue comparing the next position until a difference is found or the sequence ends.

[0099] It should be noted that the metadata of a distributed system is evenly distributed to each node according to a certain algorithm to leverage the parallel processing capabilities of multiple nodes. Each node has an MDS to manage all the metadata of that node.

[0100] In one implementation, the distributed system distributes data evenly across the nodes of the cluster (distributed system) by breaking it down into 1024 shards using consistent hashing. Each node manages certain shards, and this mapping (correspondence) is recorded in a global map within the distributed system, accessible to the MDS of each node. This global map can be stored in the control module of the distributed system.

[0101] Each node's MDS can internally use RocksDB as the key-value storage engine, storing metadata of the actual data in these shards, including file system inodes and persistent layer indexes corresponding to the data.

[0102] For example, see Figure 1 The diagram shown is an example of a metadata distribution. Figure 1 As shown in the figure, the metadata of the entire distributed system is distributed in 1024 groups, and the MDS on each node is responsible for managing the data and metadata of a certain group.

[0103] When scaling up nodes, the hash value of each metadata element itself does not change. However, the nodes distributed by shards may change due to changes in the physical location of the intervals, that is, the mapping relationship in the global map will change. For example, if there was originally a storage cluster with 3 nodes, when adding 1 more node, the old nodes need to migrate some shards to the fourth node so that the fourth node can take charge of some shard management work.

[0104] See Figure 2 The diagram shows a flowchart of a data migration method according to an embodiment of this application, applied to a migration node in a distributed system. The migration node deploys a database containing multiple data layers arranged hierarchically. Each data layer stores metadata contained in at least one corresponding shard using key-value pairs. The metadata in each data layer is arranged according to the order of the prefixes of the corresponding keys, so that multiple metadata corresponding to the same prefix are aggregated together. The prefix corresponding to any metadata is associated with the shard identifier corresponding to that metadata. The following section combines... Figure 2 The method is described below, and the specific implementation process is as follows:

[0105] Step 201: In response to the data migration event, find the metadata corresponding to the prefix associated with the shard identifier to be migrated, in descending order of data layer, and migrate the found metadata to the target node.

[0106] In one implementation, when performing step 201, the following steps may be adopted:

[0107] S2011: Create snapshots of the data in each data layer of the database and lock the metadata associated with the snapshots to prevent the metadata associated with the snapshots from being deleted.

[0108] S2012: Based on snapshots, traverse each data layer in descending order of data layer and query the metadata corresponding to the prefix associated with the shard identifier to be migrated out.

[0109] In one implementation, a special iterative read interface is set up in the database. The key input parameter of this iterative read interface is the fragment identifier, which is used to iteratively return the data blocks of the SST file matched by the fragment identifier. The returned data blocks are in the order from high level to low level, such as L2 first, then L1, and finally L0. For example, the iterative read interface can be migration_iter_get.

[0110] S2013: When reading metadata from the first data layer, deduplicatize the metadata read from the first data layer.

[0111] The database comprises multiple data layers from the first layer to the nth layer, where n is a positive integer. In this embodiment of the application, the data migration process is illustrated using only RocksDB as the database, which contains MemTable, L0, L1, and L2. L0 is the first layer.

[0112] S2014: Migrate the filtered metadata to the target node via Remote Direct Memory Access (RDMA).

[0113] See Figure 3 The diagram illustrates a shard migration process. The distributed system originally consisted of three nodes: the originating node, the target node, and a third node. Metadata was divided into 1024 shards using a hash algorithm, and these 1024 shards were stored in their respective nodes. A global map was established to map the nodes and shards. When a fourth node was added during expansion, this mapping was updated. For example, the mapping of shard 4 on the first node, after recalculation, would fall on the fourth node. Therefore, all files whose hash calculations all belong to shard 4 needed to be migrated from the first node to the fourth node.

[0114] When performing data migration at the emigration node, the emigration node's MDS needs to call the RocksDB read interface of the emigration node to obtain the full key-value pairs. Under related technologies, each key needs to be read individually, then aggregated and packaged before being sent over the network to the target node of the data to be migrated in. However, in this embodiment, the key reading operation is aggregated to reduce overall overhead. Furthermore, the aggregation is not performed on a batch read at a higher layer, but rather by analyzing the internal principles of RocksDB. Firstly, when all file metadata is initially written to RocksDB, it needs to be prefixed with a group number (i.e., shard identifier). Then, after the distributed system processes business for a period of time, metadata with the same shard identifier (e.g., shard number) prefix will gradually aggregate together. Figure 4 The diagram shows a type of metadata key that can be used for batch queries in RocksDB with the fragid as a prefix.

[0115] Let's take RocksDB, which includes MemTable, L0, L1, and L2, as an example. When writing key-value pairs through RocksDB, the first thing written is the MemTable, followed by lexicographical sorting. According to the principles of RocksDB, once the MemTable reaches a certain size, it will be flushed to disk, forming an immutable SST file, which becomes part of the L0 layer.

[0116] Layer 0 SST files are stored in the order they were written and typically contain a large number of overlapping key-value pairs. These Layer 0 SST files are aggregated and deduplicated through compaction and then merged into the higher-level Layer 1 to improve query performance and reduce disk usage.

[0117] At L1 level, multiple L0 level SST files are merged into a larger SST file, and so on, with L1 continuing to aggregate towards L2 level. These L1 and L2 SST files are arranged completely in lexicographical order, so an SST file can be understood as a data block with a lexicographical prefix interval, aggregating data with a certain lexicographical order prefix.

[0118] Therefore, key data prefixed with shard 4 may be stored in all files in L0, some SST files below L1, and some files in L2. The data in the SST files in L1 and L2 layers is centrally stored, exhibiting excellent aggregation. Therefore, if you want to migrate data with certain prefixes to another RocksDB, you don't need to completely find and migrate every single key matching the prefix. Instead, you can migrate the SST files matching the prefixes and then remove some redundant or irrelevant data.

[0119] Furthermore, a data receiving interface is also created in the target node, and the received metadata is appended to the end of the MemTable through this data receiving interface. Data can also be merged, aggregated and deduplicated through flushing and compaction to gradually sink the data.

[0120] For example, the MDS of the outgoing node reads the datablocks of the SST file one by one and sends them directly to the target node using RDMA network. The RocksDB of the target node also implements a new interface for receiving SST file datablocks (i.e., a data receiving interface) and appends the received datablocks to the end of the MemTable, such as... Figure 3 As shown, SST23 and SST24 from the L2 layer of the migrated node may be placed in the L1 and L0 layers of the target node.

[0121] During the migration of SST files, RocksDB requires certain controls. Specifically, during the migration of data at the data layer, compaction needs to be disabled to ensure the stability of the migrated SST files and prevent missing data blocks. Then, during the migration of L0 data, flush needs to be disabled to ensure the migrated data is stable and unmodified. Finally, after migrating all SST files, the target node needs to perform some cleanup operations because the migrated SST files may contain some non-required keys, which need to be deleted. Based on the previous principle of grouping metadata by prefix, this only requires calling RocksDB's batch delete interface to delete prefixes that do not belong to this node. Then, each node needs to re-enable compaction for data integration.

[0122] However, this has several disadvantages. First, disabling compaction during migration can easily lead to a significant accumulation of data in L0 during the migration process, resulting in an excessively large L0 layer. Second, disabling flush during L0 migration can lead to excessive data in MemTable, affecting query efficiency. Third, L0, L1, and L2 contain duplicate data, which will increase the merging overhead after migration.

[0123] To address the aforementioned issues, a dedicated iterator for data migration was implemented during the data migration process, eliminating the need to restrict RocksDB's compaction and flush operations.

[0124] In one implementation, a migration iterator is created. This iterator takes a snapshot of the data in each data layer and records the identifier (ID). It references the relevant MemTable and the SST files involved in L1 and L2. Then, normal compaction and flushing are performed, but the original files are not deleted after merging, ensuring that all data is traversed. During iteration, data is read from oldest to newest, first traversing L2, then L1, and finally L0. Even if some data is merged into L2, the final order is guaranteed to be correct, and new key-value pairs overwrite older ones. Since L2 and L1 layers do not overlap, data blocks matching the prefix range are read and placed in the memory pool. A pointer is then passed to the MDS layer for network transmission via RDMA, significantly reducing CPU processing.

[0125] For Layer 0, in-memory sorting and deduplication are performed, and data with prefix matching is aggregated into a memory pool for transmission. This ensures cluster stability and prevents resource usage amplification due to limitations in compaction and flushing. After the cold data migration is complete and successful, the migration iterator can be released, i.e., the snapshot reference is released, and redundant files can be normally evicted.

[0126] When a node migrates out, MDS needs to call RocksDB's read function to obtain the full key-value pair. Under similar technologies, this involves reading each key individually, aggregating and packaging them, and then sending them to the remote node over the network. The optimization approach in this embodiment is to aggregate the key read operations, reducing overall overhead. Furthermore, the aggregation is not performed on a batch read at a higher layer, but rather by analyzing the internal workings of RocksDB. Firstly, when all file metadata is initially written to RocksDB, it needs to be prefixed with the group number. After the distributed system has been in use for a period of time, the data from each shard number prefix will gradually aggregate together. For example... Figure 4 The metadata key shown can be used for batch queries in RocksDB by using the fragid as a prefix.

[0127] Step 202: During the data migration process, if a metadata transaction that needs to be written to the shard to be migrated out corresponding to the shard identifier is received, the received metadata transaction is written to the specified storage space in the MemTable of the database; the metadata transaction carries metadata; the specified storage space is only used to store the metadata transaction corresponding to the shard to be migrated out.

[0128] In one implementation, an initial memory aggregation cache is requested; during the data migration process, the following steps are executed repeatedly:

[0129] When a metadata request for a shard to be migrated is received, the corresponding metadata transaction is obtained based on the metadata request; the metadata transaction is written to the write-ahead log; after the write-ahead log is successfully written, the status of the current memory aggregation cache is determined: if the number of metadata transactions in the current memory aggregation cache has not reached the set threshold and the total data volume has not reached the set size threshold, the metadata transaction is copied to the current memory aggregation cache; if the number of metadata transactions in the memory aggregation cache reaches the set threshold or the total data volume reaches the set size threshold, a new memory aggregation cache is requested, and the metadata transaction is copied to the new memory aggregation cache; the specified storage space includes the initial memory aggregation cache and the new memory aggregation cache.

[0130] In practical applications, both quantity thresholds and size thresholds can be set according to the actual application scenario, and no restrictions are imposed here.

[0131] The target node's MemTable is configured with an aggregation event interface. It receives metadata transactions sent by the migrating node through the aggregation event interface and writes them directly into its own MemTable without additional encoding or decoding. When the amount of data in the MemTable reaches the set flush threshold, it is automatically flushed to disk storage.

[0132] In this way, an interface for appending multiple aggregation events to RocksDB incremental data (i.e., the aggregation event interface) has been added. It is similar to the ordinary write interface, and the data is put into the MemTable of the node. However, no additional transaction encoding and decoding is required. When the aggregation reaches a certain amount, it will be flushed down.

[0133] In this embodiment of the application, the overall metadata migration scheme can be decomposed into three major stages: migrating cold data, migrating incremental data (i.e., metadata transactions), and switching shard ownership.

[0134] Figure 4 In the cold data migration phase, aggregated data blocks in L2 and L1 of RocksDB that match the shard number (i.e., shard identifier) ​​are migrated. During this phase, metadata input / output (IO) (i.e., metadata requests) with the corresponding shard number can continue to be processed. While writing the metadata transaction corresponding to the metadata IO to RocksDB, a copy of the RocksDB metadata transaction is placed in memory (i.e., MemTable). After the cold data migration is completed, the newly added metadata transactions are migrated.

[0135] For business optimization, when creating the migration iterator, the business needs to match the sharded IO identification during scaling. If a match is found, an incremental metadata operation transaction is copied after a successful Write-Ahead Log (WAL) in RocksDB, and aggregated and stored in the memory pool. For example, aggregation can be performed according to a total length of 128 metadata transactions (i.e., setting a quantity threshold) or 1MB (i.e., setting a size threshold) to avoid data copying during subsequent incremental sending phases. Therefore, all IO can continue to be processed on this MDS.

[0136] Step 203: After the metadata migration of the shard to be migrated out in the data layer is completed, migrate each metadata transaction recorded in the specified storage space to the target node.

[0137] In one implementation, the following steps can be used when switching ownership of a fragment:

[0138] When the amount of metadata transactions to be migrated in the specified storage space is lower than a preset threshold, the metadata request for the shard to be migrated is suspended; the metadata transactions to be migrated and the end marker are migrated to the target node; after receiving the end response returned by the target node for the end marker, the migration end indication file containing the identifier of the shard to be migrated is stored in the database; the migration end indication file is used to indicate that the migration of the shard to be migrated is complete; the metadata corresponding to the prefix associated with the identifier of the shard to be migrated is deleted from the database; a correspondence update request is sent to the control module of the distributed system so that the control module updates the correspondence between shards and nodes, so that the correspondence between the shard to be migrated and the migrated node is updated to the correspondence between the shard to be migrated and the target node.

[0139] In practical applications, the preset threshold can be set according to the actual application scenario, and there are no restrictions here.

[0140] For example, if the incremental data to be migrated (i.e. the metadata transactions to be migrated) in the migrating node is less than 4M (i.e., the preset threshold), then it can enter the third stage (i.e., the sharding assignment stage). In this stage, MDS needs to hold the metadata requests related to shard 4 sent from the front end, and then migrate the last remaining incremental data, carrying the end mark (i.e., the end mark). When the target node receives the incremental data write request with this end mark, it considers the data migration of the shard to be complete.

[0141] After receiving the end response from the target node based on the end flag, the outgoing node persists a completed file marked with the shard number (i.e., the migration end indication file) locally, and then reports it to the control module used to control the global map to confirm the completion of the migration task. The outgoing node then cleans up the data on the original shard 4 locally and submits a batch delete transaction to RocksDB, prefixed with shard 4, to delete the data on the original shard 4.

[0142] After receiving the report information (i.e., the mapping update request) from the migrated node, the control module will publish the latest mapping (i.e. the latest mapping relationship) again to notify the global shard 4 that the migration has been completed and the target node can process the business of shard 4 normally.

[0143] See Figure 5 The diagram shown is an example of a fragmentation ownership switching. Figure 5During the migration process, after the cold data migration is complete, the aggregated transactions in the memory pool will continue to be migrated sequentially until the remaining data size is less than 4MB (i.e., the preset threshold). At this point, a hang flag will be set for the corresponding shard in this MDS. The hang flag is used to indicate that the shard is in a "suspended" state. This prevents the metadata I / O of the corresponding shard from being processed, meaning no more aggregated transactions will be generated in the memory pool. The incremental migration (i.e., metadata transaction migration) will then move the last small portion of data to the new node (i.e., the target node).

[0144] After receiving a successful response from the target node, the outgoing node removes the hang flag and, based on the pending metadata request for the shard to be migrated, returns a retry error code to the client, causing the client to resend the metadata request. At this point, the client, based on the view (i.e., the updated mapping), finds that the MDS to be sent to the target node needs to be sent according to the updated mapping, and then sends a new metadata request to the target node, thus resuming normal metadata request processing. This process is very fast; the migration and persistent return of 4MB of data via RDMA has a latency of only tens of milliseconds, which is virtually imperceptible to the business, truly achieving minimal business impact.

[0145] Furthermore, if either the originating node or the target node malfunctions, corresponding migration adjustments can be performed, using any one of the following methods:

[0146] If it is determined that the node to be migrated out is abnormal and the metadata transaction migration is not completed, the shard data migration operation will be re-executed for the shard to be migrated out.

[0147] If it is determined that the outgoing node is abnormal and the end marker was sent successfully, a corresponding relationship update request is sent to the control module of the distributed system.

[0148] If an anomaly is detected in the target node during the migration of metadata in the data layer, the shard data migration operation will be re-executed for the shard to be migrated out.

[0149] If an anomaly is detected in the target node during the metadata transaction migration, the migration message used for migrating the metadata transaction will be resent.

[0150] In this embodiment, some handling is required when an exception occurs at each stage. First, if an exception occurs in stage one (i.e., the cold data migration stage), and either the outgoing node or the target node experiences a node or process exception, the process will be re-executed. The current implementation supports reentrancy, and repeated migration of L2 and L1 data will not cause consistency issues because they are written in order. Therefore, deduplication can be achieved later through compaction.

[0151] Then, if the exception occurs in phase two (i.e., the metadata transaction migration phase), if the target node is abnormal, the migration node will retry the migration message; if the migration node is abnormal, the entire shard will be migrated again.

[0152] If an anomaly occurs in Phase 3 (i.e., the sharding assignment phase), if the target node is abnormal, the outgoing node will retry the migration message; if the outgoing node is abnormal, the entire shard will be migrated again.

[0153] If the end marker is confirmed to have been written successfully, but the global map reporting fails, after the exception is recovered, when the migrated node re-executes the process and finds that the end marker has been written, it can directly report the completion of the global map again and publish the final mapping.

[0154] The following example illustrates data migration. Assume a distributed system stores 10 billion files, which are divided into 1024 groups according to a fully random equilibrium distribution. Each group is then stored on a 3-node server. Each group contains approximately 9,765,625 key-value pairs (KVs). Each node has 12 hard drives, and each hard drive needs to handle 28.4 KVs for each group. Some nodes may have 28 groups, while others may have 29.

[0155] Each RocksDB instance is started, with each metadata key and value being 128 bytes. Each RocksDB instance stores approximately 270 million key-value pairs. During scaling, what is actually migrated is a group of key-value pairs, i.e., 9,765,625 pairs.

[0156] If RocksDB is configured with L2, L1, and L0, with 90% of the data in L2, 8% in L1, and 2% in L0 and memory, then the L0 layer will require approximately 0.02 * 9.76 million ≈ 195,200 key-value pairs.

[0157] Assuming each SST file in layer L0 contains 1000 key-value pairs, then layer L0 requires approximately 195,520 / 1000 ≈ 195 SST files. At layer L1, approximately 0.08 * 9.76 million ≈ 780,800 key-value pairs are needed. Assuming each SST file contains 500,000 key-value pairs, then layer L1 requires approximately 780,800 / 500,000 ≈ 2 SST files. At layer L2, approximately 0.9 * 9.76 million ≈ 8.784 million key-value pairs are needed. Assuming each SST file contains 5 million key-value pairs, then layer L2 requires approximately 8.784 million / 5 million ≈ 2 SST files. Therefore, in this scenario, 195 L0 files (128MB each), 2 L1 files (1GB each), and 2 L2 files (10GB each) need to be moved.

[0158] If each large SST file is migrated by reading 1MB of datablock data at a time, it would require approximately 195*128 + 2*1024 + 2*10*1024 = 47488 1MB IO operations. The number of IO operations for the newly added node is 48,000, which is several tens of times less than the original 9.76 million KV reads.

[0159] At the migration node, there are also read I / O operations. In this embodiment, reading only 1MB requires approximately 48,000 operations, while in related technologies, it requires 9.76 million reads. The significantly reduced I / O count reduces CPU overhead. Furthermore, the migration of incremental data is based on the location of foreground I / O traffic. In memory, incremental events are recorded in aggregate at 1MB or 128 I / O events, ensuring minimal writes to the other end during scaling. Overall, the number of I / O operations is approximately 1 / 200th of that in related technologies.

[0160] Distributed systems shard metadata and distribute it evenly across each node, storing it as key-value pairs in RocksDB on each node. During scaling, some key-value metadata needs to be migrated. In related technologies, this is done by reading the relevant key-value pairs from RocksDB and then aggregating and synchronizing them at the business layer. However, in this embodiment, a completely different strategy is employed compared to the common scaling method of reading and synchronizing metadata key-value pairs one by one.

[0161] The optimization focuses on two aspects: firstly, improving the efficiency of distributed system migration; and secondly, shortening the time for switching master nodes, thus reducing the impact on front-end services. The core optimization points are mainly three: First, by combining the lexicographical order and aggregation characteristics of SST files, large blocks of data with prefix distribution are migrated, significantly reducing IO operations and migration time. Furthermore, a snapshot approach is cleverly used to ensure normal data aggregation, flushing, and compaction. Second, incremental records are used for subsequent hot data. Since the migration of cold data is fast enough, the incremental data is limited, allowing for in-memory aggregation and storage. During this period, IO writes are still supported, ensuring business continuity and efficient incremental data migration. Third, the switching of the master node role is handled during the final stage of incremental data migration. The system intelligently detects the stage with the lowest switching cost for the current business and processes the data accordingly, minimizing the loss of zeros in front-end services.

[0162] Based on the same inventive concept, this application also provides a data migration apparatus. Since the principle of the above-mentioned apparatus and device in solving the problem is similar to that of a data migration method, the implementation of the above-mentioned apparatus can refer to the implementation of the method, and repeated details will not be elaborated further. This apparatus can be applied to electronic devices. This application does not limit the type of electronic device; it can be any suitable type of device, such as terminal devices and servers, etc., which will not be elaborated further in this application. The apparatus embodiment can be implemented by software, or by hardware, or a combination of software and hardware. Taking software implementation as an example, as a logically defined apparatus, it is formed by the processor of the electronic device reading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0163] See Figure 6 The diagram shown is a structural block diagram of a data migration apparatus according to an embodiment of this application. In some embodiments, the data migration apparatus exemplified in this application includes:

[0164] The lookup unit 601 is used to respond to a data migration event, find the metadata corresponding to the prefix associated with the identifier of the fragment to be migrated in descending order of data layer, and migrate the found metadata to the target node.

[0165] The receiving unit 602 is used to, during the data migration process, if it receives a metadata transaction that needs to be written into the shard to be migrated corresponding to the shard identifier, write the received metadata transaction into a designated storage space in the MemTable of the database; the metadata transaction carries metadata; the designated storage space is only used to store the metadata transaction corresponding to the shard to be migrated.

[0166] Migration unit 603 is used to migrate each metadata transaction recorded in the specified storage space to the target node after the metadata migration of the shard to be migrated out in the data layer is completed.

[0167] In one implementation, the database includes multiple data layers from the first layer to the nth layer; n is a positive integer;

[0168] The lookup unit 601 is used for:

[0169] Create snapshots of the data in each data layer of the database and lock the metadata associated with the snapshots to prevent the metadata associated with the snapshots from being deleted;

[0170] Based on the snapshot, traverse each data layer in descending order of data layer and query the metadata corresponding to the prefix associated with the shard identifier to be migrated out;

[0171] When reading metadata from the first data layer, deduplication is performed on the metadata read from the first data layer;

[0172] The filtered metadata is migrated to the target node via remote direct memory access.

[0173] In one embodiment, the receiving unit 602 is used for:

[0174] Request an initial memory aggregation cache;

[0175] During the data migration process, the following steps are executed repeatedly:

[0176] When a metadata request for a shard to be migrated out is received, the corresponding metadata transaction is obtained based on the metadata request;

[0177] Write metadata transactions to the write-ahead log;

[0178] After the write-ahead log is successfully written, determine the current state of the memory aggregation cache:

[0179] If the number of metadata transactions in the current memory aggregation cache does not reach the set threshold and the total data volume does not reach the set size threshold, then the metadata transactions will be copied to the current memory aggregation cache.

[0180] If the number of metadata transactions in the memory aggregation cache reaches a set threshold or the total data volume reaches a set size threshold, a new memory aggregation cache will be requested, and the metadata transactions will be copied to the new memory aggregation cache. The specified storage space includes the initial memory aggregation cache and the new memory aggregation cache.

[0181] In one embodiment, the migration unit 603 is further configured to:

[0182] When the amount of data in the metadata transaction to be migrated in the specified storage space is lower than a preset threshold, the metadata request for the fragment to be migrated out is suspended.

[0183] Migrate the metadata transactions to be migrated, along with the end markers, to the target node;

[0184] After receiving the end response from the target node in response to the end marker, the migration end indication file containing the identifier of the shard to be migrated out is stored in the database; the migration end indication file is used to indicate that the migration of the shard to be migrated out is complete.

[0185] Remove the metadata corresponding to the prefix associated with the shard identifier to be migrated from the database;

[0186] Send a mapping update request to the control module of the distributed system so that the control module updates the mapping between shards and nodes, thereby updating the mapping between the shard to be migrated out and the node to be migrated out to the mapping between the shard to be migrated out and the target node.

[0187] In one embodiment, the migration unit 603 is further configured to:

[0188] Use any one of the following:

[0189] If it is determined that the node to be migrated out is abnormal and the metadata transaction migration is not completed, the shard data migration operation will be re-executed for the shard to be migrated out.

[0190] If it is determined that the outgoing node is abnormal and the end marker was sent successfully, a corresponding relationship update request is sent to the control module of the distributed system.

[0191] If an anomaly is detected in the target node during the migration of metadata in the data layer, the shard data migration operation will be re-executed for the shard to be migrated out.

[0192] If an anomaly is detected in the target node during the metadata transaction migration, the migration message is resent.

[0193] The data migration method in this embodiment includes, in response to a data migration event, finding the metadata corresponding to the prefix associated with the shard identifier to be migrated out, in descending order of the data layer, and migrating the found metadata to the target node; during the data migration process, if a metadata transaction that needs to be written to the shard corresponding to the shard identifier to be migrated out is received, the received metadata transaction is written to a designated storage space in the MemTable of the database; the metadata transaction carries metadata; the designated storage space is only used to store the metadata transaction corresponding to the shard to be migrated out; after the metadata migration of the shard to be migrated out in the data layer is completed, each metadata transaction recorded in the designated storage space is migrated to the target node. In this way, batch queries can be performed using the shard identifier to be migrated out, and newly received metadata transactions can be aggregated and cached, improving data migration efficiency and reducing resource consumption.

[0194] In this embodiment of the application, an electronic device is also provided, including:

[0195] Processor; and

[0196] The memory stores computer instructions that cause the processor to execute the methods of any of the above-described embodiments.

[0197] In this application embodiment, a computer-readable storage medium is provided, storing computer instructions for causing a computer to perform the methods of any of the above embodiments.

[0198] This application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device performs the method of any of the above-described embodiments.

[0199] Figure 7 A schematic diagram of the structure of an electronic device 7000 is shown. (See also...) Figure 7 As shown, the electronic device 7000 includes a processor 7010 and a memory 7020, and optionally may also include a power supply 7030, a display unit 7040, and an input unit 7050.

[0200] The processor 7010 is the control center of the electronic device 7000. It connects various components through various interfaces and lines, and performs various functions of the electronic device 7000 by running or executing software programs and / or data stored in the memory 7020, thereby performing overall monitoring of the electronic device 7000.

[0201] In this embodiment, when the processor 7010 calls the computer program stored in the memory 7020, it executes the steps in the above embodiments.

[0202] Optionally, the processor 7010 may include one or more processing units; preferably, the processor 7010 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 7010.

[0203] The memory 7020 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the electronic device 7000, etc. In addition, the memory 7020 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device, etc.

[0204] Electronic device 7000 also includes a power supply 7030 (such as a battery) that supplies power to various components. The power supply can be logically connected to processor 7010 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0205] The display unit 7040 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 7000. In this embodiment, it is mainly used to display the display interface of various applications in the electronic device 7000, as well as text, images, and other objects displayed on the display interface. The display unit 7040 may include a display panel 7041. The display panel 7041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0206] The input unit 7050 can be used to receive information such as numbers or characters input by the user. The input unit 7050 may include a touch panel 7051 and other input devices 7052. The touch panel 7051, also known as a touch screen, can collect touch operations on or near the touch panel 7051 (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 7051).

[0207] Specifically, the touch panel 7051 can detect user touch operations and the signals generated by these operations, converting them into touch point coordinates and sending them to the processor 7010. It can also receive and execute commands from the processor 7010. Furthermore, the touch panel 7051 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. Other input devices 7052 can include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0208] Of course, the touch panel 7051 can cover the display panel 7041. When the touch panel 7051 detects a touch operation on or near it, it transmits the information to the processor 7010 to determine the type of touch event. Subsequently, the processor 7010 provides corresponding visual output on the display panel 7041 based on the type of touch event. Although in Figure 7 In this embodiment, the touch panel 7051 and the display panel 7041 are two separate components to realize the input and output functions of the electronic device 7000. However, in some embodiments, the touch panel 7051 and the display panel 7041 can be integrated to realize the input and output functions of the electronic device 7000.

[0209] The electronic device 7000 may also include one or more sensors, such as a pressure sensor, a gravity acceleration sensor, a proximity light sensor, etc. Of course, depending on the specific application, the electronic device 7000 may also include other components such as a camera. Since these components are not the focus of this application's embodiments, therefore... Figure 7It is not shown in the text and will not be described in detail here.

[0210] Those skilled in the art will understand that Figure 7 This is merely an example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than shown, or a combination of certain components, or different components.

[0211] For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

Claims

1. A method of data migration, characterized by, The application is applied to a migrated node of a distributed system, the migrated node is deployed with a database, the database has a plurality of data layers in a hierarchical order, the data layers are used to store metadata contained in at least one shard in a key-value pair manner, the metadata in each data layer is arranged in a sequence of prefixes of corresponding keys, so that a plurality of metadata corresponding to the same prefix are aggregated together; the prefix corresponding to any metadata is associated with a shard identifier corresponding to the metadata; the method comprises: In response to a data migration event, metadata corresponding to a prefix associated with a to-be-migrated shard identifier is found in a sequence of data layers from high to low, and the found metadata is migrated to a target node; During data migration, if a metadata transaction that needs to be written into a to-be-migrated shard corresponding to the to-be-migrated shard identifier is received, the received metadata transaction is written into a specified storage space in a MemTable of the database; the metadata transaction carries metadata; and the specified storage space is used only for storing metadata transactions corresponding to the to-be-migrated shard; After the metadata migration of the to-be-migrated shard in the data layer is completed, each metadata transaction recorded in the specified storage space is migrated to the target node.

2. The method of claim 1, wherein, The database comprises a plurality of data layers from a first layer to an nth layer; n is a positive integer; The method of finding metadata corresponding to a prefix associated with a to-be-migrated shard identifier in a sequence of data layers from high to low and migrating the found metadata to a target node comprises: A snapshot of data in each current data layer in the database is created, and metadata associated with the snapshot is locked to avoid the metadata associated with the snapshot being deleted; Based on the snapshot, each data layer is traversed in a sequence of data layers from high to low, and metadata corresponding to a prefix associated with the to-be-migrated shard identifier is queried out; When reading metadata from the first data layer, the read metadata in the first data layer is de-duplicated; The filtered metadata is migrated to the target node through remote direct memory access.

3. The method of claim 1, wherein, The method of writing a received metadata transaction into a specified storage space in a MemTable of the database during data migration comprises: An initial memory aggregation buffer area is applied for; During data migration, the following steps are executed in a loop: When a metadata request for the to-be-migrated shard is received, a corresponding metadata transaction is obtained based on the metadata request; The metadata transaction is written into a prewrite log; After the prewrite log is written successfully, the state of the current memory aggregation buffer area is determined: If the number of metadata transactions in the current memory aggregation buffer area does not reach a set number threshold and the total data volume does not reach a set size threshold, the metadata transaction is copied into the current memory aggregation buffer area. If the number of metadata transactions in the memory aggregation buffer reaches a set number threshold or the total data amount reaches a set size threshold, a new memory aggregation buffer is applied for, and the metadata transactions are copied into the new memory aggregation buffer; the specified storage space includes the initial memory aggregation buffer and the new memory aggregation buffer.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: suspending the metadata request for the to-be-migrated shard when the data amount of the metadata transactions to be migrated in the specified storage space is lower than a preset threshold; migrating the to-be-migrated metadata transactions and an end marker to the target node; after receiving an end response returned by the target node for the end marker, storing a migration end indication file containing the to-be-migrated shard identifier to the database; the migration end indication file is used to indicate that the migration of the to-be-migrated shard is completed; deleting the metadata corresponding to the prefix associated with the to-be-migrated shard identifier from the database; sending a correspondence updating request to a control module of the distributed system, so that the control module updates the correspondence between shards and nodes, and updates the correspondence between the to-be-migrated shard and the migrated node to the correspondence between the to-be-migrated shard and the target node.

5. The method of claim 4, wherein, The method further includes: any of the following: if it is determined that the migrated node is abnormal and the metadata transaction migration is not completed, re-executing the shard data migration operation for the to-be-migrated shard; if it is determined that the migrated node is abnormal and the end marker sending is successful, sending the correspondence updating request to the control module of the distributed system; if the target node is detected to be abnormal in the process of migrating the metadata of the data layer, re-executing the shard data migration operation for the to-be-migrated shard; if the target node is detected to be abnormal in the metadata transaction migration process, re-sending the migration message for migrating the metadata transaction.

6. An apparatus for data migration, the apparatus comprising: The migrated node applied to the distributed system, the migrated node is deployed with a database, the database has a plurality of data layers in a hierarchical increasing manner, the data layer is used to store the metadata contained in at least one shard in a key-value pair manner, the metadata in each data layer is arranged in the order of the prefixes of the corresponding keys, so that a plurality of metadata corresponding to the same prefix are aggregated together; the prefix corresponding to any metadata is associated with the shard identifier corresponding to the metadata; the device includes: a finding unit, configured to find the metadata corresponding to the prefix associated with the to-be-migrated shard identifier in the order of the data layers from high to low in response to a data migration event, and migrate the found metadata to a target node; a receiving unit, configured to, in the data migration process, if a metadata transaction needing to be written into a to-be-migrated shard corresponding to the to-be-migrated shard identifier is received, write the received metadata transaction into a specified storage space in a MemTable of the database; the metadata transaction carries metadata; the specified storage space is only used to store the metadata transactions corresponding to the to-be-migrated shard; The migration unit is configured to migrate each metadata transaction recorded in the specified storage space to the target node after metadata migration of the to-be-migrated shard in the data layer is completed.

7. The apparatus of claim 6, wherein, The database comprises a plurality of data layers from a first layer to an nth layer; n is a positive integer; The searching unit is configured to: create a snapshot of data in each current data layer in the database, and lock metadata associated with the snapshot to avoid deletion of the metadata associated with the snapshot; based on the snapshot, search metadata corresponding to a prefix associated with the to-be-migrated shard identifier in each data layer in descending order of the data layers; de-duplicate the metadata read from the first data layer when reading the metadata from the first data layer; migrate the filtered metadata to the target node through remote direct memory access.

8. The apparatus of claim 6, wherein, The receiving unit is configured to: apply an initial memory aggregation buffer area; in the data migration process, the following steps are executed in a loop: when a metadata request for the to-be-migrated shard is received, obtain a corresponding metadata transaction based on the metadata request; write the metadata transaction to a prewrite log; after successful prewrite log writing, determine the state of the current memory aggregation buffer area: if the number of metadata transactions in the current memory aggregation buffer area does not reach a set number threshold and the total data volume does not reach a set size threshold, copy the metadata transaction to the current memory aggregation buffer area; if the number of metadata transactions in the memory aggregation buffer area reaches the set number threshold or the total data volume reaches the set size threshold, apply a new memory aggregation buffer area and copy the metadata transaction to the new memory aggregation buffer area; the specified storage space comprises the initial memory aggregation buffer area and the new memory aggregation buffer area.

9. The device according to any of claims 6-8, characterized in that The migration unit is further configured to: when the data volume of the to-be-migrated metadata transaction in the specified storage space is lower than a preset threshold, suspend the metadata request for the to-be-migrated shard; migrate the to-be-migrated metadata transaction and an end marker to the target node; after receiving an end response returned by the target node for the end marker, store a migration end indication file containing the to-be-migrated shard identifier to the database; the migration end indication file is used to indicate that the to-be-migrated shard migration is completed; delete metadata corresponding to the prefix associated with the to-be-migrated shard identifier from the database; send a corresponding relationship update request to a control module of the distributed system to enable the control module to update the corresponding relationship between shards and nodes to update the corresponding relationship between the to-be-migrated shard and the migrated node to the corresponding relationship between the to-be-migrated shard and the target node.

10. The apparatus of claim 9, wherein, The migration unit is further configured to: use any of the following: if it is determined that the migrated node is abnormal and the metadata transaction migration is not completed, re-execute the shard data migration operation for the to-be-migrated shard; if it is determined that the migrated node is abnormal and the end marker is sent successfully, send the corresponding relationship update request to the control module of the distributed system; If the target node is detected to be abnormal during migration of metadata of the data layer, a shard data migration operation is re-executed for the shard to be migrated out; If the target node is detected to be abnormal during the metadata transaction migration, the migration message is retransmitted.

11. An electronic device, comprising: Comprise: a processor; and a memory storing computer instructions for causing the processor to execute the method of any one of claims 1 to 5.

Images