Data processing methods and equipment

Abstract

This disclosure provides a data processing method and apparatus. The method, during the migration of a metadata table from a source cluster to a target cluster, detects a write operation on the metadata table, performs a checkpoint operation on the metadata table to back up the existing metadata information under the checkpoint operation, migrates the existing metadata information to the target cluster, and then replays the existing metadata information based on the corresponding log files after the write operation to obtain the target metadata information after the write operation. Finally, after the metadata table migration is complete, the corresponding data file is migrated to the target cluster. This technical solution avoids data location confusion during data migration and does not affect normal read / write operations for users.

Description

Technical Field

[0001] This disclosure relates to the field of computer and network communication technology, and in particular to a data processing method and apparatus. Background Technology

[0002] In big data scenarios, distributed file systems often serve as the foundation for data storage. As data volumes continue to increase, single clusters gradually become bottlenecks for storage and access, and "partitioning," a core issue in distributed systems, has become a major pain point. In the industry, data partitioning typically impacts the normal read and write operations of upper-layer business processes.

[0003] In existing technologies, when a user performs a write operation, a data write priority is provided. Based on this data write priority, the incremental data information corresponding to the write operation is written to the new cluster first. If the write fails, it is written to the old cluster. Other existing metadata information can be migrated offline.

[0004] However, in the above method, when users need to perform read and write operations on incremental data, they need to first try reading and writing in the new cluster, and if unsuccessful, try reading and writing again in the old cluster, which affects the user's read and write efficiency to some extent. Summary of the Invention

[0005] This disclosure provides a data processing method and apparatus to avoid data location confusion during data migration, which could affect normal user read and write operations.

[0006] In a first aspect, embodiments of this disclosure provide a data processing method, including:

[0007] During the process of migrating the metadata table from the source cluster to the target cluster, if a write operation is detected on the metadata table, a checkpoint operation is performed on the metadata table to back up the current existing metadata information in the metadata table under the checkpoint operation.

[0008] After migrating the current existing metadata information to the target cluster, the current existing metadata information is replayed based on the corresponding log file after the write operation is completed, so as to obtain the target metadata information after the write operation is performed on the current existing metadata information;

[0009] After the metadata table migration is completed, the data files corresponding to the metadata table are migrated to the target cluster.

[0010] In a second aspect, embodiments of this disclosure provide a data processing apparatus, including:

[0011] The backup unit is used to detect the current write operation on the metadata table during the process of migrating the metadata table from the source cluster to the target cluster, and to perform a checkpoint operation on the metadata table to back up the current existing metadata information in the metadata table under the checkpoint operation.

[0012] The replay unit is used to replay the current existing metadata information based on the log file after the write operation is completed after migrating the current existing metadata information to the target cluster, so as to obtain the target metadata information after the write operation is performed on the current existing metadata information.

[0013] The migration unit is used to migrate the data files corresponding to the metadata table to the target cluster after the metadata table migration is completed.

[0014] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0015] The memory stores computer-executed instructions;

[0016] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the data processing method described in the first aspect of the possible design above.

[0017] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data processing method described in the possible design of the first aspect above.

[0018] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the data processing method described in the possible design of the first aspect above.

[0019] The data processing method and device provided in this embodiment detects write operations on the metadata table during the migration of the metadata table from the source cluster to the target cluster. A checkpoint operation is then performed on the metadata table to back up the current existing metadata information under the checkpoint operation. After migrating the current existing metadata information to the target cluster, a replay operation is performed on the current existing metadata information based on the corresponding log files after the write operation to obtain the target metadata information after the write operation. After the metadata table migration is complete, the data files corresponding to the metadata table are migrated to the target cluster. This technical solution, by determining the current existing metadata information through checkpoint operations, recording the corresponding operation when a write operation occurs, and then performing the corresponding write operation on the current existing metadata information in the target cluster, avoids confusion during data migration and does not affect normal read and write operations for users. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a diagram illustrating user access within an existing distributed system.

[0022] Figure 2 A schematic diagram of the existing dynamic subtree splitting;

[0023] Figure 3 Flowchart of the data processing method provided in the embodiments of this disclosure Figure 1 ;

[0024] Figure 4 Flowchart of the data processing method provided in the embodiments of this disclosure Figure 2 ;

[0025] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] In big data scenarios, distributed file systems often serve as the foundation for data storage. As data volumes continue to increase, single clusters gradually become bottlenecks for storage and access, and "partitioning," a core issue in distributed systems, has become a pain point. In the industry, data partitioning typically impacts the normal read and write operations of upper-layer applications, meaning users are "aware" of the impact. How to achieve user-insensitive subtree partitioning in a distributed file system has become a pressing problem, exemplified by the Hadoop Distributed File System (HDFS).

[0029] From the perspective of the same directory tree namespace, the data layer (DN) is shared among multiple clusters; the metadata layer (NN) is independent of each other and is connected through the routing layer (Proxy).

[0030] Distributed file systems typically divide a complete directory tree into many subtrees, with each subtree corresponding to a partition (NN cluster). As the amount of data continues to increase, a single cluster gradually becomes a bottleneck for storage and access, requiring data re-partitioning.

[0031] The main challenges of dynamic subtree splitting (i.e., repartitioning data):

[0032] 1. How to ensure that user read and write operations are not affected during the splitting process.

[0033] 2. After the split is completed, how to ensure that the data is correct and that the user's semantics remain unchanged compared to before the split?

[0034] The common approach in existing technologies is as follows: the root paths of each NameNode cluster are mounted to the routing table, and the proxy periodically pulls the routing table and forwards user traffic according to the routing table. In the routing table, typically only one backend NameNode cluster is mounted on one path, and one backend NameNode cluster is mounted on only one path.

[0035] Figure 1 A diagram illustrating user access in an existing distributed system, such as... Figure 1As shown, it includes: a metadata routing layer, a routing table (stored in a mount table, for example, the path of / a is nn0; the path of / a / b is nn1), a distributed cluster nn0 (path: / a / c), and a distributed cluster nn1 (path: / a / b / c).

[0036] For example, if a user needs to access a file at a certain path, based on the longest match principle, such as accessing / a / b, the access will be forwarded to nn1.

[0037] Regarding the problem of dynamic subtree splitting Figure 2 This is a schematic diagram of the existing dynamic subtree splitting.

[0038] Combination Figure 2 The general idea is as follows:

[0039] 1. The routing table supports multiple mounts on a single path (for example, the path of / a / b is nn0, and the paths of / a / b / c are nn0 and nn1). At the same time, backend NN clusters mounted on the same path have different priorities (nn1 is greater than nn0).

[0040] 2. User traffic is routed to a higher-priority backend NN cluster (migration destination cluster nn1). If routing fails, it falls back to a lower-priority backend NN cluster (migration origin cluster nn0). This probing mechanism allows incremental data to be written to the current migration cluster while existing metadata is read normally.

[0041] 3. Offline migrate the current existing metadata information to the target NN cluster. After the migration is completed, restore the routing table to single mount mode.

[0042] The technical problems existing in this prior art are:

[0043] 1. The proxy periodically pulls the routing table, which cannot guarantee strong consistency between routing tables of different proxy instances. This leads to inconsistencies in incremental writes for different users (clients), affecting user read and write operations to some extent. Specifically, if the first attempt to read data from nn1 fails, an error will be returned to the user, or the second attempt will try to read from nn0, resulting in relatively low read and write efficiency.

[0044] 2. If the current existing metadata and incremental data of the path to be split may be managed by two backend NameNodes at the same time, then multi-path operations involving two clusters will not be possible.

[0045] Based on the existing technical problems, the inventors propose the following: If, during the migration of the metadata table, a user is expected to access the metadata table, a checkpoint operation can be performed on the metadata table in the source cluster to obtain the current existing metadata information. At this time, the user's access request is executed normally, and the corresponding log file can be obtained after execution. After the current existing metadata information is injected into the target cluster, incremental metadata information can be written to the current existing metadata information using the log file. Then, the call service of the target cluster is started, and services can be provided to users during subsequent accesses. Since the metadata table is migrated, only the corresponding data of the metadata table needs to be migrated.

[0046] The following are specific method embodiments provided in this embodiment. The execution subject of this method embodiment can be an electronic device, specifically a control server in a distributed cluster that manages various servers.

[0047] Figure 3 Flowchart of the data processing method provided in the embodiments of this disclosure Figure 1 .like Figure 3 As shown, the data processing method includes:

[0048] Step 31: During the process of migrating the metadata table from the source cluster to the target cluster, a write operation on the metadata table is detected. A checkpoint operation is then performed on the metadata table to back up the existing metadata information in the metadata table under the checkpoint operation.

[0049] In this step, as the amount of data in the source cluster continues to increase, the source cluster gradually becomes a bottleneck for storage and access, requiring data re-partitioning, which means that the specified data needs to be migrated to the target cluster. For example, the specified data can be the metadata table and the corresponding data files of the metadata table, which can be migrated to the target cluster. During the migration process, users may make access requests to the metadata table, which may include write operations such as modifying, deleting, and editing the metadata table.

[0050] At this point, a checkpoint operation is performed on the metadata table in the source cluster, and the metadata table of the subtree to be split in the checkpoint is read, that is, the current existing metadata information in the metadata table.

[0051] Optionally, in this implementation, a whitelist can be set, which records which metadata tables do not need to be migrated. During migration, metadata tables in the whitelist can be filtered out, and only metadata tables not on the whitelist can be migrated.

[0052] Optionally, a suitable cluster can be selected as the target cluster based on the cluster's capacity, resource utilization, and the tidal relationship between the metadata table and the corresponding data files.

[0053] Step 32: After migrating the current existing metadata information to the target cluster, replay the current existing metadata information based on the corresponding log files after the write operation is completed, so as to obtain the target metadata information after the write operation on the current existing metadata information.

[0054] The log file is a file generated based on write operations, which can be an Editlog;

[0055] In this step, since the write operation performed above will make corresponding changes to the metadata table in the source cluster, the changes are recorded in the log file. After the current existing metadata information is injected into the target cluster, the log file can be used to update the current existing metadata information in the target cluster to obtain the metadata corresponding to the write operation in the source cluster (i.e., including the current existing metadata information and the incremental data corresponding to the write operation).

[0056] Optionally, during the process of injecting existing metadata information into the target cluster, users' read and write operations are not affected. Incremental metadata information is replayed through Editlog (EditLog is the binlog used for master-slave synchronization between the source cluster and the target cluster), which is equivalent to temporarily adding the target cluster to the source cluster as a special slave node.

[0057] During the implementation of this step, there is no stage where the metadata table of the directory tree of the path to be split is managed by two backend clusters at the same time. When users use it, before the migration is completed, the source cluster provides services, and after the migration is completed, the target cluster provides services.

[0058] It should be understood that the source cluster and target cluster in this embodiment are clusters in a distributed system, which may include multiple servers or related electronic devices.

[0059] Furthermore, based on the updated current metadata information and the address of the updated current metadata information in the target cluster, the routing table in the distributed system where the target cluster is located is updated to obtain the updated routing table.

[0060] In this implementation, after the incremental data information replay is completed, the metadata table migration is also completed. At this time, the routing table in the distributed system is updated using the target metadata information after the write operation on the current existing metadata information, and the address of the target metadata information in the target cluster.

[0061] Furthermore, upon receiving the next read / write request for the metadata table, the path information of the metadata information corresponding to the next read / write request is determined in the updated routing table. Then, based on the path information of the metadata information corresponding to the next read / write request, the read / write operation corresponding to the metadata information of the next read / write request is performed.

[0062] Under this implementation, based on the updated routing table, when a user performs read or write operations, the corresponding data file of the metadata table can be accurately operated according to the information recorded in the routing table. That is, after the user initiates a read or write request, the content recorded in the updated routing table and the content carried in the request can be matched to determine the metadata information actually operated on by the read or write request and the data corresponding to the metadata information, and then the read and write of the metadata information and the data corresponding to the metadata information can be completed.

[0063] Step 33: After the metadata table migration is completed, migrate the data files corresponding to the metadata table to the target cluster.

[0064] In this step, since the data files may also change in response to write operations, after the metadata table migration is completed, the data files corresponding to the metadata table can also be migrated to the target cluster.

[0065] In this implementation, data files can be migrated block by block sequentially. During the metadata migration process, write operations on metadata occur, which will cause related processing of the data files. At this time, the content of the data files has been changed, so the data files can be directly migrated to the target cluster.

[0066] The data files are managed and stored by the cluster pool (DN), and there is no relationship between the data in different files.

[0067] Optionally, this step can be implemented as follows: if the source cluster where the data file corresponding to the metadata is located shares the same resource pool as the target cluster, the reference information of the data file corresponding to the metadata table is migrated to the target cluster; if the source cluster where the data file corresponding to the metadata is located does not share the same resource pool as the target cluster, the data file corresponding to the metadata table is migrated to the target cluster.

[0068] Under this implementation, the source cluster and the target cluster can choose to migrate data files using either a shallow copy (hardlink, sharing the same DN pool) or a deep copy (copy, not sharing the same DN pool) depending on whether they share the same DN pool.

[0069] For example, in a shallow copy, the reference information of the data file can be a shortcut to the data file, and the shortcut to the data file is stored in the target cluster; in a deep copy, the data file is directly stored in the target cluster.

[0070] Once all the data in a file within a data cluster has been migrated, that file is considered migrated and is then managed by the target cluster.

[0071] During the entire splitting (i.e. migration) process of the data files, there is a stage where the data of the files is not simultaneously managed by two backend clusters (i.e., the source cluster and the target cluster).

[0072] Furthermore, based on the unique identifier of the data file recorded in the metadata table, the path information of the data file in the target cluster is determined, and the path information of the data file is updated in the metadata table.

[0073] In this implementation, it is necessary to associate the data files and metadata tables (target equivalent data information) in the target cluster, that is, the storage path of the corresponding data files can be known from the metadata table.

[0074] The metadata table carries a unique identifier for the data file. The electronic device identifies the data file that matches the unique identifier in the target cluster, and then, based on the path information of the data file, returns to the metadata table to record the path information of the corresponding data file, that is, adds the path information of the data file to the target equivalent data information.

[0075] Furthermore, after the metadata tables and corresponding data files in the source cluster are migrated to the target cluster, remote procedure call services for the metadata tables in the source cluster are stopped, while remote procedure call services for the metadata tables in the target cluster are provided.

[0076] After the metadata table and corresponding data files are successfully migrated to the target cluster, when the user accesses the corresponding metadata table, the target cluster provides remote procedure call services for the metadata table, that is, the remote procedure call services for the metadata table in the source cluster are turned off.

[0077] In addition, in some implementations, metadata tables and corresponding data files that have been migrated from the source cluster can be cleared, reducing the space resource usage of the source cluster.

[0078] The data processing method provided in this disclosure, during the migration of a metadata table from the source cluster to the target cluster, detects a write operation on the metadata table, performs a checkpoint operation on the metadata table to back up the current existing metadata information under the checkpoint operation, migrates the current existing metadata information to the target cluster, and then replays the current existing metadata information based on the corresponding log file after the write operation is completed to obtain the target metadata information after the write operation on the current existing metadata information. After the metadata table migration is completed, the data file corresponding to the metadata table is migrated to the target cluster. This technical solution, by determining the current existing metadata information through checkpoint operation, recording the corresponding operation when a write operation occurs, and then performing the corresponding write operation on the current existing metadata information in the target cluster, can avoid confusion during data migration and does not affect normal read and write operations for users.

[0079] Based on the above embodiments, Figure 4 Flowchart of the data processing method provided in the embodiments of this disclosure Figure 2 .like Figure 4 As shown, this data processing method is explained from the perspectives of DN (data node, i.e., data file), source cluster scnn, target cluster dstnn, and ZK (ZooKeeper, an open-source distributed coordination service framework, i.e., the mount table where the routing table is located):

[0080] I. Preparation Phase (Preparation before migrating metadata tables and corresponding data files):

[0081] 1. Send the split dancenn command to electronic devices (e.g., Alfred in a distributed system) to set a whitelist for the directories to be migrated;

[0082] 2. The electronic device creates and starts the split dancenn task, and periodically reports the current status of the task to the operation and maintenance platform;

[0083] 3. The electronic device sends a checkpoint command to the dancenn active node (i.e., the "source cluster") to which the split path belongs (this implementation is used to back up the current metadata information in the metadata table under the source cluster later);

[0084] 4. The source cluster confirms whether the split task (i.e., the task of splitting the data table from the source cluster to the target cluster) is executable (whether the data table exists in the whitelist);

[0085] a. Executable: Performs preliminary work for split tasks, and returns an updated task context to the electronic device upon completion;

[0086] b. Not executable: The exception reason is put into the task context and returned to the electronic device, and the process is terminated.

[0087] II. Metadata Migration and Namespace Management Handover (The migration process of the metadata table includes the migration of the metadata table and the updating of the routing table. Under this implementation, the metadata table and data files in the source cluster continue to provide read and write services to users):

[0088] 1. The source cluster executes the checkpoint task and generates a split begin log. Upon completion, it returns the checkpoint path, split begintxid, and bk ip to the electronic devices, along with other task contexts.

[0089] 2. Distribute the split task to the target cluster's dancenn active node (i.e., the "target cluster"). The target cluster performs necessary checks, such as whether metadata conflicts will occur.

[0090] 3. After the check passes, the target cluster loads the specified path from the RocksDB (i.e., the embedded key-value storage engine) checkpoint. During this period, newly added paths are not visible to the user.

[0091] 4. After loading, the split path is readable but not writable on the target cluster. The following actions occur simultaneously:

[0092] a. The target cluster starts consuming the incremental edit log (i.e., the log file corresponding to write operations) of the relevant paths after the split begin txid (starting from the given id address) from the src bk ip in the task context of the connection task, processing it according to normal write logic while retaining the inodes of the source cluster. The target cluster maintains a pending migration block table, which is initialized through a checkpointfile. Block info and inode info related to block changes in the incremental edit log are merged into this table. This structure is persisted in the target cluster's RocksDB-specific CF.

[0093] b. When electronic devices start consuming the pending migration block table in the target cluster, they begin the block migration (hardlink). During this process, any non-existent blocks are ignored and recorded as ignored blocks for subsequent data verification.

[0094] 5. Electronic devices periodically check the gap in the target cluster's incremental edit log consumption. When the gap is less than a certain value, the following tasks are performed simultaneously:

[0095] a. Notify the source cluster to stop reading and writing services to the namespace of the split path, write to the namespacesplit finish log (i.e., the namespace split completion log), and update the relevant txid to the task context.

[0096] i. The source cluster receives a forward exception (an exception propagation mechanism) from the relevant path RPC (i.e., remote procedure call), which contains information about the target cluster. After receiving it, nnproxy forwards the RPC to the target cluster for processing.

[0097] ii.block In cases where the target cluster does not have fully available metadata for files, the metadata of the files is maintained by two nns together. The target cluster is responsible for maintaining the directory tree metadata, and the source cluster is responsible for maintaining the metadata related to the data blocks.

[0098] iii. All metadata RPCs are first sent to the target cluster, which then decides whether to forward block-related RPCs to the source cluster for processing. Forwarded block RPCs (block-level calls) will have a special identifier, and the source cluster will only process block-related RPCs with this special identifier.

[0099] b. The electronic device changes the mountable in zk, mounting dst to the root node of the split path (i.e., updating the routing table).

[0100] When c.nnproxy receives a forward exception, it checks whether its own mountable has expired and actively refreshes the mountable (there is a backoff mechanism, and forward exceptions generated by the target cluster are not refreshed).

[0101] 6. The target cluster continuously consumes the incremental edit log. During this period, it does not provide metadata services for the split path until it encounters the namespacesplit finish log. At this point, the handover of namespace metadata management for the split path is complete, and the task cannot be rolled back or canceled.

[0102] III. Block Transition Period: Block migration and handover of block-related metadata during the relocation of electronic devices (i.e., migration of file data corresponding to the metadata table).

[0103] 1. Block relocation complete:

[0104] a. The source cluster writes the split finish log, and the target cluster consumes this log and then writes it back into the split finish log to inform the standby nn;

[0105] b. Notify the operations and maintenance platform that the metadata migration is complete and the project is entering the final stage.

[0106] 2. Block migration in progress: Notify the src / target cluster of the information of blocks that have not been migrated and enter the "block transition period".

[0107] 3. Block migration timeout: Notify the operations and maintenance platform of the specific anomaly in the task and allow for manual intervention.

[0108] IV. Final Stage: (Processing after the migration of metadata tables and data files is completed);

[0109] 1. Move the metadata of the source cluster's migrated directory to a specific backup path, which will be deleted after 2 days (time can be set). After completion, disable the forward exception (data persistence storage for task context) mechanism of the relevant path.

[0110] 2. Clean up intermediate files such as RocksDB (i.e., the embedded key-value storage engine) checkpoints, save them to a specific HDFS path, save the path to the task context, and notify the operations and maintenance platform that the task is complete after completion.

[0111] 3. Remove the quota whitelist from the directory being migrated;

[0112] 4. Persist the task context.

[0113] This disclosure provides a data processing method whose implementation principle and technical effects are similar to those involved in the above embodiments, and will not be repeated here.

[0114] Based on the above method embodiments, Figure 5 This is a schematic diagram of the structure of the data processing device provided in the embodiments of this disclosure, such as... Figure 5 As shown, the data processing device includes:

[0115] Backup unit 51 is used to perform a checkpoint operation on the metadata table when the metadata table in the source cluster is migrated to the target cluster, in order to back up the existing metadata information in the metadata table under the checkpoint operation.

[0116] The replay unit 52 is used to replay the current existing metadata information based on the corresponding log file after the write operation is completed after migrating the current existing metadata information to the target cluster, so as to obtain the target metadata information after the write operation is performed on the current existing metadata information.

[0117] Migration unit 53 is used to migrate the data files corresponding to the metadata table to the target cluster after the metadata table migration is completed.

[0118] According to one or more embodiments of this disclosure, the relocation unit 53 is specifically used for:

[0119] If the source cluster and the target cluster share the same resource pool, the reference information of the data file corresponding to the metadata table should be migrated to the target cluster.

[0120] If the source cluster and the target cluster do not share the same resource pool, the data file corresponding to the metadata table will be migrated to the target cluster.

[0121] According to one or more embodiments of this disclosure, the relocation unit 53 is further configured to:

[0122] Based on the unique identifier of the data file recorded in the metadata table, the path information of the data file in the target cluster is determined, and the path information of the data file is updated in the metadata table.

[0123] According to one or more embodiments of this disclosure, after migrating the current existing metadata information to the target cluster, the migration unit 53 further performs a replay operation on the current existing metadata information based on the corresponding log file after the write operation is completed, so as to obtain the target metadata information after the write operation on the current existing metadata information.

[0124] Based on the target metadata information and the path information of the target metadata information in the target cluster, the routing table in the distributed system where the target cluster is located is updated to obtain the updated routing table.

[0125] According to one or more embodiments of this disclosure, the processing unit is configured to:

[0126] Upon receiving the next read / write request for the metadata table, the path information of the metadata information corresponding to the next read / write request is determined in the updated routing table;

[0127] Based on the path information of the metadata information corresponding to the next read / write request, the read / write operation corresponding to the metadata information of the next read / write request is performed.

[0128] According to one or more embodiments of this disclosure, the processing unit is further configured to:

[0129] After migrating the metadata tables and corresponding data files in the source cluster to the target cluster, stop the remote procedure call service for the metadata tables in the source cluster, and provide the remote procedure call service for the metadata tables in the target cluster.

[0130] The data processing device provided in this embodiment has a similar technical solution and effect to those in the above embodiments, and will not be described again here.

[0131] To implement the above embodiments, this disclosure also provides an electronic device. Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure, with reference to... Figure 6 This electronic device can be a terminal device.

[0132] The terminal devices may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (such as in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0133] like Figure 6As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 61, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 62 or a program loaded from storage device 68 into random access memory (RAM) 63. The RAM 63 also stores various programs and data required for the operation of the electronic device. The processing unit 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0134] Typically, the following devices can be connected to I / O interface 65: input devices 66 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 67 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 68 including, for example, magnetic tapes, hard disks, etc.; and communication devices 69. Communication device 69 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0135] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 69, or installed from a storage device 68, or installed from a ROM 62. When the computer program is executed by the processing device 61, it performs the functions defined in the methods of embodiments of this disclosure.

[0136] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0137] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0138] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0139] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0142] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0143] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0144] In a first aspect, according to one or more embodiments of the present disclosure, a data processing method is provided, comprising:

[0145] During the process of migrating the metadata table from the source cluster to the target cluster, if a write operation is detected on the metadata table, a checkpoint operation is performed on the metadata table to back up the current existing metadata information in the metadata table under the checkpoint operation.

[0146] After migrating the current existing metadata information to the target cluster, the current existing metadata information is replayed based on the corresponding log file after the write operation is completed, so as to obtain the target metadata information after the write operation is performed on the current existing metadata information;

[0147] After the metadata table migration is completed, the data files corresponding to the metadata table are migrated to the target cluster.

[0148] According to one or more embodiments of this disclosure, migrating the data file corresponding to the metadata table to the target cluster includes:

[0149] If the source cluster where the data file corresponding to the metadata resides shares the same resource pool as the target cluster, the reference information of the data file corresponding to the metadata table will be migrated to the target cluster;

[0150] If the source cluster where the data file corresponding to the metadata resides does not share the same resource pool as the target cluster, the data file corresponding to the metadata table will be migrated to the target cluster.

[0151] According to one or more embodiments of this disclosure, the method further includes:

[0152] Based on the unique identifier of the data file recorded in the metadata table, the path information of the data file is determined in the target cluster, and the path information of the data file is updated in the metadata table.

[0153] According to one or more embodiments of this disclosure, after migrating the current existing metadata information to the target cluster, and after performing a replay operation on the current existing metadata information based on the corresponding log file after the write operation is completed to obtain the target metadata information after the write operation on the current existing metadata information, the method further includes:

[0154] Based on the target metadata information and the path information of the target metadata information in the target cluster, the routing table in the distributed system where the target cluster is located is updated to obtain the updated routing table.

[0155] According to one or more embodiments of this disclosure, the method is further used for:

[0156] Upon receiving the next read / write request for the metadata table, the path information of the metadata information corresponding to the next read / write request is determined in the updated routing table;

[0157] Based on the path information of the metadata information corresponding to the next read / write request, the read / write operation corresponding to the next read / write request is performed on the metadata information corresponding to the next read / write request.

[0158] According to one or more embodiments of this disclosure, the method further includes:

[0159] After the metadata table and the corresponding data file in the source cluster are migrated to the target cluster, the remote procedure call service for the metadata table in the source cluster is stopped, and the remote procedure call service for the metadata table in the target cluster is provided.

[0160] Secondly, according to one or more embodiments of this disclosure, a data processing apparatus is provided, comprising:

[0161] The backup unit is used to detect the current write operation on the metadata table during the process of migrating the metadata table from the source cluster to the target cluster, and to perform a checkpoint operation on the metadata table to back up the current existing metadata information in the metadata table under the checkpoint operation.

[0162] The replay unit is used to replay the current existing metadata information based on the log file after the write operation is completed after migrating the current existing metadata information to the target cluster, so as to obtain the target metadata information after the write operation is performed on the current existing metadata information.

[0163] The migration unit is used to migrate the data files corresponding to the metadata table to the target cluster after the metadata table migration is completed.

[0164] According to one or more embodiments of this disclosure, the relocation unit is specifically used for:

[0165] If the source cluster where the data file corresponding to the metadata resides shares the same resource pool as the target cluster, the reference information of the data file corresponding to the metadata table will be migrated to the target cluster;

[0166] If the source cluster where the data file corresponding to the metadata resides does not share the same resource pool as the target cluster, the data file corresponding to the metadata table will be migrated to the target cluster.

[0167] According to one or more embodiments of this disclosure, the relocation unit is further configured to:

[0168] Based on the unique identifier of the data file recorded in the metadata table, the path information of the data file is determined in the target cluster, and the path information of the data file is updated in the metadata table.

[0169] According to one or more embodiments of this disclosure, after migrating the current existing metadata information to the target cluster, and after performing a replay operation on the current existing metadata information based on the corresponding log file after the write operation is completed to obtain the target metadata information after the write operation on the current existing metadata information, the migration unit is further configured to:

[0170] Based on the target metadata information and the path information of the target metadata information in the target cluster, the routing table in the distributed system where the target cluster is located is updated to obtain the updated routing table.

[0171] According to one or more embodiments of this disclosure, the processing unit is configured to:

[0172] Upon receiving the next read / write request for the metadata table, the path information of the metadata information corresponding to the next read / write request is determined in the updated routing table;

[0173] Based on the path information of the metadata information corresponding to the next read / write request, the read / write operation corresponding to the next read / write request is performed on the metadata information corresponding to the next read / write request.

[0174] According to one or more embodiments of this disclosure, the processing unit is configured to:

[0175] After the metadata table and the corresponding data file in the source cluster are migrated to the target cluster, the remote procedure call service for the metadata table in the source cluster is stopped, and the remote procedure call service for the metadata table in the target cluster is provided.

[0176] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0177] The memory stores computer-executed instructions;

[0178] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the data processing method as described in the first aspect and various possible designs of the first aspect.

[0179] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the data processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0180] Fifthly, according to one or more embodiments of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the data processing method described in the first aspect and various possible designs of the first aspect.

[0181] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0182] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0183] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A data processing method, characterized in that, include: During the process of migrating the metadata table from the source cluster to the target cluster, a write operation on the metadata table is detected, the write operation is recorded, and a checkpoint operation is performed on the metadata table to back up the current existing metadata information in the metadata table under the checkpoint operation. After migrating the current existing metadata information to the target cluster, a replay operation is performed on the current existing metadata information based on the corresponding log file after the write operation is completed, so as to obtain the target metadata information after the write operation is performed on the current existing metadata information. According to the target metadata information and the path information of the target metadata information in the target cluster, the routing table in the distributed system where the target cluster is located is updated to obtain the updated routing table. The updated routing table is used for the next read and write request. The replay operation includes: after recording the write operation, performing the write operation on the current existing metadata information in the target cluster. After the metadata table migration is completed, the data files corresponding to the metadata table are migrated to the target cluster.

2. The method according to claim 1, characterized in that, The step of migrating the data file corresponding to the metadata table to the target cluster includes: If the source cluster where the data file corresponding to the metadata resides shares the same resource pool as the target cluster, the reference information of the data file corresponding to the metadata table will be migrated to the target cluster; If the source cluster where the data file corresponding to the metadata resides does not share the same resource pool as the target cluster, the data file corresponding to the metadata table will be migrated to the target cluster.

3. The method according to claim 1, characterized in that, The method further includes: Based on the unique identifier of the data file recorded in the metadata table, the path information of the data file is determined in the target cluster, and the path information of the data file is updated in the metadata table.

4. The method according to claim 1, characterized in that, The method further includes: Upon receiving the next read / write request for the metadata table, the path information of the metadata information corresponding to the next read / write request is determined in the updated routing table; Based on the path information of the metadata information corresponding to the next read / write request, the read / write operation corresponding to the next read / write request is performed on the metadata information corresponding to the next read / write request.

5. The method according to any one of claims 1-3, characterized in that, The method further includes: After the metadata table and the corresponding data file in the source cluster are migrated to the target cluster, the remote procedure call service for the metadata table in the source cluster is stopped, and the remote procedure call service for the metadata table in the target cluster is provided.

6. A data processing device, characterized in that, include: The backup unit is used to detect the current write operation on the metadata table during the process of migrating the metadata table from the source cluster to the target cluster, record the write operation, and perform a checkpoint operation on the metadata table to back up the current existing metadata information in the metadata table under the checkpoint operation. The replay unit is used to replay the current existing metadata information after migrating it to the target cluster, based on the corresponding log file after the write operation is completed, to obtain the target metadata information after the write operation on the current existing metadata information. Based on the target metadata information and its path information in the target cluster, the unit updates the routing table in the distributed system where the target cluster resides to obtain an updated routing table. The updated routing table is used for the next read / write request. The replay operation includes: performing the write operation on the current existing metadata information in the target cluster after recording the write operation. The migration unit is used to migrate the data files corresponding to the metadata table to the target cluster after the metadata table migration is completed.

7. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 5.

Images