Automatic source layer pasting cross-cluster data migration method and device, equipment and medium

Abstract

The invention relates to the technical field of big data and data migration, and discloses an automatic source layer pasting cross-cluster data migration method and device, computer equipment and a storage medium. The automatic source layer cross-cluster data migration method comprises the following steps: synchronizing a running script and a scheduling task of an old cluster to a new cluster; modifying a data extraction task of the stock data in the old cluster into a distributed copying task of the stock data through a template of the derivative task, and migrating and synchronizing the stock data in the source pasting layer of the old cluster to the source pasting layer of the new cluster at one time by using a distributed copying tool; and running the scheduling task, and when the scheduling platform detects that the running state of the old cluster meets the incremental data migration condition, modifying a data extraction task of the incremental data in the old cluster into a distributed copy task of the incremental data through the template of the derivative task, and migrating and synchronizing incremental data in the old cluster source pasting layer to the new cluster source pasting layer by using a distributed copying tool.

Description

Technical Field

[0001] The present invention relates to the field of big data and data migration technology, and in particular to an automatic source layer cross-cluster data migration method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer technology and Internet technology, big data technology has emerged. Big data technology includes data storage, data analysis, data mining and other technologies. The basis of these technologies is to collect data. In the actual application process, data migration is inevitably involved, especially when the big data cluster room is maintained and relocated, or the data in the local server cluster is migrated to the cloud, there will be an urgent task of migrating massive data.

[0003] In the traditional data migration process, technicians need to edit the code scripts for data migration and migrate data from the database according to the code scripts. Since editing the corresponding code scripts during the data migration process is time-consuming, the migration efficiency of the data in the database is low. Moreover, the entire data migration process requires a lot of manual intervention, the operation steps are complicated, the process is long, and the user's business level requirements are relatively high. In addition, in the traditional data migration process, the integrity of the migrated data cannot be guaranteed, so when the cluster and the old cluster are switched, the source data of the downstream application cannot be guaranteed to be consistent, and the current business operation cannot be guaranteed. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an automated source layer cross-cluster data migration method, device, computer equipment and storage medium to solve the problems that traditional data migration methods require technicians to edit code scripts for data migration, the data migration process requires a lot of manual intervention, the data migration efficiency is low, and the integrity of the migrated data is poor and cannot ensure the consistency of the source data of downstream applications.

[0005] On the first aspect, an automated source layer cross-cluster data migration method is provided, including: synchronizing the running scripts and scheduling tasks of the old cluster to the new cluster; modifying the task of extracting the existing data in the old cluster into a distributed copy task of the existing data through the template of the derivative task, and using the distributed copy tool to migrate and synchronize the existing data in the old cluster source layer to the new cluster source layer at one time; running the scheduling task, when the scheduling platform detects that the running status of the old cluster meets the incremental data migration conditions, modifying the task of extracting the incremental data in the old cluster into a distributed copy task of the incremental data through the template of the derivative task, and using the distributed copy tool to migrate and synchronize the incremental data in the old cluster source layer to the new cluster source layer.

[0006] In a second aspect, an automatic source layer cross-cluster data migration device is provided, comprising:

[0007] The task synchronization module is used to synchronize the running scripts and scheduling tasks of the old cluster to the new cluster;

[0008] The stock data synchronization module is used to modify the extraction task of the stock data in the old cluster into a distributed copy task of the stock data through the template of the derivative task, and use the distributed copy tool to migrate and synchronize the stock data in the old cluster source layer to the new cluster source layer at one time;

[0009] The incremental data synchronization module is used to run scheduling tasks. When the scheduling platform detects that the running status of the old cluster meets the incremental data migration conditions, the incremental data extraction task in the old cluster is modified into a distributed copy task of the incremental data through the template of the derivative task, and the distributed copy tool is used to migrate and synchronize the incremental data in the old cluster source layer to the new cluster source layer.

[0010] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned automatic source layer cross-cluster data migration method when executing the computer program.

[0011] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the above-mentioned automatic source layer cross-cluster data migration method are implemented.

[0012] The above-mentioned automatic source layer cross-cluster data migration method, device, computer equipment and storage medium synchronize the running script and scheduling tasks of the old cluster to the new cluster. The new cluster and the old cluster use the same set of tasks, which can ensure that the new cluster can be switched online at any time; secondly, there is no need to edit the code script for data migration. It is only necessary to modify the extraction task in the old cluster into a distributed copy task through the template of the derivative task, and execute the distributed copy task to migrate the data in the source layer of the old cluster to the source layer of the new cluster, which saves a lot of script development time and improves the efficiency of data migration; thirdly, according to the set incremental number According to the migration logic, when the running status of the old cluster meets the incremental data migration conditions, the incremental data is automatically migrated to the new cluster source layer without human intervention. The operation is simple and error-prone, reducing the difficulty of cross-cluster data migration. Finally, the distributed copy tool is used to synchronize the data migration in the old cluster source layer to the new cluster source layer, which can not only quickly copy the data in the old cluster source layer to the new cluster source layer, but also ensure the consistency of the data in the new and old cluster source layers. When the cluster and the old cluster are switched, the source data of the downstream application can be guaranteed to be consistent, thereby ensuring that the current business is not affected. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0014] Figure 1 It is a flow chart of an automatic source layer cross-cluster data migration method in one embodiment of the present invention;

[0015] Figure 2 yes Figure 1 A schematic flow chart of a specific implementation of step S30;

[0016] Figure 3 It is a structural schematic diagram of an automatic source layer cross-cluster data migration device in one embodiment of the present invention;

[0017] Figure 4 It is a structural diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] Embodiments of the present application provide a method, device, computer equipment and storage medium for automatic source layer cross-cluster data migration. Among them, the automatic source layer cross-cluster data migration method can be applied to a distributed storage system, which includes multiple server clusters and can realize cross-cluster data migration. The present invention records the server cluster storing the data to be migrated as the old cluster and the server cluster receiving the data to be migrated as the new cluster. The present invention is described in detail below through specific embodiments. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0020] Figure 1 A flow chart of an automatic source layer cross-cluster data migration method provided by an embodiment of the present invention is shown as follows: Figure 1 As shown, the automatic source layer cross-cluster data migration method includes steps S10 to S30:

[0021] S10: Synchronize the running scripts and scheduling tasks of the old cluster to the new cluster.

[0022] Among them, data processing scripts refer to scripts used for automated data processing, mainly used for tasks such as data analysis, data extraction, and data report generation. By writing scripts, operations such as data screening, sorting, and aggregation can be achieved, thereby improving data processing efficiency and accuracy.

[0023] Scheduling tasks are tasks that schedule various tasks within the cluster. The scheduling platform ensures that tasks are automatically executed according to established plans and strategies by specifying task dependencies and execution order, thereby improving work efficiency and reducing the need for manual intervention.

[0024] Synchronize the running scripts and scheduling tasks of the old cluster to the new cluster. The new and old clusters use the same set of tasks to ensure that the new cluster can be switched online at any time.

[0025] S20: The extraction task of the stock data in the old cluster is modified into a distributed copy task of the stock data through the template of the derivative task, and the distributed copy tool is used to migrate and synchronize the stock data in the old cluster source layer to the new cluster source layer at one time.

[0026] The data migration between the new cluster and the old cluster is divided into two parts: stock data migration and incremental data migration. When the old cluster is running, there is an extraction task in the system. By executing the extraction task, the extraction tool can be used to extract data from the data source and import it into the old cluster source layer. For example, by executing the Sqoop extraction task, the old cluster can use the Sqoop tool to extract stock data from the data source and import it into the old cluster source layer.

[0027] In this step, the extraction task of the stock data in the old cluster is first modified into a distributed copy task of the stock data through the template of the derivative task. Specifically, the extraction task of the stock data in the old cluster is first parsed, and the corresponding task configuration parameters are obtained through the parsing. Then, the obtained task configuration parameters are added to the template of the derivative task to generate a distributed copy task of the stock data, that is, the extraction task of the stock data in the old cluster is modified into a distributed copy task of the stock data. After that, the distributed copy task of the stock data is executed, and the distributed copy tool is used to migrate and synchronize the stock data in the old cluster source layer to the new cluster source layer at one time.

[0028] Exemplarily, when executing step S20, the Sqoop extraction task of the stock data in the old cluster can be modified into a distributed copy Distcp task of the stock data through the template of the derivative task, and the Distcp tool can be used to migrate and synchronize the stock data in the old cluster source layer to the new cluster source layer at one time.

[0029] S30: Run the scheduling task. When the scheduling platform detects that the running status of the old cluster meets the incremental data migration conditions, the incremental data extraction task in the old cluster is modified into a distributed copy task of the incremental data through the template of the derivative task. The distributed copy tool is used to synchronize the incremental data migration in the old cluster source layer to the new cluster source layer.

[0030] like Figure 2 As shown, the step S30 includes steps S31 to S33:

[0031] S31: Establish a task of converting a number to a distributed copy function, and determine whether a flag of the task of converting a number to a distributed copy function is effective.

[0032] The extract-to-distributed copy function task is a task for modifying the extract task of incremental data in the old cluster into a distributed copy task of incremental data. When executing the extract-to-distributed copy function task, the extract task of incremental data in the old cluster can be modified into a distributed copy task of incremental data through the template of the derivative task. Exemplarily, the extract-to-distributed copy function task can be a task that can modify the Sqoop extract task of incremental data in the old cluster into a distributed copy Distcp task of incremental data.

[0033] In the specific implementation, a task of converting the number of data to distributed copy function is established in advance, and the task has a parameter list, and the parameter list defines the flags. The scheduling platform will regularly check whether these flags are set. When it is detected that these flags are set, it indicates that the flags of the task of converting the number of data to distributed copy function are effective, and the scheduling platform performs the corresponding scheduling operation; when it is detected that these flags are not set, it indicates that the flags of the task of converting the number of data to distributed copy function are not effective, and the scheduling platform does not perform the corresponding scheduling operation, and returns to continue to regularly check whether these flags are set.

[0034] S32: If the flag is valid, determine whether there is incremental data in the old cluster source layer.

[0035] When it is recognized that the flag is effective, it is further determined whether there is incremental data in the old cluster source layer. The following steps S321 and S322 can be used to determine whether there is incremental data in the old cluster source layer:

[0036] S321: query and access the old cluster path whose path modification time is today. If data is found on the old cluster path whose path modification time is today, it indicates that there is incremental data in the old cluster source layer.

[0037] S322: If no data is found on the old cluster path whose path modification time is today, the polling number is set to perform polling access on the old cluster path whose path modification time is today; if data is found on the old cluster path whose path modification time is today within the polling number, it indicates that there is incremental data in the old cluster source layer; if no data is found on the old cluster path whose path modification time is today within the polling number, it indicates that there is no incremental data in the old cluster source layer.

[0038] For step S321, when the old cluster is online, if data is stored in the old cluster source layer, the old cluster path will be modified. Therefore, by querying and accessing the modified old cluster path at a fixed point every day, it can be determined whether there is incremental data in the cluster source layer. When data is queried on the old cluster path whose path modification time is the same day, it indicates that there is incremental data in the old cluster source layer; when no data is queried on the old cluster path whose path modification time is the same day, it indicates that there is no incremental data in the old cluster source layer, and an error message "the old cluster path does not exist" is returned. Among them, querying and accessing the old cluster path whose path modification time is the same day can limit the data to be accessed to the incremental data added on the same day, ensuring that the latest data of the day can be obtained when the incremental data is migrated, and avoiding obtaining old data before the same day.

[0039] For step S322, when no data is found on the old cluster path whose path modification time is today, it will not be directly determined that there is no incremental data in the old cluster source layer. Instead, the polling number will be set to poll the old cluster path whose path modification time is today, and the access will be made once every 30 minutes. If data is found on the old cluster path whose path modification time is today within the polling number, it indicates that there is incremental data in the old cluster source layer; if data is not found on the old cluster path whose path modification time is today after the polling number exceeds the number, it indicates that there is no incremental data in the old cluster source layer.

[0040] S33: If there is incremental data, the incremental data extraction task in the old cluster is modified into a distributed copy task of the incremental data through the template of the derivative task, and the incremental data in the old cluster source layer is migrated and synchronized to the new cluster source layer using the distributed copy tool.

[0041] When it is detected that there is incremental data in the old cluster source layer, the extraction task to distributed copy function will be executed, and the extraction task of incremental data in the old cluster will be modified into the distributed copy task of incremental data through the template of the derivative task. Specifically, the extraction task of incremental data in the old cluster is first parsed, and the corresponding task configuration parameters are obtained through the parsing, and then the obtained task configuration parameters are added to the template of the derivative task to generate the distributed copy task of incremental data, that is, the extraction task of incremental data in the old cluster is modified into the distributed copy task of incremental data. After that, the distributed copy task of incremental data is executed, and the distributed copy tool is used to migrate and synchronize the incremental data in the old cluster source layer to the new cluster source layer. Among them, when the distributed copy tool is used to migrate and synchronize the incremental data in the old cluster source layer to the new cluster source layer, the data (incremental data) is pulled from the old cluster by the get method, so it does not affect the operation of the old cluster, and the time for extracting data is also controllable, which can realize fast data migration.

[0042] See again Figure 2 In some embodiments, after using the distributed copy tool to migrate and synchronize the incremental data in the old cluster source layer to the new cluster source layer, that is, after step S33, step S34 is also included.

[0043] S34: Determine whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer; if they are consistent, determine whether the data synchronization type is partition synchronization, and if so, the partition needs to be repaired.

[0044] After completing the migration and synchronization of the incremental data in the old cluster source layer to the new cluster source layer, you need to verify whether the incremental data in the new cluster and the old cluster source layer are consistent. If not, you need to resynchronize the incremental data. After consistency, determine again whether the synchronization type is partition synchronization. If so, you need to repair the partition.

[0045] Among them, you can use the disk usage viewing tool or migration platform to determine whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer:

[0046] Use the disk usage viewing tool to determine whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer: Use the disk usage viewing tool to view the disk space occupied by the incremental data in the new cluster and the old cluster respectively. If the disk space occupied by the incremental data in the new cluster and the old cluster is consistent, it indicates whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer. For example, use the du command tool to view the disk space occupied by the incremental data in the new cluster and the old cluster respectively, and then determine whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer.

[0047] Use the migration platform and other methods to determine whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer: determine whether the metadata of the new cluster library table is consistent with the metadata of the old cluster library table; determine whether the data volume of the new cluster is consistent with the data volume of the old cluster; determine whether the data records of the new cluster are consistent with the data records of the old cluster; if the metadata of the new cluster library table is consistent with the metadata of the old cluster library table, the data volume of the new cluster is consistent with the data volume of the old cluster, and the data records of the new cluster are consistent with the data records of the old cluster, then it indicates that the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer.

[0048] After detecting that the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer, determine again whether the synchronization type is partition synchronization. If so, the partition needs to be repaired. Specifically, use the "MSCK REPAIR TABLE+table name" command to repair the table partition. After the command is successfully executed, the incremental data has been migrated.

[0049] The automatic source layer cross-cluster data migration method provided in the embodiment of the present invention synchronizes the running scripts and scheduling tasks of the old cluster to the new cluster. The new cluster and the old cluster use the same set of tasks, which can ensure that the new cluster can be switched online at any time; secondly, there is no need to edit the code script for data migration. It is only necessary to modify the extraction task in the old cluster into a distributed copy task through the template of the derivative task, and execute the distributed copy task to migrate the data in the source layer of the old cluster to the source layer of the new cluster, which saves a lot of script development time and improves the efficiency of data migration; thirdly, according to the set incremental data migration logic, when the running state of the old cluster meets the incremental data migration requirement, the new cluster can be switched online at any time. When the migration conditions are met, the incremental data is automatically migrated to the new cluster source layer without the need for human intervention in the program. The operation is simple and not prone to errors, which reduces the difficulty of cross-cluster data migration. Finally, compared with the prior art that uses a sampling tool to migrate and synchronize data in the old cluster source layer to the new cluster source layer, the present invention uses a distributed copy tool to migrate and synchronize data in the old cluster source layer to the new cluster source layer, which can not only quickly copy the data in the old cluster source layer to the new cluster source layer, but also ensure the consistency of data in the new and old cluster source layers, so that when the cluster and the old cluster are switched, the source data of the downstream application can be guaranteed to be consistent, thereby ensuring that the current business is not affected.

[0050] In some embodiments, the method for automatic source layer cross-cluster data migration also includes: if it is detected that the old cluster is offline, the incremental data is extracted from the data source using a sampling tool and imported into the new cluster source layer. Specifically, after the old cluster is offline, the new cluster source layer can no longer obtain incremental data from the old cluster source layer through distributed copying. At this time, the new cluster is online and runs, and the incremental data is extracted from the data source using a sampling tool and imported into the new cluster source layer. Exemplarily, the new cluster can extract incremental data from the data source and import it into the new cluster source layer using the sqoop import sampling tool.

[0051] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0052] In one embodiment, an automatic source layer cross-cluster data migration device is provided, and the automatic source layer cross-cluster data migration device corresponds to the automatic source layer cross-cluster data migration method in the above embodiment. Figure 3 As shown, the automatic source layer cross-cluster data migration device includes a task synchronization module 10, a stock data synchronization module 20 and an incremental data synchronization module 30. The detailed description of each functional module is as follows:

[0053] The task synchronization module 10 is used to execute step S10 in the automatic source layer cross-cluster data migration method in the above embodiment, that is, to synchronize the running scripts and scheduling tasks of the old cluster to the new cluster.

[0054] The stock data synchronization module 20 is used to execute step S20 in the automatic source layer cross-cluster data migration method in the above embodiment, that is, to modify the extraction task of the stock data in the old cluster into a distributed copy task of the stock data through the template of the derivative task, and use the distributed copy tool to migrate and synchronize the stock data in the old cluster source layer to the new cluster source layer at one time.

[0055] The incremental data synchronization module 30 is used to execute step S30 in the automatic source layer cross-cluster data migration method in the above embodiment, that is, to run the scheduling task. When the scheduling platform detects that the operating status of the old cluster meets the incremental data migration conditions, the incremental data extraction task in the old cluster is modified into a distributed copy task of the incremental data through the template of the derivative task, and the distributed copy tool is used to migrate and synchronize the incremental data in the old cluster source layer to the new cluster source layer.

[0056] In one embodiment, the incremental data synchronization module 30 is specifically used to:

[0057] Establish a task of converting a number to a distributed copy function, and determine whether the flag of the task of converting a number to a distributed copy function is effective;

[0058] If the flag is valid, determine whether there is incremental data in the old cluster source layer;

[0059] If there is incremental data, the incremental data extraction task in the old cluster is modified into a distributed copy task of incremental data through the template of the derivative task, and the incremental data in the old cluster source layer is migrated and synchronized to the new cluster source layer using the distributed copy tool.

[0060] In one embodiment, the incremental data synchronization module 30 is specifically used to:

[0061] Query and access the old cluster paths whose path modification time is today;

[0062] If data is queried on the old cluster path whose path modification time is today, it indicates that there is incremental data in the old cluster source layer;

[0063] If no data is found on the old cluster path whose path modification time is today, set the polling times to poll the old cluster path whose path modification time is today.

[0064] If data is queried on the old cluster path whose path modification time is today within the polling times, it indicates that there is incremental data in the old cluster source layer;

[0065] If no data is found on the old cluster path whose path modification time is today within the polling times, it indicates that there is no incremental data in the old cluster source layer.

[0066] In one embodiment, the incremental data synchronization module 30 is further used to:

[0067] Determine whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer;

[0068] If they are consistent, determine whether the data synchronization type is partition synchronization. If so, the partition needs to be repaired.

[0069] In one embodiment, the incremental data synchronization module 30 is specifically used to:

[0070] Determine whether the metadata of the new cluster library table is consistent with the metadata of the old cluster library table;

[0071] Determine whether the data volume of the new cluster is consistent with that of the old cluster;

[0072] Determine whether the new cluster data record is consistent with the old cluster data record;

[0073] If the metadata of the new cluster library table is consistent with the metadata of the old cluster library table, the data volume of the new cluster is consistent with the data volume of the old cluster, and the data records of the new cluster are consistent with the data records of the old cluster, it means that the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer.

[0074] In one embodiment, the incremental data synchronization module 30 is specifically used to:

[0075] Use the disk usage viewing tool to view the disk space occupied by the incremental data in the new cluster and the old cluster respectively. If the disk space occupied by the incremental data in the new cluster and the old cluster is the same, it indicates whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer.

[0076] In one embodiment, the incremental data synchronization module 30 is further used to:

[0077] If it is detected that the old cluster is offline, the extraction tool is used to extract incremental data from the data source and import it into the new cluster source layer.

[0078] The automatic source layer cross-cluster data migration device provided in the embodiment of the present invention synchronizes the running scripts and scheduling tasks of the old cluster to the new cluster. The new cluster and the old cluster use the same set of tasks, which can ensure that the new cluster can be switched online at any time; secondly, there is no need to edit the code script for data migration. It is only necessary to modify the extraction task in the old cluster into a distributed copy task through the template of the derivative task, and execute the distributed copy task to migrate the data in the source layer of the old cluster to the source layer of the new cluster, which saves a lot of script development time and improves the efficiency of data migration; thirdly, according to the set incremental data migration, Logic, when the running status of the old cluster meets the incremental data migration conditions, the incremental data is automatically migrated to the new cluster source layer without human intervention. The operation is simple and error-prone, reducing the difficulty of cross-cluster data migration. Finally, the distributed copy tool is used to synchronize the data migration in the old cluster source layer to the new cluster source layer, which can not only quickly copy the data in the old cluster source layer to the new cluster source layer, but also ensure the consistency of the data in the new and old cluster source layers. When the cluster and the old cluster are switched, the source data of the downstream application can be guaranteed to be consistent, thereby ensuring that the current business is not affected.

[0079] For the specific definition of the automatic source layer cross-cluster data migration device, please refer to the definition of the automatic source layer cross-cluster data migration method above, which will not be repeated here. Each module in the above-mentioned automatic source layer cross-cluster data migration device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0080] In one embodiment, a computer device is provided, whose internal structure diagram can be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with other external electronic devices through a network connection. When the computer program is executed by the processor, it is implemented Figure 1 Functions or steps of the method for automatic source layer cross-cluster data migration in the illustrated embodiment.

[0081] In one embodiment, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, Figure 1 Functions or steps of the method for automatic source layer cross-cluster data migration in the illustrated embodiment.

[0082] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0083] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0084] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. An automated source layer cross-cluster data migration method, characterized in that: include: Synchronize the running scripts and scheduling tasks of the old cluster to the new cluster; Use the template of the derivative task to modify the extraction task of the existing data in the old cluster into a distributed copy task of the existing data, and use the distributed copy tool to migrate and synchronize the existing data in the old cluster source layer to the new cluster source layer at one time; Run the scheduling task. When the scheduling platform detects that the running status of the old cluster meets the incremental data migration conditions, the incremental data extraction task in the old cluster is modified into a distributed copy task of the incremental data through the template of the derivative task. The distributed copy tool is used to synchronize the incremental data migration in the old cluster source layer to the new cluster source layer.

2. The method for automatic source layer cross-cluster data migration according to claim 1, characterized in that: The running scheduling task, when the scheduling platform detects that the running status of the old cluster meets the incremental data migration conditions, modifies the incremental data extraction task in the old cluster into a distributed copy task of the incremental data through the template of the derivative task, and uses the distributed copy tool to synchronize the incremental data migration in the old cluster source layer to the new cluster source layer, including: Establish a task of converting a number to a distributed copy function, and determine whether the flag of the task of converting a number to a distributed copy function is effective; If the flag is valid, determine whether there is incremental data in the old cluster source layer; If there is incremental data, the incremental data extraction task in the old cluster is modified into a distributed copy task of incremental data through the template of the derivative task, and the incremental data in the old cluster source layer is migrated and synchronized to the new cluster source layer using the distributed copy tool.

3. The automatic source layer cross-cluster data migration method according to claim 2, characterized in that: The determining whether there is incremental data in the old cluster source layer includes: Query and access the old cluster paths whose path modification time is today; If data is queried on the old cluster path whose path modification time is today, it indicates that there is incremental data in the old cluster source layer; If no data is found on the old cluster path whose path modification time is today, set the polling times to poll the old cluster path whose path modification time is today. If data is queried on the old cluster path whose path modification time is today within the polling times, it indicates that there is incremental data in the old cluster source layer; If no data is found on the old cluster path whose path modification time is today within the polling times, it indicates that there is no incremental data in the old cluster source layer.

4. The method for automatic source layer cross-cluster data migration as claimed in claim 2, characterized in that: After the distributed copy tool is used to migrate and synchronize the incremental data in the old cluster source layer to the new cluster source layer, the following steps are included: Determine whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer; If they are consistent, determine whether the data synchronization type is partition synchronization. If so, the partition needs to be repaired.

5. The method for automatic source layer cross-cluster data migration as claimed in claim 4, characterized in that: The determining whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer includes: Determine whether the metadata of the new cluster library table is consistent with the metadata of the old cluster library table; Determine whether the data volume of the new cluster is consistent with that of the old cluster; Determine whether the new cluster data record is consistent with the old cluster data record; If the metadata of the new cluster library table is consistent with the metadata of the old cluster library table, the data volume of the new cluster is consistent with the data volume of the old cluster, and the data records of the new cluster are consistent with the data records of the old cluster, it means that the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer.

6. The automatic source layer cross-cluster data migration method according to claim 4, characterized in that: The determining whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer includes: Use the disk usage viewing tool to view the disk space occupied by the incremental data in the new cluster and the old cluster respectively. If the disk space occupied by the incremental data in the new cluster and the old cluster is the same, it indicates whether the incremental data in the new cluster source layer is consistent with the incremental data in the old cluster source layer.

7. The method for automatic source layer cross-cluster data migration according to any one of claims 1 to 6, characterized in that: include: If it is detected that the old cluster is offline, use the extraction tool to extract incremental data from the data source and import it into the new cluster source layer.

8. An automatic source layer cross-cluster data migration device, characterized in that: include: The task synchronization module is used to synchronize the running scripts and scheduling tasks of the old cluster to the new cluster; The stock data synchronization module is used to modify the extraction task of the stock data in the old cluster into a distributed copy task of the stock data through the template of the derivative task, and use the distributed copy tool to migrate and synchronize the stock data in the old cluster source layer to the new cluster source layer at one time; The incremental data synchronization module is used to run scheduling tasks. When the scheduling platform detects that the running status of the old cluster meets the incremental data migration conditions, the incremental data extraction task in the old cluster is modified into a distributed copy task of the incremental data through the template of the derivative task, and the distributed copy tool is used to migrate and synchronize the incremental data in the old cluster source layer to the new cluster source layer.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for automatic source layer cross-cluster data migration as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, characterized in that: When the computer program is executed by a processor, the steps of the method for automatic source layer cross-cluster data migration as claimed in any one of claims 1 to 7 are implemented.

Images