Cluster migration method and device, equipment and medium

Abstract

The invention belongs to the field of big data, and relates to a cluster migration method, which comprises the steps of migrating historical data of a source cluster to a target cluster based on acquired source cluster information of the source cluster, target cluster information of the target cluster and a migration strategy, and synchronizing a first task of the source cluster to the target cluster, source data and target data corresponding to the first task are obtained; deploying the second task to the source cluster and the target cluster at the same time to obtain source real-time data and target real-time data corresponding to the second task; based on the data, performing data consistency verification on the source cluster and the target cluster; and if the data consistency verification is passed, performing subsequent operation to complete cluster migration. The invention further provides a device, equipment and a medium. In addition, the invention also relates to a block chain technology, and the source cluster information and the target cluster information can be stored in a block chain. According to the method, the historical data and the real-time data can be synchronously migrated during cluster migration, and the data integrity is ensured.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a cluster migration method, device, equipment and medium. Background Art

[0002] In the widespread application of big data technology, big data clusters are the core components of data storage and processing, and their stability and efficiency are crucial. However, with the expansion of business and the iteration of technology, the migration demand for big data clusters has gradually emerged, especially in large enterprises, where computer room cluster migration has become an indispensable task. However, there is a relative lack of tools for consistent migration of big data clusters on the current market.

[0003] The reason is that, on the one hand, cluster migration is not a problem that all enterprises will face frequently. Only large enterprises with a certain scale and business complexity will need to migrate computer room clusters due to business expansion or technology upgrades. This relatively niche demand has led to insufficient development of corresponding migration tools on the market. On the other hand, the implementation of big data cluster migration is extremely difficult. During the migration process, business continuity must be ensured while ensuring the consistency of data before and after migration. This requires the migration tool to not only have efficient data synchronization capabilities, but also be able to handle various complex data relationships and business logic during the migration process. In addition, with the rapid development of big data technology, most traditional migration solutions only migrate data for offline clusters, but lack effective support for the migration of real-time data. Therefore, it is urgent to develop a method that can synchronously migrate historical data and real-time data to ensure data integrity during cluster migration. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a cluster migration method, apparatus, device and medium to solve the problem that it is difficult to synchronously migrate historical data and real-time data to ensure data integrity during existing cluster migration.

[0005] In order to solve the above technical problems, the present application embodiment provides a cluster migration method, which adopts the following technical solutions:

[0006] Obtain the source cluster information of the source cluster, the target cluster information of the target cluster and the migration strategy; based on the source cluster information, the target cluster information and the migration strategy, use the data synchronization service to migrate the historical data of the source cluster to the target cluster, and use the task migration module to synchronize the first task of the source cluster to the target cluster, so as to obtain the source data corresponding to the first task from the source cluster and the target data corresponding to the first task from the target cluster; when the second task is detected, deploy the second task to the source cluster and the target cluster at the same time, so as to obtain the source real-time data corresponding to the second task from the source cluster and the target real-time data corresponding to the second task from the target cluster; based on the historical data, source data, target data, source real-time data and target real-time data, use the cluster migration module to verify the data consistency of the source cluster and the target cluster; if the data consistency verification passes, switch the database task synchronization source of the target cluster from the source cluster to the database of the target cluster, and obtain the operating indicator data of the target cluster. When the operating indicator data meets the preset requirements, stop the task scheduling of the source cluster to complete the cluster migration.

[0007] Furthermore, before the step of obtaining source cluster information of the source cluster, target cluster information of the target cluster, and the migration strategy, the method further includes:

[0008] Obtain the data architecture information of the source cluster, which includes data mode information, data volume, and data distribution information; determine the migration strategy and architecture design information of the target cluster based on the data mode information, data volume, and data distribution information, and build the target cluster based on the architecture design information.

[0009] Furthermore, based on the source cluster information, the target cluster information and the migration strategy, the steps of using the data synchronization service to migrate the historical data of the source cluster to the target cluster specifically include:

[0010] Based on the source cluster information, target cluster information and migration strategy, determine the data lake task of the target cluster; based on the data lake task, obtain the data table and field information corresponding to the source cluster; based on the data table and field information, determine the historical data that needs to be synchronized in the source cluster, and use data migration instructions to synchronize the historical data to the target cluster.

[0011] Further, the task migration module is used to synchronize the first task of the source cluster to the target cluster, so as to obtain source data corresponding to the first task from the source cluster and obtain target data corresponding to the first task from the target cluster, which specifically includes:

[0012] Obtain task information of the first task of the source cluster, and create a corresponding first task in the target cluster based on the task information; obtain the real-time data stream generated for the first task, and use a distributed message queue to write the real-time data stream to the source cluster and the target cluster respectively, so as to obtain the source data corresponding to the source cluster and the target data corresponding to the target cluster.

[0013] Further, when the second task is detected, the second task is deployed to the source cluster and the target cluster at the same time to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster, specifically including:

[0014] According to the task release process of the data development platform, the metadata information of the second task is obtained; the metadata information of the second task is parsed to obtain the configuration information of the second task on the source cluster and the target cluster; using a distributed task scheduling framework, the configuration information is deployed to the source cluster and the target cluster at the same time to start the second task of the source cluster and the target cluster respectively, obtain the source real-time data of the source cluster, and obtain the target real-time data of the target data cluster.

[0015] Furthermore, based on the historical data, source data, target data, source real-time data and target real-time data, a cluster migration module is used to verify the data consistency of the source cluster and the target cluster, specifically including:

[0016] Obtain target metadata information of the source cluster and the target cluster; compare the table structures of the source cluster and the target cluster based on the target metadata information, historical data, source data, target data, source real-time data and target real-time data, and determine the data tables that need to be verified for data consistency based on the comparison results; use the cluster migration module to perform a full table scan on the data table, use a hash function to calculate a hash value for each row of data in the data table, and sum the hash values ​​to obtain a hash row sum result for each table; compare the hash row sum results of the same data tables in the source cluster and the target cluster; if the hash row sum results are the same, it is determined that the data consistency verification of the source cluster and the target cluster has passed; if the hash row sum results are different, it is determined that the data consistency verification of the source cluster and the target cluster has failed.

[0017] In order to solve the above technical problems, the embodiment of the present application further provides a cluster migration device, which adopts the following technical solutions:

[0018] The acquisition module is used to obtain the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy;

[0019] A migration module is used to migrate the historical data of the source cluster to the target cluster using the data synchronization service based on the source cluster information, the target cluster information and the migration strategy, and to synchronize the first task of the source cluster to the target cluster using the task migration module, so as to obtain the source data corresponding to the first task from the source cluster and the target data corresponding to the first task from the target cluster;

[0020] A deployment module, configured to deploy the second task to the source cluster and the target cluster simultaneously when the second task is detected, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster;

[0021] The verification module is used to verify the data consistency of the source cluster and the target cluster using the cluster migration module based on historical data, source data, target data, source real-time data, and target real-time data;

[0022] The switching module is used to switch the database task synchronization source of the target cluster from the source cluster to the database of the target cluster if the data consistency verification passes, and obtain the operating indicator data of the target cluster. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

[0023] Furthermore, the migration module includes:

[0024] Determine the submodule, which is used to determine the data lake task of the target cluster based on the source cluster information, the target cluster information, and the migration strategy;

[0025] The first acquisition submodule is used to obtain the data table and field information corresponding to the source cluster based on the data lake task;

[0026] The synchronization submodule is used to determine the historical data that needs to be synchronized in the source cluster based on the data table and field information, and synchronize the historical data to the target cluster using data migration instructions.

[0027] In order to solve the above technical problems, an embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above cluster migration method are implemented.

[0028] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above cluster migration method.

[0029] Compared with the prior art, the embodiment of the present application has the following beneficial effects: through refined management and efficient migration strategy, intelligent data synchronization and task migration of the source cluster and the target cluster are realized. First, the seamless migration of historical data is ensured through the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenge of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after the migration. Finally, after the data consistency is verified and the target cluster operation indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, and provides an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0031] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0032] Figure 2 It is a flowchart of a cluster migration method provided by this application;

[0033] Figure 3 It is a structural schematic diagram of a cluster migration device provided by the present application;

[0034] Figure 4 It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0036] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0039] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0040] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0041] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0042] It should be noted that the cluster migration method provided in the embodiment of the present application is generally executed by a server, and accordingly, the cluster migration device is generally arranged in the server.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0044] Continue to refer Figure 2 , shows a flow chart of an embodiment of a cluster migration method according to the present application. The cluster migration method comprises the following steps:

[0045] Step S201 : obtaining source cluster information of a source cluster, target cluster information of a target cluster, and a migration strategy.

[0046] In this embodiment, the cluster migration method is executed on the electronic device (eg Figure 1 The server shown in the figure) can obtain the source cluster information, the target cluster information and the migration strategy through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0047] The source cluster refers to the data cluster that is currently using or storing the data to be migrated. The source cluster contains the original data architecture, data model, data volume, and data distribution information, and is the starting point of the data migration process.

[0048] The source cluster information refers to detailed data about the source cluster. The source cluster information includes but is not limited to data architecture information (such as data mode, data volume, data distribution), cluster configuration, node information, etc. The source cluster information can be used to evaluate migration requirements and formulate migration strategies.

[0049] The target cluster refers to the destination data cluster for data migration. The target cluster can be built according to the migration strategy and architecture design information to accommodate the data migrated from the source cluster.

[0050] The target cluster information refers to all relevant configurations and data about the target cluster, including architecture design information, cluster size, storage configuration, etc. This information is used to ensure the accuracy of data migration and the applicability of the target cluster.

[0051] The migration strategy is a detailed migration plan based on the data architecture information of the source cluster and the requirements of the target cluster. The migration strategy can include the order, method, and time of data migration to ensure the efficiency and security of data migration.

[0052] Step S202, based on the source cluster information, the target cluster information and the migration strategy, uses the data synchronization service to migrate the historical data of the source cluster to the target cluster, and uses the task migration module to synchronize the first task of the source cluster to the target cluster, so as to obtain the source data corresponding to the first task from the source cluster and obtain the target data corresponding to the first task from the target cluster.

[0053] Among them, data synchronization service refers to a service used to transfer data between two data clusters, which can ensure the integrity and consistency of data during the migration process.

[0054] The historical data refers to all historical records or historical information that have been stored in the source cluster before data migration and need to be migrated to the target cluster. These data are the main objects in the migration process.

[0055] The task migration module refers to a component used to migrate the first task (such as a data processing task, an analysis task, etc.) in the source cluster to the target cluster, and it ensures that the task can run seamlessly on the target cluster.

[0056] The first task refers to the initial task in the source cluster that needs to be migrated to the target cluster, for example, a task involving data processing, calculation, or analysis.

[0057] The source data refers to the data generated when the first task is executed in the source cluster. These data will be compared and verified with the target data generated when the first task is executed in the target cluster to ensure data consistency between the source cluster and the target cluster.

[0058] The target data refers to the data generated when the target cluster executes the same task (first task) as the source cluster. The target data will be used to compare with the source data to verify the consistency and accuracy of the data migration.

[0059] Step S203, when the second task is detected, the second task is deployed to the source cluster and the target cluster at the same time, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster.

[0060] The second task refers to an additional task that needs to be deployed and run simultaneously on the source cluster and the target cluster in addition to the first task during the data migration process.

[0061] The source real-time data refers to the data generated or processed in real time by the second task in the source cluster during the data migration process. These source real-time data will be compared and verified with the target real-time data generated by executing the second task in the target cluster to ensure the real-time and accuracy of data migration.

[0062] The target real-time data refers to the data generated or processed in real time by the same second task as the source cluster in the target cluster during the data migration process. These target real-time data will be used to compare with the source real-time data to verify the consistency of the real-time data migration.

[0063] Step S204 , based on the historical data, source data, target data, source real-time data and target real-time data, a cluster migration module is used to verify data consistency between the source cluster and the target cluster.

[0064] The cluster migration module refers to a component used to execute the entire cluster migration process.

[0065] Among them, data consistency verification refers to the process of comparing and verifying the data in the source cluster and the target cluster after the data migration is completed, to ensure that the migrated data is completely consistent with the original data in structure and content.

[0066] Step S205, if the data consistency verification passes, the database task synchronization source of the target cluster is switched from the source cluster to the database of the target cluster, and the operating indicator data of the target cluster is obtained. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

[0067] The database task synchronization source refers to the source in the source cluster for generating and sending database tasks. These tasks will be synchronized to the target cluster during the migration process to ensure the continuity and consistency of database operations.

[0068] The database of the target cluster refers to the database system located in the target cluster, which will receive the data migrated from the source cluster and serve as the center for storing and processing the migrated data.

[0069] The operation indicator data refers to various performance indicator data of the target cluster during its operation after migration, such as response time, throughput, error rate, etc., which are used to evaluate the performance and stability of the target cluster.

[0070] The preset requirements refer to the specific standards or thresholds set for the target cluster's operating indicator data during the cluster migration process. Only when these indicator data meet the preset requirements can the cluster migration be considered successful and the task scheduling of the source cluster be stopped.

[0071] Among them, task scheduling refers to the process of allocating, executing and monitoring tasks in the source cluster.

[0072] Among them, cluster migration refers to the process of migrating the entire data cluster from the source cluster to the target cluster, including multiple steps such as data migration, task migration, and data consistency verification.

[0073] The embodiment of the present application can realize intelligent data synchronization and task migration of the source cluster and the target cluster through refined management and efficient migration strategies. First, the seamless migration of historical data is ensured through the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenge of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after the migration. Finally, after the data consistency is verified and the target cluster operation indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

[0074] In some optional implementations of this embodiment, before obtaining the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy in step 201, the following steps are specifically included:

[0075] Obtain the data architecture information of the source cluster, which includes data mode information, data volume, and data distribution information; determine the migration strategy and architecture design information of the target cluster based on the data mode information, data volume, and data distribution information, and build the target cluster based on the architecture design information.

[0076] In one example, take the credit card system of a large bank as an example. The source cluster of the system stores a large amount of user information, transaction records, credit scores and other data. First, the source cluster is fully scanned by a data scanning tool to obtain data architecture information. Data mode information can include data table structure, field type, index, etc.; data volume is obtained by counting the number of records in each table, file size and other indicators; data distribution information is obtained by analyzing the distribution of data in the source cluster (such as hot data, cold data, etc.). Next, the migration strategy can be determined based on the data architecture information of the source cluster and the characteristics of the financial system. At the same time, the architecture design information of the target cluster is designed based on the data volume and data distribution information. Considering the stability and scalability of the financial system, the target cluster adopts a distributed architecture and is equipped with high-performance storage and computing resources. Based on the architecture design information, the target cluster is built. For example, suitable hardware devices are selected, including high-performance servers, storage devices, network devices, etc. Then, software such as database management systems and middleware are installed and configured. Finally, according to the migration strategy, the data of the source cluster is migrated to the target cluster.

[0077] The embodiment of the present application can provide a solid data foundation for the formulation of migration strategies by comprehensively acquiring the data architecture information of the source cluster, including detailed data mode information, accurate data volume statistics, and in-depth data distribution analysis. This helps to ensure the integrity of the data structure and the accuracy of the data volume during the migration process, and avoid data loss or omissions. At the same time, the migration strategy formulated based on these data architecture information can fully consider the characteristics and business needs of the data and achieve efficient data migration. In addition, the target cluster architecture design based on the data architecture information design can ensure that the target cluster meets business needs in terms of performance, capacity, and scalability, and provide strong support for subsequent data processing and business applications.

[0078] In some optional implementations of this embodiment, step S202, based on the source cluster information, the target cluster information and the migration strategy, uses the data synchronization service to migrate the historical data of the source cluster to the target cluster, specifically including the following steps:

[0079] Based on the source cluster information, target cluster information and migration strategy, determine the data lake task of the target cluster; based on the data lake task, obtain the data table and field information corresponding to the source cluster; based on the data table and field information, determine the historical data that needs to be synchronized in the source cluster, and use data migration instructions to synchronize the historical data to the target cluster.

[0080] In one example, taking the credit system of a large bank as an example, the system needs to be migrated from the existing source cluster to a new target cluster to achieve centralized data management and efficient analysis. First, the data lake task of the target cluster can be determined based on the database type, data scale, data distribution and other information of the source cluster, as well as the storage architecture, computing power, data format and other requirements of the target cluster, combined with the migration strategy. Then, according to the requirements of the data lake task, the corresponding data table and field information are obtained from the source cluster. This includes the name, structure, index, etc. of the data table, as well as the name, type, length, etc. of the field. This information is the basis for subsequent data migration. Then, based on the obtained data table and field information, determine the historical data that needs to be synchronized in the source cluster. Finally, use data migration instructions (such as the distcp command) to synchronize the historical data in the source cluster to the target cluster.

[0081] The embodiment of the present application can accurately determine the data lake task of the target cluster by comprehensively considering the source cluster information, the target cluster information and the migration strategy, ensuring that the purpose of data migration is clear and meets business needs. On this basis, the data table and field information corresponding to the source cluster are accurately obtained, providing strong support for the screening and synchronization of historical data. At the same time, the historical data that needs to be synchronized is determined based on the data table and field information, and the data migration instructions are used for synchronization, which not only ensures the integrity and accuracy of the data, but also significantly improves the degree of automation of data migration.

[0082] In some optional implementations of this embodiment, step S202 uses a task migration module to synchronize the first task of the source cluster to the target cluster, so as to obtain source data corresponding to the first task from the source cluster and obtain target data corresponding to the first task from the target cluster, which specifically includes the following steps:

[0083] Obtain task information of the first task of the source cluster, and create a corresponding first task in the target cluster based on the task information; obtain the real-time data stream generated for the first task, and use a distributed message queue to write the real-time data stream to the source cluster and the target cluster respectively, so as to obtain the source data corresponding to the source cluster and the target data corresponding to the target cluster.

[0084] In one example, a transaction system of a large bank is used as an example. The system is migrating from an old version cluster (source cluster) to a new version cluster (target cluster). In order to ensure the real-time synchronization of transaction data and the consistency of tasks during the migration process, the technical solution proposed in this embodiment can be adopted. First, the task information of the real-time transaction task (first task) of the transaction system in the source cluster can be obtained, including task type, task parameters, trigger conditions, etc. Then, based on this information, a corresponding transaction task (first task) is created in the target cluster to ensure that the transaction tasks in the old and new clusters are consistent in function and logic. Next, for the transaction task, the data stream generated during the transaction is captured in real time. In order to ensure the real-time synchronization of data, a distributed message queue (such as Kafka) is used as a medium for data transmission. The real-time data stream is written to the source cluster and the target cluster at the same time, generating source data and target data respectively. During the migration process, by monitoring the status of the distributed message queue and the writing of data, the progress and status of data synchronization can be grasped in real time. Once data synchronization is abnormal or delayed, intervention and adjustment can be made immediately to ensure the real-time and accuracy of the data. In addition, in order to verify the consistency and accuracy of data during the migration process, a data verification and comparison mechanism can also be used. After the migration is completed, the source data and the target data are fully compared and verified to ensure that the two are completely consistent in content and structure. Through the implementation of this embodiment, the bank transaction system can achieve smooth migration from the source cluster to the target cluster. During the migration process, the real-time synchronization of data and the consistency of tasks are effectively guaranteed, ensuring the continuity and stability of the transaction system.

[0085] The embodiment of the present application can obtain the task information of the first task of the source cluster and accurately create the corresponding task in the target cluster, thereby ensuring the consistency of the new and old clusters at the task level, and effectively avoiding data loss or errors caused by task mismatch. At the same time, a distributed message queue is used to capture and synchronize the real-time data stream generated by the first task in real time, thereby realizing instant transmission and writing of data, significantly reducing data latency, and improving the real-time and accuracy of data. In addition, writing the real-time data stream to the source cluster and the target cluster at the same time not only ensures redundant storage of data, but also facilitates data verification and comparison during migration or upgrade, further ensuring the integrity and consistency of the data.

[0086] In some optional implementations of this embodiment, step S203, when the second task is detected, deploying the second task to the source cluster and the target cluster at the same time, so as to obtain source real-time data corresponding to the second task from the source cluster, and obtain target real-time data corresponding to the second task from the target cluster, specifically includes the following steps:

[0087] According to the task release process of the data development platform, the metadata information of the second task is obtained; the metadata information of the second task is parsed to obtain the configuration information of the second task on the source cluster and the target cluster; using a distributed task scheduling framework, the configuration information is deployed to the source cluster and the target cluster at the same time to start the second task of the source cluster and the target cluster respectively, obtain the source real-time data of the source cluster, and obtain the target real-time data of the target data cluster.

[0088] In one example, taking the real-time risk control system of a large bank as an example, the system needs to synchronize real-time risk control tasks and data between the source cluster (old version risk control system) and the target cluster (new version risk control system). In order to ensure the real-time and accuracy of the data, the technical solution proposed in this embodiment can be adopted. First, the metadata information of the real-time risk control task (second task) can be obtained according to the task release process of the data development platform. These metadata information may include the name, type, parameters, trigger conditions, etc. of the task, which is the basis for task execution. Then, the obtained metadata is parsed to obtain the configuration information of the real-time risk control task on the source cluster and the target cluster. These configuration information includes the execution environment, resource allocation, input and output parameters, etc. of the task, which is the key to the deployment and execution of the task on the cluster. Next, a distributed task scheduling framework is used to deploy the parsed configuration information to the source cluster and the target cluster at the same time. The distributed task scheduling framework can ensure the parallel execution and real-time scheduling of tasks on multiple clusters, thereby meeting the requirements of data real-time. After the task deployment is completed, the real-time risk control tasks on the source cluster and the target cluster begin to execute, and generate source real-time data and target real-time data respectively. In addition, in order to verify the accuracy and consistency of data synchronization, a data verification and comparison mechanism can also be used. During the task execution process, the source real-time data and the target real-time data can be compared and verified to ensure that the two are completely consistent in content and structure.

[0089] The embodiment of the present application can accurately obtain the metadata of the second task through the task release process of the data development platform, laying a solid foundation for subsequent data processing. Deep analysis of the metadata can accurately extract the configuration information of the source cluster and the target cluster, ensuring seamless docking of tasks on the two clusters. The distributed task scheduling framework is used to realize the parallel deployment of configuration information, and the second tasks of the source cluster and the target cluster are started at the same time, which significantly improves the efficiency of data processing. In addition, the solution can also capture the source real-time data of the source cluster and the target real-time data of the target cluster in real time to ensure the timeliness and accuracy of the data.

[0090] In some optional implementations of this embodiment, step S204, based on historical data, source data, target data, source real-time data, and target real-time data, a cluster migration module is used to verify data consistency between the source cluster and the target cluster, specifically including the following steps:

[0091] Obtain target metadata information of the source cluster and the target cluster; compare the table structures of the source cluster and the target cluster based on the target metadata information, historical data, source data, target data, source real-time data and target real-time data, and determine the data tables that need to be verified for data consistency based on the comparison results; use the cluster migration module to perform a full table scan on the data table, use a hash function to calculate a hash value for each row of data in the data table, and sum the hash values ​​to obtain a hash row sum result for each table; compare the hash row sum results of the same data tables in the source cluster and the target cluster; if the hash row sum results are the same, it is determined that the data consistency verification of the source cluster and the target cluster has passed; if the hash row sum results are different, it is determined that the data consistency verification of the source cluster and the target cluster has failed.

[0092] In one example, taking the data migration of a customer information management system of a large bank as an example, the system needs to migrate sensitive data such as basic customer information and transaction records from the source cluster to the target cluster. In order to ensure the consistency of the data after migration, the technical solution proposed in this embodiment can be adopted. First, the target metadata information of the source cluster and the target cluster is obtained, which may include the name, field type, field length, etc. of the data table. Then, based on the target metadata information, historical data, source data, target data, source real-time data, and target real-time data, the table structure of the source cluster and the target cluster is compared. Through the comparison, the data table that needs to be verified for data consistency is determined. Next, the cluster migration module is used to perform a full table scan on these data tables. During the scanning process, a hash function (such as MD5 or SHA-256) can be used to calculate each row of data to obtain a hash value. Then, these hash values ​​are summed to obtain the hash row summation result of each table. This step is intended to ensure that the data does not change during the migration process through the collision resistance characteristics of the hash function. Finally, the hash row summation results of the same data table in the source cluster and the target cluster are compared. If the hash row sum results are the same, it is determined that the data consistency verification of the source cluster and the target cluster has passed, indicating that the data has maintained integrity and consistency during the migration process. If the hash row sum results are different, it is determined that the data consistency verification has not passed, and further inspection is required for possible problems in the data migration process, such as data loss, data corruption, or data synchronization delay. Through the implementation of this embodiment, the bank customer information management system can realize data consistency verification after data migration. During the verification process, not only the accuracy and consistency of the data are ensured, but also the reliability and security of data migration are improved.

[0093] The embodiment of the present application can obtain the target metadata information of the source cluster and the target cluster, and compare the table structure by integrating historical data, source data, target data, source real-time data and target real-time data, and accurately locate the data table that needs to be verified. Utilize the full table scan function of the cluster migration module, combined with the collision resistance of the hash function, generate a unique hash value for each row of data, and obtain the hash row summation result of the table by summing, effectively avoiding data tampering or loss. Comparing the hash row summation results of the same data table of the source cluster and the target cluster can intuitively reflect the data consistency status. If the results are the same, the verification is passed, indicating that the data migration is accurate. Otherwise, the data inconsistency problem is quickly identified, which is convenient for timely investigation and correction. The solution provided in this embodiment can not only improve the reliability of data migration, but also significantly enhance the security and stability of data processing.

[0094] It should be emphasized that in order to further ensure the privacy and security of the source cluster information of the above-mentioned source cluster, the target cluster information of the target cluster, and the migration strategy, the source cluster information of the above-mentioned source cluster, the target cluster information of the target cluster, and the migration strategy can also be stored in a node of a blockchain.

[0095] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0097] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0098] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a cluster migration device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0099] like Figure 3 As shown, the cluster migration device 400 of this embodiment includes: an acquisition module 401, a migration module 402, a deployment module 403, a verification module 404 and a switching module 405. Among them:

[0100] An acquisition module 401 is used to acquire source cluster information of a source cluster, target cluster information of a target cluster, and a migration strategy;

[0101] Migration module 402, used to migrate historical data of the source cluster to the target cluster using data synchronization service based on source cluster information, target cluster information and migration strategy, and synchronize the first task of the source cluster to the target cluster using task migration module, so as to obtain source data corresponding to the first task from the source cluster and obtain target data corresponding to the first task from the target cluster;

[0102] A deployment module 403 is used to deploy the second task to the source cluster and the target cluster simultaneously when the second task is detected, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster;

[0103] A verification module 404 is used to verify data consistency between the source cluster and the target cluster using a cluster migration module based on historical data, source data, target data, source real-time data, and target real-time data;

[0104] The switching module 405 is used to switch the database task synchronization source of the target cluster from the source cluster to the database of the target cluster if the data consistency verification passes, and obtain the operating indicator data of the target cluster. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

[0105] The embodiment of the present application can realize intelligent data synchronization and task migration of the source cluster and the target cluster through refined management and efficient migration strategies. First, the seamless migration of historical data is ensured through the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenge of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after the migration. Finally, after the data consistency is verified and the target cluster operation indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

[0106] In one embodiment, the migration module 402 includes:

[0107] Determine the submodule, which is used to determine the data lake task of the target cluster based on the source cluster information, the target cluster information, and the migration strategy;

[0108] The first acquisition submodule is used to obtain the data table and field information corresponding to the source cluster based on the data lake task;

[0109] The synchronization submodule is used to determine the historical data that needs to be synchronized in the source cluster based on the data table and field information, and synchronize the historical data to the target cluster using data migration instructions.

[0110] The embodiment of the present application can accurately determine the data lake task of the target cluster by comprehensively considering the source cluster information, the target cluster information and the migration strategy, ensuring that the purpose of data migration is clear and meets business needs. On this basis, the data table and field information corresponding to the source cluster are accurately obtained, providing strong support for the screening and synchronization of historical data. At the same time, the historical data that needs to be synchronized is determined based on the data table and field information, and the data migration instructions are used for synchronization, which not only ensures the integrity and accuracy of the data, but also significantly improves the degree of automation of data migration.

[0111] In one embodiment, the migration module 402 includes:

[0112] A second acquisition submodule is used to acquire task information of the first task of the source cluster, and create a corresponding first task in the target cluster based on the task information;

[0113] The writing submodule is used to obtain the real-time data stream generated for the first task, and use a distributed message queue to write the real-time data stream to the source cluster and the target cluster respectively, to obtain the source data corresponding to the source cluster and the target data corresponding to the target cluster.

[0114] The embodiment of the present application can obtain the task information of the first task of the source cluster and accurately create the corresponding task in the target cluster, thereby ensuring the consistency of the new and old clusters at the task level, and effectively avoiding data loss or errors caused by task mismatch. At the same time, a distributed message queue is used to capture and synchronize the real-time data stream generated by the first task in real time, thereby realizing instant transmission and writing of data, significantly reducing data latency, and improving the real-time and accuracy of data. In addition, writing the real-time data stream to the source cluster and the target cluster at the same time not only ensures redundant storage of data, but also facilitates data verification and comparison during migration or upgrade, further ensuring the integrity and consistency of the data.

[0115] In one embodiment, the deployment module 403 includes:

[0116] A second acquisition submodule is used to acquire metadata information of the second task according to the task release process of the data development platform;

[0117] A parsing submodule, used to parse the metadata information of the second task to obtain configuration information of the second task on the source cluster and the target cluster;

[0118] The deployment submodule is used to deploy the configuration information to the source cluster and the target cluster simultaneously using a distributed task scheduling framework to start the second task of the source cluster and the target cluster respectively, obtain the source real-time data of the source cluster, and obtain the target real-time data of the target data cluster.

[0119] The embodiment of the present application can accurately obtain the metadata of the second task through the task release process of the data development platform, laying a solid foundation for subsequent data processing. Deep analysis of the metadata can accurately extract the configuration information of the source cluster and the target cluster, ensuring seamless docking of tasks on the two clusters. The distributed task scheduling framework is used to realize the parallel deployment of configuration information, and the second tasks of the source cluster and the target cluster are started at the same time, which significantly improves the efficiency of data processing. In addition, the solution can also capture the source real-time data of the source cluster and the target real-time data of the target cluster in real time to ensure the timeliness and accuracy of the data.

[0120] In one embodiment, the verification module 404 includes:

[0121] The third acquisition submodule is used to obtain target metadata information of the source cluster and the target cluster;

[0122] The comparison submodule is used to compare the table structures of the source cluster and the target cluster based on the target metadata information, historical data, source data, target data, source real-time data, and target real-time data, and determine the data table that needs to be verified for data consistency based on the comparison results;

[0123] The scanning submodule is used to use the cluster migration module to perform a full table scan on the data table, calculate each row of data in the data table using a hash function to obtain a hash value, and sum the hash values ​​to obtain the hash row sum result of each table;

[0124] The comparison submodule is used to compare the hash row sum results of the same data table in the source cluster and the target cluster;

[0125] The determination submodule is used to determine that the data consistency verification of the source cluster and the target cluster has passed if the hash row sum results are the same, and to determine that the data consistency verification of the source cluster and the target cluster has failed if the hash row sum results are different.

[0126] The embodiment of the present application can obtain the target metadata information of the source cluster and the target cluster, and comprehensively compare the table structure of historical data, source data, target data, source real-time data and target real-time data to accurately locate the data table that needs to be verified. Utilize the full table scan function of the cluster migration module, combined with the collision resistance of the hash function, generate a unique hash value for each row of data, and obtain the hash row summation result of the table by summing, effectively avoiding data tampering or loss. Comparing the hash row summation results of the same data table of the source cluster and the target cluster can intuitively reflect the data consistency status. If the results are the same, the verification is passed, indicating that the data migration is accurate; otherwise, the data inconsistency problem is quickly identified, which is convenient for timely investigation and correction. The solution provided by this embodiment can not only improve the reliability of data migration, but also significantly enhance the security and stability of data processing.

[0127] In one embodiment, the cluster migration device further includes:

[0128] An information acquisition module is used to obtain data architecture information of the source cluster, including data mode information, data volume, and data distribution information;

[0129] The construction module is used to determine the migration strategy and the architecture design information of the target cluster according to the data mode information, data volume and data distribution information, and to build the target cluster based on the architecture design information.

[0130] The embodiment of the present application can provide a solid data foundation for the formulation of migration strategies by comprehensively acquiring the data architecture information of the source cluster, including detailed data mode information, accurate data volume statistics, and in-depth data distribution analysis. This helps to ensure the integrity of the data structure and the accuracy of the data volume during the migration process, and avoid data loss or omissions. At the same time, the migration strategy formulated based on these data architecture information can fully consider the characteristics and business needs of the data and achieve efficient data migration. In addition, the target cluster architecture design based on the data architecture information design can ensure that the target cluster meets business needs in terms of performance, capacity, and scalability, and provide strong support for subsequent data processing and business applications.

[0131] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0132] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 6 having a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASIC), programmable gate arrays (FPGA), digital signal processors (DSP), embedded devices, etc.

[0133] Computer devices can be computing devices such as desktop computers, notebooks, PDAs, and cloud servers. Computer devices can interact with users through keyboards, mice, remote controls, touch pads, or voice control devices.

[0134] The memory 61 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 61 can be an internal storage unit of the computer device 6, such as a hard disk or memory of the computer device 6. In other embodiments, the memory 61 can also be an external storage device of the computer device 6, such as a plug-in hard disk equipped on the computer device 6, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 61 can also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions of the cluster migration method, etc. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are to be output.

[0135] The processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to run computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for running a cluster migration method.

[0136] The network interface 63 may include a wireless network interface or a wired network interface, and the network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0137] The embodiment of the present application can realize intelligent data synchronization and task migration of the source cluster and the target cluster through refined management and efficient migration strategies. First, the seamless migration of historical data is ensured through the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenge of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after the migration. Finally, after the data consistency is verified and the target cluster operation indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

[0138] The present application also provides another implementation, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the cluster migration method as described above.

[0139] The embodiment of the present application can realize intelligent data synchronization and task migration of the source cluster and the target cluster through refined management and efficient migration strategies. First, the seamless migration of historical data is ensured through the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenge of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after the migration. Finally, after the data consistency is verified and the target cluster operation indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0141] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0142] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A cluster migration method, characterized in that: The steps include: Obtain the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy; Based on the source cluster information, the target cluster information and the migration strategy, a data synchronization service is used to migrate the historical data of the source cluster to the target cluster, and a task migration module is used to synchronize the first task of the source cluster to the target cluster, so as to obtain source data corresponding to the first task from the source cluster and obtain target data corresponding to the first task from the target cluster; When a second task is detected, the second task is deployed to the source cluster and the target cluster at the same time, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster; Based on the historical data, the source data, the target data, the source real-time data and the target real-time data, a cluster migration module is used to perform data consistency verification on the source cluster and the target cluster; If the data consistency verification passes, the database task synchronization source of the target cluster is switched from the source cluster to the database of the target cluster, and the operating indicator data of the target cluster is obtained. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

2. The method according to claim 1, characterized in that Before the step of obtaining source cluster information of the source cluster, target cluster information of the target cluster, and the migration strategy, the step further includes: Acquire data architecture information of the source cluster, wherein the data architecture information includes data mode information, data volume, and data distribution information; A migration strategy and architecture design information of the target cluster are determined according to the data mode information, the data volume, and the data distribution information, and the target cluster is constructed based on the architecture design information.

3. The method according to claim 2, characterized in that The step of migrating the historical data of the source cluster to the target cluster using a data synchronization service based on the source cluster information, the target cluster information and the migration strategy specifically includes: Determine the data lake task of the target cluster based on the source cluster information, the target cluster information and the migration strategy; Based on the data lake task, obtain data table and field information corresponding to the source cluster; Based on the data table and field information, determine the historical data that needs to be synchronized in the source cluster, and use data migration instructions to synchronize the historical data to the target cluster.

4. The method according to claim 1, characterized in that: The step of using the task migration module to synchronize the first task of the source cluster to the target cluster to obtain source data corresponding to the first task from the source cluster and to obtain target data corresponding to the first task from the target cluster specifically includes: Acquire task information of a first task of the source cluster, and create a corresponding first task in the target cluster based on the task information; The real-time data stream generated for the first task is obtained, and the real-time data stream is written to the source cluster and the target cluster respectively using a distributed message queue to obtain source data corresponding to the source cluster and target data corresponding to the target cluster.

5. The method according to claim 1, characterized in that The step of simultaneously deploying the second task to the source cluster and the target cluster when the second task is detected, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster, specifically includes: According to the task release process of the data development platform, metadata information of the second task is obtained; Parsing metadata information of the second task to obtain configuration information of the second task on the source cluster and the target cluster; The configuration information is simultaneously deployed to the source cluster and the target cluster using a distributed task scheduling framework to start the second tasks of the source cluster and the target cluster respectively, obtain source real-time data of the source cluster, and obtain target real-time data of the target data cluster.

6. The method according to claim 1, characterized in that The step of using a cluster migration module to verify data consistency between the source cluster and the target cluster based on the historical data, the source data, the target data, the source real-time data, and the target real-time data specifically includes: Get the target metadata information of the source cluster and the target cluster; Based on the target metadata information, the historical data, the source data, the target data, the source real-time data, and the target real-time data, the table structures of the source cluster and the target cluster are compared, and based on the comparison result, a data table that needs to be verified for data consistency is determined; Using a cluster migration module, performing a full table scan on the data table, calculating each row of data in the data table using a hash function to obtain a hash value, and summing the hash values ​​to obtain a hash row summation result of each table; Comparing hash row sum results of the same data table in the source cluster and the target cluster; If the hash row sum results are the same, it is determined that the data consistency verification of the source cluster and the target cluster has passed; if the hash row sum results are different, it is determined that the data consistency verification of the source cluster and the target cluster has failed.

7. A cluster migration device, characterized in that: include: The acquisition module is used to obtain the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy; A migration module, configured to migrate the historical data of the source cluster to the target cluster using a data synchronization service based on the source cluster information, the target cluster information and the migration strategy, and to synchronize the first task of the source cluster to the target cluster using a task migration module, so as to obtain source data corresponding to the first task from the source cluster and obtain target data corresponding to the first task from the target cluster; A deployment module, configured to deploy the second task to the source cluster and the target cluster simultaneously when the second task is detected, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster; A verification module, configured to perform data consistency verification on the source cluster and the target cluster using a cluster migration module based on the historical data, the source data, the target data, the source real-time data, and the target real-time data; The switching module is used to switch the database task synchronization source of the target cluster from the source cluster to the database of the target cluster if the data consistency verification passes, and obtain the operating indicator data of the target cluster. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

8. The device according to claim 7, characterized in that The migration module includes: A determination submodule, configured to determine a data lake task of the target cluster based on the source cluster information, the target cluster information, and the migration strategy; A first acquisition submodule is used to acquire data tables and field information corresponding to the source cluster based on the data lake task; The synchronization submodule is used to determine the historical data that needs to be synchronized in the source cluster based on the data table and field information, and synchronize the historical data to the target cluster using data migration instructions.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the cluster migration method according to any one of claims 1 to 6 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the cluster migration method according to any one of claims 1 to 6 are implemented.

Images