Cluster migration method, device, equipment and medium

Abstract

The present application belongs to the field of big data and relates to a cluster migration method, including: based on the acquired source cluster information of the source cluster, the target cluster information of the target cluster and the migration strategy, migrating the historical data of the source cluster to the target cluster, synchronizing the first task of the source cluster to the target cluster to obtain the source data and target data corresponding to the first task; deploying the second task to the source cluster and the target cluster at the same time to obtain the source real-time data and target real-time data corresponding to the second task; based on the above data, performing data consistency verification on the source cluster and the target cluster; if the data consistency verification passes, performing subsequent operations to complete the cluster migration. The present application also provides a device, equipment and medium. In addition, the present application also relates to blockchain technology, and the source cluster information and the target cluster information can be stored in the blockchain. The present application can realize the synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a cluster migration method, apparatus, device, and medium. Background Art

[0002] In the widespread application of big data technologies, big data clusters, as core components for data storage and processing, are crucial for stability and efficiency. However, with business expansion and technological advancements, the need for big data cluster migration is gradually emerging. Especially in large enterprises, data center cluster migration has become an indispensable task. However, tools for consistent big data cluster migration are currently lacking in the market.

[0003] The reasons for this are, firstly, that cluster migration isn't a frequent problem for all enterprises. Only large enterprises with a certain scale and business complexity will need to migrate their data center clusters due to business expansion or technology upgrades. This relatively niche demand has led to a lack of development of corresponding migration tools in the market. Secondly, implementing big data cluster migration is extremely difficult. During the migration process, business continuity and data consistency before and after the migration must be ensured. This requires migration tools to not only have efficient data synchronization capabilities but also be able to handle a variety of complex data relationships and business logic during the migration process. Furthermore, with the rapid development of big data technology, traditional migration solutions mostly focus solely on migrating offline cluster data and lack effective support for real-time data migration. Therefore, there is an urgent need to develop a method that can simultaneously migrate historical and real-time data during cluster migration to ensure data integrity. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a cluster migration method, apparatus, device and medium to solve the problem that it is difficult to synchronize historical data and real-time data to ensure data integrity during existing cluster migration.

[0005] To solve the above technical problems, the present invention provides a cluster migration method, which adopts the following technical solutions:

[0006] Obtain the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy; based on the source cluster information, the target cluster information, and the migration strategy, use the data synchronization service to migrate the historical data of the source cluster to the target cluster, and use the task migration module to synchronize the first task of the source cluster to the target cluster to obtain the source data corresponding to the first task from the source cluster and the target data corresponding to the first task from the target cluster; when the second task is detected, deploy the second task to the source cluster and the target cluster at the same time to obtain the source real-time data corresponding to the second task from the source cluster and the target real-time data corresponding to the second task from the target cluster; based on the historical data, source data, target data, source real-time data, and target real-time data, use the cluster migration module to verify the data consistency of the source cluster and the target cluster; if the data consistency verification passes, switch the database task synchronization source of the target cluster from the source cluster to the database of the target cluster, and obtain the operating indicator data of the target cluster. When the operating indicator data meets the preset requirements, stop the task scheduling of the source cluster to complete the cluster migration.

[0007] Furthermore, before obtaining the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy, the following steps are also included:

[0008] Obtain the data architecture information of the source cluster, which includes data model information, data volume, and data distribution information; determine the migration strategy and the architecture design information of the target cluster based on the data model information, data volume, and data distribution information; and build the target cluster based on the architecture design information.

[0009] Furthermore, based on the source cluster information, target cluster information, and migration strategy, the data synchronization service is used to migrate the historical data of the source cluster to the target cluster. Specifically, the steps include:

[0010] Based on the source cluster information, target cluster information, and migration strategy, determine the data lake task for the target cluster. Based on the data lake task, obtain the data table and field information corresponding to the source cluster. Based on the data table and field information, determine the historical data that needs to be synchronized in the source cluster, and use data migration instructions to synchronize the historical data to the target cluster.

[0011] Furthermore, the task migration module is used to synchronize the first task of the source cluster to the target cluster, so as to obtain source data corresponding to the first task from the source cluster and target data corresponding to the first task from the target cluster. Specifically, the steps include:

[0012] Obtain the task information of the first task of the source cluster, and create the corresponding first task in the target cluster based on the task information; obtain the real-time data stream generated for the first task, and use a distributed message queue to write the real-time data stream to the source cluster and the target cluster respectively, to obtain the source data corresponding to the source cluster and the target data corresponding to the target cluster.

[0013] Furthermore, when the second task is detected, the second task is deployed to the source cluster and the target cluster simultaneously to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster, specifically including:

[0014] According to the task release process of the data development platform, the metadata information of the second task is obtained; the metadata information of the second task is parsed to obtain the configuration information of the second task on the source cluster and the target cluster; using a distributed task scheduling framework, the configuration information is deployed to the source cluster and the target cluster at the same time to start the second task of the source cluster and the target cluster respectively, obtain the source real-time data of the source cluster, and obtain the target real-time data of the target data cluster.

[0015] Furthermore, based on the historical data, source data, target data, source real-time data, and target real-time data, the cluster migration module is used to verify the data consistency of the source cluster and the target cluster. Specifically, the steps include:

[0016] Obtain target metadata information of the source cluster and the target cluster; compare the table structures of the source cluster and the target cluster based on the target metadata information, historical data, source data, target data, source real-time data, and target real-time data, and determine the data tables that require data consistency verification based on the comparison results; use the cluster migration module to perform a full table scan of the data table, calculate each row of data in the data table using a hash function to obtain a hash value, and sum the hash values ​​to obtain a hash row sum result for each table; compare the hash row sum results of the same data tables in the source cluster and the target cluster; if the hash row sum results are the same, it is determined that the data consistency verification of the source cluster and the target cluster has passed; if the hash row sum results are different, it is determined that the data consistency verification of the source cluster and the target cluster has failed.

[0017] In order to solve the above technical problems, the embodiment of the present application further provides a cluster migration device, which adopts the following technical solutions:

[0018] The acquisition module is used to obtain the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy;

[0019] The migration module is used to migrate the historical data of the source cluster to the target cluster using the data synchronization service based on the source cluster information, the target cluster information and the migration strategy. The task migration module is used to synchronize the first task of the source cluster to the target cluster to obtain the source data corresponding to the first task from the source cluster and the target data corresponding to the first task from the target cluster.

[0020] a deployment module, configured to, when a second task is detected, deploy the second task to both the source cluster and the target cluster, so as to obtain source real-time data corresponding to the second task from the source cluster and target real-time data corresponding to the second task from the target cluster;

[0021] The verification module is used to verify the data consistency of the source and target clusters based on historical data, source data, target data, source real-time data, and target real-time data using the cluster migration module.

[0022] The switching module is used to switch the database task synchronization source of the target cluster from the source cluster to the database of the target cluster if the data consistency verification passes, and obtain the operating indicator data of the target cluster. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

[0023] Furthermore, the migration module includes:

[0024] Determine the submodule, which is used to determine the data lake tasks of the target cluster based on the source cluster information, target cluster information, and migration strategy;

[0025] The first acquisition submodule is used to obtain the data table and field information corresponding to the source cluster based on the data lake task;

[0026] The synchronization submodule is used to determine the historical data that needs to be synchronized in the source cluster based on data table and field information, and use data migration instructions to synchronize the historical data to the target cluster.

[0027] In order to solve the above technical problems, an embodiment of the present application further provides a computer device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the above cluster migration method are implemented.

[0028] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above cluster migration method.

[0029] Compared with the prior art, the embodiments of the present application mainly have the following beneficial effects: through refined management and efficient migration strategies, intelligent data synchronization and task migration of the source cluster and the target cluster are realized. First, the seamless migration of historical data is ensured through the data synchronization service, and at the same time, the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenges of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after migration. Finally, after the data consistency is verified and the target cluster operating indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0032] Figure 2 This is a flow chart of a cluster migration method provided by this application;

[0033] Figure 3 This is a schematic diagram of the structure of a cluster migration device provided by this application;

[0034] Figure 4 This is a structural diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0036] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0038] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0039] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0040] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0041] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0042] It should be noted that the cluster migration method provided in the embodiment of the present application is generally executed by a server, and accordingly, the cluster migration device is generally set in the server.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0044] Continue to refer Figure 2 , shows a flow chart of an embodiment of a cluster migration method according to the present application. The cluster migration method includes the following steps:

[0045] Step S201 : Acquire source cluster information of a source cluster, target cluster information of a target cluster, and a migration strategy.

[0046] In this embodiment, the cluster migration method is executed on the electronic device (eg Figure 1 The server shown in FIG. 1 may obtain source cluster information, target cluster information, and migration policies through a wired or wireless connection. It should be noted that the wireless connection methods described above may include, but are not limited to, 3G / 4G / 5G connections, WiFi connections, Bluetooth connections, WiMAX connections, Zigbee connections, UWB (ultra wideband) connections, and other currently known or future developed wireless connection methods.

[0047] The source cluster is the data cluster currently using or storing the data to be migrated. It contains the original data architecture, data model, data volume, and data distribution information, and is the starting point of the data migration process.

[0048] Source cluster information refers to detailed data about the source cluster. This includes, but is not limited to, data architecture information (such as data schema, data volume, and data distribution), cluster configuration, and node information. This information can be used to assess migration requirements and develop migration strategies.

[0049] The target cluster is the destination data cluster for data migration. The target cluster can be built based on the migration strategy and architecture design information to accommodate the data migrated from the source cluster.

[0050] The target cluster information refers to all relevant configurations and data about the target cluster, including architecture design information, cluster size, storage configuration, etc. This information is used to ensure the accuracy of data migration and the applicability of the target cluster.

[0051] A migration strategy is a detailed migration plan based on the source cluster's data architecture and the target cluster's requirements. This strategy can include the order, method, and timing of data migration to ensure efficient and secure data migration.

[0052] In step S202, based on the source cluster information, the target cluster information and the migration strategy, the data synchronization service is used to migrate the historical data of the source cluster to the target cluster, and the task migration module is used to synchronize the first task of the source cluster to the target cluster, so as to obtain the source data corresponding to the first task from the source cluster and obtain the target data corresponding to the first task from the target cluster.

[0053] Among them, data synchronization service refers to a service used to transfer data between two data clusters, which can ensure the integrity and consistency of data during the migration process.

[0054] Among them, historical data refers to all historical records or historical information stored in the source cluster before data migration that needs to be migrated to the target cluster. This data is the main object in the migration process.

[0055] The task migration module refers to a component used to migrate the first task (such as a data processing task, an analysis task, etc.) in the source cluster to the target cluster, and it ensures that the task can run seamlessly on the target cluster.

[0056] The first task refers to the initial task in the source cluster that needs to be migrated to the target cluster, such as a task involving data processing, computing, or analysis.

[0057] The source data refers to the data generated when the first task is executed in the source cluster. This data will be compared and verified with the target data generated when the first task is executed in the target cluster to ensure data consistency between the source cluster and the target cluster.

[0058] The target data refers to the data generated when the target cluster executes the same task (first task) as the source cluster. The target data will be used to compare with the source data to verify the consistency and accuracy of the data migration.

[0059] Step S203 , when the second task is detected, the second task is deployed to the source cluster and the target cluster simultaneously, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster.

[0060] The second task refers to an additional task that needs to be deployed and run simultaneously on the source cluster and the target cluster during the data migration process, in addition to the first task.

[0061] Source real-time data refers to the data generated or processed in real time by the second task in the source cluster during the data migration process. This source real-time data is compared and verified with the target real-time data generated by the second task in the target cluster to ensure the real-time and accuracy of the data migration.

[0062] Target real-time data refers to the data generated or processed in real time by the same second task in the target cluster as in the source cluster during the data migration process. This target real-time data is compared with the source real-time data to verify the consistency of the real-time data migration.

[0063] Step S204 : Based on the historical data, source data, target data, source real-time data, and target real-time data, a cluster migration module is used to perform data consistency verification on the source cluster and the target cluster.

[0064] The cluster migration module refers to the component used to execute the entire cluster migration process.

[0065] Data consistency verification refers to the process of comparing and verifying the data in the source cluster and the target cluster after data migration is completed to ensure that the migrated data is completely consistent with the original data in structure and content.

[0066] In step S205, if the data consistency verification passes, the database task synchronization source of the target cluster is switched from the source cluster to the database of the target cluster, and the operating indicator data of the target cluster is obtained. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

[0067] The database task synchronization source refers to the source in the source cluster used to generate and send database tasks. These tasks will be synchronized to the target cluster during the migration process to ensure the continuity and consistency of database operations.

[0068] The database of the target cluster refers to the database system located in the target cluster, which will receive the data migrated from the source cluster and serve as the center for storing and processing the migrated data.

[0069] The operation indicator data refers to various performance indicator data of the target cluster during its operation after migration, such as response time, throughput, error rate, etc., which are used to evaluate the performance and stability of the target cluster.

[0070] The preset requirements refer to the specific standards or thresholds set for the target cluster's operating indicator data during the cluster migration process. Only when these indicator data meet the preset requirements can the cluster migration be considered successful and task scheduling of the source cluster be stopped.

[0071] Task scheduling refers to the process of allocating, executing, and monitoring tasks in the source cluster.

[0072] Among them, cluster migration refers to the process of migrating the entire data cluster from the source cluster to the target cluster, including multiple steps such as data migration, task migration, and data consistency verification.

[0073] The embodiment of the present application can realize intelligent data synchronization and task migration of the source cluster and the target cluster through refined management and efficient migration strategies. First, the seamless migration of historical data is ensured by the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenges of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after migration. Finally, after the data consistency is verified and the target cluster operating indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

[0074] In some optional implementations of this embodiment, before obtaining the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy in step 201, the following steps are further included:

[0075] Obtain the data architecture information of the source cluster, which includes data model information, data volume, and data distribution information; determine the migration strategy and the architecture design information of the target cluster based on the data model information, data volume, and data distribution information; and build the target cluster based on the architecture design information.

[0076] In one example, consider the credit card system of a large bank. The system's source cluster stores a vast amount of user information, transaction records, credit scores, and other data. First, a data scanning tool is used to thoroughly scan the source cluster to obtain data architecture information. Data schema information can include table structure, field types, and indexes. Data volume is determined by counting metrics such as the number of records in each table and file size. Data distribution information is obtained by analyzing the distribution of data within the source cluster (e.g., hot and cold data). Next, a migration strategy is determined based on the source cluster's data architecture information and the characteristics of the financial system. Simultaneously, the architecture of the target cluster is designed based on the data volume and distribution information. Considering the stability and scalability of the financial system, the target cluster adopts a distributed architecture and is equipped with high-performance storage and computing resources. Based on this architecture design information, the target cluster is constructed. For example, appropriate hardware is selected, including high-performance servers, storage devices, and network equipment. Next, software such as the database management system and middleware are installed and configured. Finally, according to the migration strategy, the data from the source cluster is migrated to the target cluster.

[0077] The embodiments of the present application can provide a solid data foundation for the formulation of migration strategies by comprehensively acquiring the data architecture information of the source cluster, including detailed data model information, accurate data volume statistics, and in-depth data distribution analysis. This helps to ensure the integrity of the data structure and the accuracy of the data volume during the migration process, and avoid data loss or omissions. At the same time, the migration strategy formulated based on these data architecture information can fully consider the characteristics of the data and business needs, and achieve efficient data migration. In addition, the target cluster architecture design based on the data architecture information can ensure that the target cluster meets business needs in terms of performance, capacity, and scalability, and provide strong support for subsequent data processing and business applications.

[0078] In some optional implementations of this embodiment, step S202, based on the source cluster information, the target cluster information, and the migration strategy, uses a data synchronization service to migrate the historical data of the source cluster to the target cluster, specifically including the following steps:

[0079] Based on the source cluster information, target cluster information, and migration strategy, determine the data lake task for the target cluster. Based on the data lake task, obtain the data table and field information corresponding to the source cluster. Based on the data table and field information, determine the historical data that needs to be synchronized in the source cluster, and use data migration instructions to synchronize the historical data to the target cluster.

[0080] In one example, taking the credit system of a large bank as an example, the system needs to be migrated from the existing source cluster to a brand new target cluster to achieve centralized data management and efficient analysis. First, based on the database type, data scale, data distribution and other information of the source cluster, as well as the storage architecture, computing power, data format and other requirements of the target cluster, combined with the migration strategy, the data lake task of the target cluster can be determined. Then, according to the requirements of the data lake task, the corresponding data table and field information are obtained from the source cluster. This includes the name, structure, index, etc. of the data table, as well as the name, type, length, etc. of the field. This information is the basis for subsequent data migration. Then, based on the obtained data table and field information, the historical data that needs to be synchronized in the source cluster is determined. Finally, data migration instructions (such as the distcp command) are used to synchronize the historical data in the source cluster to the target cluster.

[0081] The embodiment of the present application can accurately determine the data lake tasks of the target cluster by comprehensively considering the source cluster information, target cluster information and migration strategy, ensuring that the purpose of data migration is clear and meets business needs. On this basis, the data table and field information corresponding to the source cluster is accurately obtained, providing strong support for the screening and synchronization of historical data. At the same time, the historical data that needs to be synchronized is determined based on the data table and field information, and data migration instructions are used for synchronization, which not only ensures the integrity and accuracy of the data, but also significantly improves the degree of automation of data migration.

[0082] In some optional implementations of this embodiment, step S202 uses a task migration module to synchronize the first task of the source cluster to the target cluster, thereby obtaining source data corresponding to the first task from the source cluster and obtaining target data corresponding to the first task from the target cluster. Specifically, the following steps are included:

[0083] Obtain the task information of the first task of the source cluster, and create the corresponding first task in the target cluster based on the task information; obtain the real-time data stream generated for the first task, and use a distributed message queue to write the real-time data stream to the source cluster and the target cluster respectively, to obtain the source data corresponding to the source cluster and the target data corresponding to the target cluster.

[0084] In one example, a large bank's trading system is undergoing a migration from an old cluster (source cluster) to a new cluster (target cluster). To ensure real-time synchronization of transaction data and task consistency during the migration process, the technical solution proposed in this embodiment can be employed. First, task information for a real-time transaction task (first task) in the trading system in the source cluster can be obtained, including task type, task parameters, trigger conditions, and so on. Then, based on this information, a corresponding transaction task (first task) is created in the target cluster, ensuring functional and logical consistency between the old and new clusters. Next, the data stream generated during the transaction is captured in real time for this transaction task. To ensure real-time data synchronization, a distributed message queue (such as Kafka) is used as the data transmission medium. The real-time data stream is written simultaneously to the source and target clusters, generating source and target data, respectively. During the migration process, the progress and status of data synchronization can be monitored in real time by monitoring the status of the distributed message queue and the data write status. If any data synchronization anomalies or delays are detected, immediate intervention and adjustments can be made to ensure data real-time and accuracy. Furthermore, to verify the consistency and accuracy of data during the migration process, a data checksum and comparison mechanism can be employed. After the migration is complete, the source and target data are fully compared and verified to ensure complete consistency in content and structure. Through the implementation of this embodiment, the bank transaction system can achieve smooth migration from the source cluster to the target cluster. During the migration process, real-time data synchronization and task consistency are effectively guaranteed, ensuring the continuity and stability of the transaction system.

[0085] The embodiment of the present application can ensure the consistency of the old and new clusters at the task level by obtaining the task information of the first task of the source cluster and accurately creating the corresponding task in the target cluster, effectively avoiding data loss or errors caused by task mismatch. At the same time, a distributed message queue is used to capture and synchronize the real-time data stream generated by the first task in real time, realizing instant transmission and writing of data, significantly reducing data latency, and improving the real-time and accuracy of data. In addition, writing the real-time data stream to the source cluster and the target cluster at the same time not only ensures redundant storage of data, but also facilitates data verification and comparison during the migration or upgrade process, further ensuring the integrity and consistency of the data.

[0086] In some optional implementations of this embodiment, step S203, when the second task is detected, deploying the second task to both the source cluster and the target cluster to obtain source real-time data corresponding to the second task from the source cluster and target real-time data corresponding to the second task from the target cluster, specifically includes the following steps:

[0087] According to the task release process of the data development platform, the metadata information of the second task is obtained; the metadata information of the second task is parsed to obtain the configuration information of the second task on the source cluster and the target cluster; using a distributed task scheduling framework, the configuration information is deployed to the source cluster and the target cluster at the same time to start the second task of the source cluster and the target cluster respectively, obtain the source real-time data of the source cluster, and obtain the target real-time data of the target data cluster.

[0088] In one example, a large bank's real-time risk control system requires synchronizing real-time risk control tasks and data between a source cluster (the legacy risk control system) and a target cluster (the newer risk control system). To ensure real-time data accuracy, the technical solution proposed in this embodiment can be employed. First, metadata for the real-time risk control task (the second task) can be obtained based on the task release process of the data development platform. This metadata, which includes the task's name, type, parameters, trigger conditions, and other information, serves as the basis for task execution. The obtained metadata is then parsed to obtain the configuration information for the real-time risk control task on the source and target clusters. This configuration information, including the task's execution environment, resource allocation, and input and output parameters, is crucial for task deployment and execution on the clusters. Next, a distributed task scheduling framework is used to simultaneously deploy the parsed configuration information to both the source and target clusters. This distributed task scheduling framework ensures parallel execution and real-time scheduling of tasks across multiple clusters, thereby meeting data real-time requirements. After task deployment is complete, the real-time risk control task begins executing on the source and target clusters, generating source and target real-time data, respectively. In addition, to verify the accuracy and consistency of data synchronization, a data verification and comparison mechanism can be used. During task execution, the source real-time data and the target real-time data can be compared and verified to ensure that they are completely consistent in content and structure.

[0089] The embodiment of the present application can accurately obtain the metadata of the second task through the task release process of the data development platform, laying a solid foundation for subsequent data processing. Deep analysis of the metadata can accurately extract the configuration information of the source cluster and the target cluster, ensuring seamless docking of tasks on the two clusters. Using the distributed task scheduling framework, the parallel deployment of configuration information is achieved, and the second tasks of the source cluster and the target cluster are started at the same time, significantly improving the efficiency of data processing. In addition, the solution can also capture the source real-time data of the source cluster and the target real-time data of the target cluster in real time to ensure the timeliness and accuracy of the data.

[0090] In some optional implementations of this embodiment, step S204, based on historical data, source data, target data, source real-time data, and target real-time data, uses a cluster migration module to verify data consistency between the source cluster and the target cluster. Specifically, the following steps are included:

[0091] Obtain target metadata information of the source cluster and the target cluster; compare the table structures of the source cluster and the target cluster based on the target metadata information, historical data, source data, target data, source real-time data, and target real-time data, and determine the data tables that require data consistency verification based on the comparison results; use the cluster migration module to perform a full table scan of the data table, calculate each row of data in the data table using a hash function to obtain a hash value, and sum the hash values ​​to obtain a hash row sum result for each table; compare the hash row sum results of the same data tables in the source cluster and the target cluster; if the hash row sum results are the same, it is determined that the data consistency verification of the source cluster and the target cluster has passed; if the hash row sum results are different, it is determined that the data consistency verification of the source cluster and the target cluster has failed.

[0092] In one example, consider the data migration of a large bank's customer information management system. This system needs to migrate sensitive data, such as basic customer information and transaction records, from a source cluster to a target cluster. To ensure data consistency after migration, the technical solution proposed in this embodiment can be employed. First, target metadata information for the source and target clusters is obtained. This information may include table names, field types, and lengths. Then, based on the target metadata information, historical data, source data, target data, source real-time data, and target real-time data, the table structures of the source and target clusters are compared. This comparison identifies the data tables requiring data consistency verification. Next, the cluster migration module performs a full table scan on these tables. During the scan, a hash function (such as MD5 or SHA-256) is used to calculate a hash value for each row of data. These hash values ​​are then summed to obtain the hash row sum for each table. This step leverages the collision resistance of the hash function to ensure that data remains unchanged during the migration process. Finally, the hash row sums for the same tables in the source and target clusters are compared. If the hash row sum results are the same, the data consistency verification for the source and target clusters has passed, indicating that the data integrity and consistency were maintained during the migration process. If the hash row sum results are different, the data consistency verification has failed, and further investigation is required for possible issues during the data migration process, such as data loss, data corruption, or data synchronization delays. Through the implementation of this embodiment, the bank customer information management system can perform data consistency verification after data migration. This verification process not only ensures data accuracy and consistency, but also improves the reliability and security of data migration.

[0093] The embodiment of the present application can obtain the target metadata information of the source cluster and the target cluster, and comprehensively compare the table structure with the historical data, source data, target data, source real-time data and target real-time data to accurately locate the data table that needs to be verified. Utilizing the full table scan function of the cluster migration module, combined with the collision resistance of the hash function, a unique hash value is generated for each row of data, and the hash row summation result of the table is obtained by summing, effectively avoiding data tampering or loss. Comparing the hash row summation results of the same data table of the source cluster and the target cluster can intuitively reflect the data consistency status. If the results are the same, the verification is passed, indicating that the data migration is accurate. Otherwise, the data inconsistency problem is quickly identified, which is convenient for timely investigation and correction. The solution provided by this embodiment can not only improve the reliability of data migration, but also significantly enhance the security and stability of data processing.

[0094] It should be emphasized that in order to further ensure the privacy and security of the source cluster information of the above-mentioned source cluster, the target cluster information of the target cluster, and the migration strategy, the source cluster information of the above-mentioned source cluster, the target cluster information of the target cluster, and the migration strategy can also be stored in a node of a blockchain.

[0095] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0096] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0097] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0098] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a cluster migration device. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0099] like Figure 3 As shown, the cluster migration device 400 of this embodiment includes: an acquisition module 401, a migration module 402, a deployment module 403, a verification module 404 and a switching module 405. Among them:

[0100] An acquisition module 401 is configured to acquire source cluster information of a source cluster, target cluster information of a target cluster, and a migration strategy;

[0101] Migration module 402 is configured to migrate the historical data of the source cluster to the target cluster using the data synchronization service based on the source cluster information, the target cluster information, and the migration strategy, and to synchronize the first task of the source cluster to the target cluster using the task migration module, thereby obtaining the source data corresponding to the first task from the source cluster and the target data corresponding to the first task from the target cluster.

[0102] A deployment module 403 is configured to, when a second task is detected, simultaneously deploy the second task to a source cluster and a target cluster, so as to obtain source real-time data corresponding to the second task from the source cluster and target real-time data corresponding to the second task from the target cluster;

[0103] Verification module 404, configured to verify data consistency between the source cluster and the target cluster using the cluster migration module based on historical data, source data, target data, source real-time data, and target real-time data;

[0104] The switching module 405 is used to switch the database task synchronization source of the target cluster from the source cluster to the database of the target cluster if the data consistency verification passes, and obtain the operating indicator data of the target cluster. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

[0105] The embodiment of the present application can realize intelligent data synchronization and task migration of the source cluster and the target cluster through refined management and efficient migration strategies. First, the seamless migration of historical data is ensured by the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenges of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after migration. Finally, after the data consistency is verified and the target cluster operating indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

[0106] In one embodiment, the migration module 402 includes:

[0107] Determine the submodule, which is used to determine the data lake tasks of the target cluster based on the source cluster information, target cluster information, and migration strategy;

[0108] The first acquisition submodule is used to obtain the data table and field information corresponding to the source cluster based on the data lake task;

[0109] The synchronization submodule is used to determine the historical data that needs to be synchronized in the source cluster based on data table and field information, and use data migration instructions to synchronize the historical data to the target cluster.

[0110] The embodiment of the present application can accurately determine the data lake tasks of the target cluster by comprehensively considering the source cluster information, target cluster information and migration strategy, ensuring that the purpose of data migration is clear and meets business needs. On this basis, the data table and field information corresponding to the source cluster is accurately obtained, providing strong support for the screening and synchronization of historical data. At the same time, the historical data that needs to be synchronized is determined based on the data table and field information, and data migration instructions are used for synchronization, which not only ensures the integrity and accuracy of the data, but also significantly improves the degree of automation of data migration.

[0111] In one embodiment, the migration module 402 includes:

[0112] A second acquisition submodule is configured to acquire task information of the first task of the source cluster and create a corresponding first task in the target cluster based on the task information;

[0113] The writing submodule is used to obtain the real-time data stream generated for the first task, and use a distributed message queue to write the real-time data stream to the source cluster and the target cluster respectively to obtain the source data corresponding to the source cluster and the target data corresponding to the target cluster.

[0114] The embodiment of the present application can ensure the consistency of the old and new clusters at the task level by obtaining the task information of the first task of the source cluster and accurately creating the corresponding task in the target cluster, effectively avoiding data loss or errors caused by task mismatch. At the same time, a distributed message queue is used to capture and synchronize the real-time data stream generated by the first task in real time, realizing instant transmission and writing of data, significantly reducing data latency, and improving the real-time and accuracy of data. In addition, writing the real-time data stream to the source cluster and the target cluster at the same time not only ensures redundant storage of data, but also facilitates data verification and comparison during the migration or upgrade process, further ensuring the integrity and consistency of the data.

[0115] In one embodiment, the deployment module 403 includes:

[0116] A second acquisition submodule is used to obtain metadata information of the second task according to the task release process of the data development platform;

[0117] A parsing submodule, configured to parse the metadata information of the second task to obtain configuration information of the second task on the source cluster and the target cluster;

[0118] The deployment submodule is used to deploy the configuration information to the source cluster and the target cluster simultaneously using a distributed task scheduling framework to start the second task of the source cluster and the target cluster respectively, obtain the source real-time data of the source cluster, and obtain the target real-time data of the target data cluster.

[0119] The embodiment of the present application can accurately obtain the metadata of the second task through the task release process of the data development platform, laying a solid foundation for subsequent data processing. Deep analysis of the metadata can accurately extract the configuration information of the source cluster and the target cluster, ensuring seamless docking of tasks on the two clusters. Using the distributed task scheduling framework, the parallel deployment of configuration information is achieved, and the second tasks of the source cluster and the target cluster are started at the same time, significantly improving the efficiency of data processing. In addition, the solution can also capture the source real-time data of the source cluster and the target real-time data of the target cluster in real time to ensure the timeliness and accuracy of the data.

[0120] In one embodiment, the verification module 404 includes:

[0121] The third acquisition submodule is used to obtain target metadata information of the source cluster and the target cluster;

[0122] The comparison submodule is used to compare the table structures of the source cluster and the target cluster based on the target metadata information, historical data, source data, target data, source real-time data, and target real-time data. Based on the comparison results, it determines the data tables that need to be verified for data consistency.

[0123] The scanning submodule is used to use the cluster migration module to perform a full table scan on the data table, calculate each row of data in the data table using a hash function to obtain a hash value, and sum the hash values ​​to obtain the hash row sum result of each table;

[0124] The comparison submodule is used to compare the hash row sum results of the same data table in the source cluster and the target cluster;

[0125] The determination submodule is used to determine that the data consistency verification of the source cluster and the target cluster has passed if the hash row sum results are the same, and to determine that the data consistency verification of the source cluster and the target cluster has failed if the hash row sum results are different.

[0126] The embodiment of the present application can obtain the target metadata information of the source cluster and the target cluster, and comprehensively compare the table structure with the historical data, source data, target data, source real-time data and target real-time data to accurately locate the data table that needs to be verified. Utilizing the full table scan function of the cluster migration module, combined with the collision resistance of the hash function, a unique hash value is generated for each row of data, and the hash row summation result of the table is obtained by summing, effectively avoiding data tampering or loss. Comparing the hash row summation results of the same data table of the source cluster and the target cluster can intuitively reflect the data consistency status. If the results are the same, the verification is passed, indicating that the data migration is accurate; otherwise, the data inconsistency problem is quickly identified, which is convenient for timely investigation and correction. The solution provided by this embodiment can not only improve the reliability of data migration, but also significantly enhance the security and stability of data processing.

[0127] In one embodiment, the cluster migration apparatus further includes:

[0128] The information acquisition module is used to obtain the data architecture information of the source cluster, including data model information, data volume, and data distribution information;

[0129] The construction module is used to determine the migration strategy and the architecture design information of the target cluster according to the data model information, data volume and data distribution information, and to build the target cluster based on the architecture design information.

[0130] The embodiments of the present application can provide a solid data foundation for the formulation of migration strategies by comprehensively acquiring the data architecture information of the source cluster, including detailed data model information, accurate data volume statistics, and in-depth data distribution analysis. This helps to ensure the integrity of the data structure and the accuracy of the data volume during the migration process, and avoid data loss or omissions. At the same time, the migration strategy formulated based on these data architecture information can fully consider the characteristics of the data and business needs, and achieve efficient data migration. In addition, the target cluster architecture design based on the data architecture information can ensure that the target cluster meets business needs in terms of performance, capacity, and scalability, and provide strong support for subsequent data processing and business applications.

[0131] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0132] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 with a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0133] Computer devices can be desktop computers, laptops, PDAs, cloud servers, etc. Computer devices can interact with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0134] Memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, memory 61 may be an internal storage unit of computer device 6, such as the hard disk or memory of computer device 6. In other embodiments, memory 61 may also be an external storage device of computer device 6, such as a plug-in hard disk, smart memory card (SMC), secure digital (SD) card, flash memory card, etc. equipped on computer device 6. Of course, memory 61 may also include both internal storage units and external storage devices of computer device 6. In this embodiment, memory 61 is generally used to store the operating system and various application software installed on computer device 6, such as computer-readable instructions for the cluster migration method. In addition, memory 61 may also be used to temporarily store various types of data that have been output or are about to be output.

[0135] In some embodiments, processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. Processor 62 is generally used to control the overall operation of computer device 6. In this embodiment, processor 62 is used to execute computer-readable instructions stored in memory 61 or process data, such as computer-readable instructions for executing a cluster migration method.

[0136] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0137] The embodiment of the present application can realize intelligent data synchronization and task migration of the source cluster and the target cluster through refined management and efficient migration strategies. First, the seamless migration of historical data is ensured by the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenges of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after migration. Finally, after the data consistency is verified and the target cluster operating indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

[0138] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions. The computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the cluster migration method as described above.

[0139] The embodiment of the present application can realize intelligent data synchronization and task migration of the source cluster and the target cluster through refined management and efficient migration strategies. First, the seamless migration of historical data is ensured by the data synchronization service, and the task migration module ensures the synchronization of tasks and data, laying the foundation for subsequent data consistency verification. Innovatively, for the second task, dual clusters are deployed in parallel to synchronize the real-time data of the source cluster and the target cluster in real time, effectively responding to the challenges of real-time data migration. The data consistency verification of the cluster migration module ensures the accuracy of the data before and after migration. Finally, after the data consistency is verified and the target cluster operating indicators meet the standards, the database task synchronization source is smoothly switched and the source cluster task scheduling is stopped, realizing cluster migration without business interruption. This solution not only improves the migration efficiency, but also ensures data consistency and business continuity during the migration process, providing an efficient and reliable solution for big data cluster migration. It can achieve synchronous migration of historical data and real-time data during cluster migration to ensure data integrity.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods of each embodiment of the present application.

[0141] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0142] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A cluster migration method, characterized in that: The steps include: Obtain the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy; Based on the source cluster information, the target cluster information, and the migration strategy, a data synchronization service is used to migrate the historical data of the source cluster to the target cluster. A task migration module is used to synchronize the first task of the source cluster to the target cluster, so as to obtain the source data corresponding to the first task from the source cluster and the target data corresponding to the first task from the target cluster. When a second task is detected, the second task is deployed to the source cluster and the target cluster simultaneously, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster; Based on the historical data, the source data, the target data, the source real-time data, and the target real-time data, using a cluster migration module to perform data consistency verification on the source cluster and the target cluster; If the data consistency verification passes, the database task synchronization source of the target cluster is switched from the source cluster to the database of the target cluster, and the operating indicator data of the target cluster is obtained. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

2. The method according to claim 1, characterized in that Before the step of obtaining source cluster information of the source cluster, target cluster information of the target cluster, and the migration strategy, the method further includes: Obtaining data architecture information of the source cluster, wherein the data architecture information includes data mode information, data volume, and data distribution information; A migration strategy and architecture design information of the target cluster are determined according to the data pattern information, the data volume, and the data distribution information, and the target cluster is constructed based on the architecture design information.

3. The method according to claim 2, characterized in that The step of migrating the historical data of the source cluster to the target cluster using a data synchronization service based on the source cluster information, the target cluster information, and the migration strategy specifically includes: Determine a data lake task for the target cluster based on the source cluster information, the target cluster information, and the migration strategy; Based on the data lake task, obtain the data table and field information corresponding to the source cluster; Based on the data table and field information, historical data that needs to be synchronized in the source cluster is determined, and data migration instructions are used to synchronize the historical data to the target cluster.

4. The method according to claim 1, wherein The step of using the task migration module to synchronize the first task of the source cluster to the target cluster to obtain source data corresponding to the first task from the source cluster and to obtain target data corresponding to the first task from the target cluster specifically includes: Acquire task information of a first task of the source cluster, and create a corresponding first task in the target cluster based on the task information; A real-time data stream generated for the first task is obtained, and the real-time data stream is written to the source cluster and the target cluster respectively using a distributed message queue to obtain source data corresponding to the source cluster and target data corresponding to the target cluster.

5. The method according to claim 1, wherein The step of simultaneously deploying the second task to the source cluster and the target cluster when the second task is detected, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster, specifically includes: Obtain metadata information for the second task according to the task release process of the data development platform; Parsing metadata information of the second task to obtain configuration information of the second task on the source cluster and the target cluster; A distributed task scheduling framework is used to simultaneously deploy the configuration information to the source cluster and the target cluster to start the second task of the source cluster and the target cluster respectively, obtain the source real-time data of the source cluster, and obtain the target real-time data of the target data cluster.

6. The method according to claim 1, characterized in that The step of using a cluster migration module to perform data consistency verification on the source cluster and the target cluster based on the historical data, the source data, the target data, the source real-time data, and the target real-time data specifically includes: Get the target metadata information of the source cluster and the target cluster; Comparing the table structures of the source cluster and the target cluster based on the target metadata information, the historical data, the source data, the target data, the source real-time data, and the target real-time data, and determining, based on the comparison results, a data table requiring data consistency verification; Using a cluster migration module, performing a full table scan on the data table, calculating each row of data in the data table using a hash function to obtain a hash value, and summing the hash values ​​to obtain a hash row sum result for each table; Comparing hash row sum results of the same data table in the source cluster and the target cluster; If the hash row sum results are the same, it is determined that the data consistency verification of the source cluster and the target cluster has passed; if the hash row sum results are different, it is determined that the data consistency verification of the source cluster and the target cluster has failed.

7. A cluster migration device, characterized in that: include: The acquisition module is used to obtain the source cluster information of the source cluster, the target cluster information of the target cluster, and the migration strategy; A migration module is configured to migrate the historical data of the source cluster to the target cluster using a data synchronization service based on the source cluster information, the target cluster information, and the migration strategy, and to synchronize the first task of the source cluster to the target cluster using a task migration module, so as to obtain the source data corresponding to the first task from the source cluster and the target data corresponding to the first task from the target cluster; a deployment module, configured to, when a second task is detected, simultaneously deploy the second task to the source cluster and the target cluster, so as to obtain source real-time data corresponding to the second task from the source cluster and obtain target real-time data corresponding to the second task from the target cluster; a verification module, configured to perform data consistency verification on the source cluster and the target cluster using a cluster migration module based on the historical data, the source data, the target data, the source real-time data, and the target real-time data; The switching module is used to switch the database task synchronization source of the target cluster from the source cluster to the database of the target cluster if the data consistency verification passes, and obtain the operating indicator data of the target cluster. When the operating indicator data meets the preset requirements, the task scheduling of the source cluster is stopped to complete the cluster migration.

8. The device according to claim 7, characterized in that The migration module includes: a determination submodule, configured to determine the data lake task of the target cluster based on the source cluster information, the target cluster information, and the migration strategy; A first acquisition submodule is configured to acquire data tables and field information corresponding to the source cluster based on the data lake task; The synchronization submodule is used to determine the historical data that needs to be synchronized in the source cluster based on the data table and field information, and synchronize the historical data to the target cluster using data migration instructions.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the cluster migration method according to any one of claims 1 to 6 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the cluster migration method according to any one of claims 1 to 6.

Images