Data verification method and device for big data cluster migration, equipment and medium

Abstract

The invention relates to the field of data processing, and provides a data verification method and device for big data cluster migration, equipment and a medium, the verification method comprises the following steps: newly establishing a target data cluster on an original data cluster, preprocessing the target data cluster, and marking the preprocessed target data cluster to obtain an ODS synchronization task; the method comprises the following steps: identifying an import command in an ODS synchronization task, capturing incremental change table data from an original data cluster by a target data cluster according to an identification result until the data volumes of current storage data of the target data cluster and the original data cluster are the same, and then performing synchronization and merging to obtain two merged tables. According to the embodiment, the data of the two clusters can be kept consistent through the imported incremental change table data, and the problem that migration data are inconsistent is effectively avoided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data verification method, device, equipment and medium for big data cluster migration. Background Art

[0002] With the rapid development and popularization of big data technology, the scale of big data clusters is expanding, and the demand for cross-city big data cluster relocation is also increasing. During the relocation process, it is necessary to ensure that the big data cluster does not stop running to maintain business continuity and stability.

[0003] In order to achieve smooth switching of big data clusters, the existing technology builds a new big data cluster and realizes parallel operation with the original big data cluster. During the parallel process, the data of the new big data cluster needs to be synchronized with the data of the original big data cluster.

[0004] The existing synchronization solutions mainly include the one-read-two-write solution, the two-run solution, and the task splitting solution. For the one-read-two-write solution, although the solution can realize the synchronous update of the data of the two clusters, it requires a lot of manpower to transform the data synchronization program of the original big data cluster, and the transformation process may have an adverse effect on the stability of the original big data cluster. In addition, there is a bandwidth performance bottleneck problem when writing a large amount of data from the original big data cluster to the new big data cluster; for the two-run solution, the original big data cluster and the new big data cluster are allowed to retrieve data from the source library respectively without affecting each other. However, this solution will cause inconsistency in the ODS layer data due to the difference in synchronization time points, which will in turn affect the consistency of the data results processed by the downstream programs of the ODS layer under the two clusters. This not only requires a lot of manpower to check the differences, but also affects the relocation results; for the task splitting solution, by splitting the import action and merge action in the original big data cluster and the new big data cluster, only one-read-two-write is performed for the imported incremental change data after the split. This solution also requires a lot of manpower to split the synchronization program and workflow, and may have an adverse effect on the program stability of the original big data cluster. Summary of the invention

[0005] The embodiments of the present invention provide a data verification method, device, equipment and medium for big data cluster migration, aiming to solve the problem that existing synchronization solutions affect data consistency.

[0006] In a first aspect, an embodiment of the present invention provides a data verification method for big data cluster migration, comprising:

[0007] Based on the original data cluster, create a target data cluster, and preprocess the target data cluster;

[0008] Mark the preprocessed target data cluster to obtain the ODS synchronization task;

[0009] Identify the import command in the ODS synchronization task, and if the identification is successful, fetch the incremental change table data from the original data cluster to the target data cluster;

[0010] Determine whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume. If they are the same data volume, synchronize and merge the two storage data and the incremental change table data on the corresponding data clusters to obtain two merged tables.

[0011] Verify the data consistency of the two merged tables.

[0012] In a second aspect, an embodiment of the present invention provides a data verification device based on big data cluster migration, comprising:

[0013] A preprocessing unit, used to create a target data cluster based on the original data cluster, and preprocess the target data cluster;

[0014] A marking unit is used to mark the preprocessed target data cluster to obtain an ODS synchronization task;

[0015] an identification unit, used for identifying the import command in the ODS synchronization task, and if the identification is successful, fetching the incremental change table data from the original data cluster to the target data cluster;

[0016] A merging unit, used to determine whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume, and if they are the same data volume, synchronize and merge the two storage data and the incremental change table data on the corresponding data clusters to obtain two merged tables;

[0017] A verification unit is used to verify the data consistency of the two merged tables.

[0018] In a third aspect, an embodiment of the present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data verification method as described above when executing the computer program.

[0019] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the data verification method as described above is implemented.

[0020] The embodiment of the present invention provides a data verification method, device, equipment and medium for big data cluster migration. The verification method creates a new target data cluster on the original data cluster, preprocesses the target data cluster, and then marks the preprocessed target data cluster to obtain an ODS synchronization task. By identifying the import command in the ODS synchronization task, the target data cluster grabs the incremental change table data from the original data cluster according to the identification result, until the data volume of the current storage data of the target data cluster and the original data cluster is the same, and then synchronizes and merges to obtain two merged tables. The incremental change table data imported by this embodiment enables the data of the two clusters to be consistent, effectively avoiding the problem of inconsistent migration data, and the imported incremental change table data is smaller than the full data after the merger. At this time, it is only necessary to grab the incremental change table data from the original data cluster, and there is no need to copy the full data every day, which greatly reduces the bandwidth bottleneck of cross-city data migration. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of an application environment of a data verification method for big data cluster migration in one embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a process of a data verification method for big data cluster migration in one embodiment of the present invention;

[0024] Figure 3 for Figure 2 A schematic flow chart of a specific implementation of step S10;

[0025] Figure 4 for Figure 2 A schematic flow chart of a specific implementation of step S20;

[0026] Figure 5 for Figure 2 A schematic flow chart of a specific implementation of step S30;

[0027] Figure 6 for Figure 2 A schematic flow chart of a specific implementation of step S50;

[0028] Figure 7 A schematic diagram of a process of a data verification method for big data cluster migration in another embodiment of the present invention;

[0029] Figure 8 A schematic diagram of the structure of a data verification device based on big data cluster migration in one embodiment of the present invention;

[0030] Fig. 9 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0031] Fig.10 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0034] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0035] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0036] The data verification method for big data cluster migration provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, wherein the client communicates with the server through a network, the server can create a target data cluster based on the original data cluster, and preprocess the target data cluster; mark the preprocessed target data cluster to obtain an ODS synchronization task; identify the import command in the ODS synchronization task, and if the identification is successful, grab the incremental change table data from the original data cluster to the target data cluster; determine whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume, and if they are the same data volume, synchronize and merge the two storage data and the incremental change table data on the corresponding data clusters respectively to obtain two merged tables; verify the data consistency of the two merged tables. In the present invention, for complex insurance entities under the insurance business, a new data cluster (i.e., target data cluster) is created outside the original data cluster, and the target data cluster is preprocessed, marked, identified, synchronized, and merged, thereby realizing data migration between the two clusters, and the two clusters after the data migration are completed are verified to determine the consistency of the data. In this migration process, the data of the two clusters can be kept consistent, effectively avoiding the problem of inconsistent migration data, and the imported incremental change table data is smaller than the full data after the merger. At this time, it is only necessary to grab the incremental change table data from the original data cluster, without copying the full data every day, which greatly reduces the bandwidth bottleneck of cross-city data migration. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0037] See also Figure 2 , an embodiment of the present invention provides a data verification method for big data cluster migration, including S10-S50:

[0038] S10, creating a target data cluster based on the original data cluster, and preprocessing the target data cluster;

[0039] In this step, data migration needs to be performed between two data clusters. In addition to the original data cluster (S_Cluster), a target data cluster (T_Cluster) needs to be created and preprocessed to unify the environment, programs, and data of the target data cluster with those of the original data cluster, thereby reducing the asynchrony of the migrated data caused by differences in various factors.

[0040] In one embodiment, if Figure 3 As shown, S10 includes:

[0041] S11, obtaining environment information, configuration information and data information of the original data cluster;

[0042] S12: Create a target data cluster based on the environment information, and copy the configuration information and data information to the target data cluster.

[0043] This embodiment is a preprocessing process of the target data cluster. In order to maintain consistency with the environment of the original data cluster, it is necessary to obtain the environmental information of the original data cluster. Both the original data cluster and the target data cluster include HADOOP components and HIVE components. Therefore, the environmental information obtained is the version of the HADOOP components and HIVE components in the original data cluster. The versions of the HADOOP components and HIVE components in the target data cluster are then kept consistent with the versions of the HADOOP components and HIVE components in the original data cluster, effectively preventing the risk of migration switching due to different versions.

[0044] In order to keep the program consistent with the original data cluster, it is also necessary to obtain the configuration information of the original data cluster, which includes the ODS synchronization program, table structure, permissions, and task configuration, and then copy the configuration information to the target data cluster to effectively prevent program inconsistencies from affecting the migration process. In the specific implementation, since the table structure, permissions, and task configuration information are stored in the database, the corresponding information in the database can be directly copied to the target cluster database. For script programs (such as ODS synchronization programs) that are not in the database, they are transferred to the machine of the target data cluster through files.

[0045] In order to maintain data consistency with the original data cluster, it is also necessary to obtain the data information of the original data cluster, that is, to initialize the ODS table data of the original data cluster to the target data cluster. The initialization principle is: use the proprietary tool of the DISTCP command (distributed copy command) to copy files and repair data to achieve the purpose of quickly initializing data and effectively prevent data inconsistency from affecting the migration process.

[0046] S20, marking the preprocessed target data cluster to obtain an ODS synchronization task;

[0047] In this step, the tasks in the preprocessed target data cluster need to be marked so that the system can identify these marked tasks as ODS (Operational Data Store) synchronization tasks. ODS synchronization tasks are used for subsequent processing of incremental data changes and merging actions.

[0048] In one embodiment, if Figure 4 As shown, S20 includes:

[0049] S21, obtaining tasks in the preprocessed target data cluster;

[0050] S22. Add the configuration record of the task to the configuration table to obtain the ODS synchronization task.

[0051] In this embodiment, there are a large number of tasks in the target data cluster, but not all tasks need to be processed. Therefore, all tasks in the target data cluster after preprocessing are first obtained, and the target tasks that need to be processed are selected from all tasks, and the target tasks are marked. The marking method is: the task configuration record of the target task is added to the configuration table, that is, the tasks in the configuration table are deemed to be marked successfully. The tasks that are marked successfully are ODS synchronization tasks. In specific implementation, the task scheduling platform will identify whether the task is in the configuration table when starting the task.

[0052] S30, identifying the import command in the ODS synchronization task, and if the identification is successful, fetching the incremental change table data from the original data cluster to the target data cluster;

[0053] In the specific implementation, the original data cluster and the target data cluster both grab the incremental change table data from the business source library before the migration starts. In this step, the ODS synchronization task contains the Sqoop import command (i.e., the import command). During the migration process, it is necessary to identify the import command in the ODS synchronization task. If the identification is successful, the target data cluster grabs the incremental change table data from the original data cluster, and the original data cluster still grabs the incremental change table data from the business source library; if the identification is not successful, the target data cluster and the original data cluster still grab the incremental change table data from the business source library, that is, when the import command in the ODS synchronization task is identified, the target data cluster will switch from the business source library to the original data cluster to grab the incremental change table data until the target data cluster migration is completed, and the original data cluster stops running. This migration method improves the migration rate.

[0054] In a specific embodiment, Figure 5 As shown, you can identify the import command in the ODS synchronization task as follows:

[0055] S31, converting the import command into a Java command;

[0056] S32, using a recognition function to identify whether the Java command contains the import keyword;

[0057] S33: If the import keyword is identified, it is determined that the identification is successful, and the incremental change table data is captured from the original data cluster to the target data cluster.

[0058] The recognition function may adopt a string matching function, that is, by checking whether the string of the Java command contains a specific substring (import), if so, the recognition is determined to be successful. This embodiment can quickly identify the ODS synchronization task through this recognition method, thereby improving the efficiency of subsequent processes.

[0059] S40, determining whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume; if they are the same data volume, synchronizing and merging the two storage data and the incremental change table data on the corresponding data clusters respectively to obtain two merged tables;

[0060] In this step, when the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume, it means that the two clusters have the same data baseline before synchronization. Then, the data stored in each cluster and the incremental change table data captured by each cluster are synchronized and merged respectively to ensure that the data changes of the two clusters can be synchronized in real time within the same time period, and finally merged into a corresponding unified data set (i.e., merged table). In the specific implementation, the amount of the current storage data of the two clusters is completely the same or consistent at a certain time point (which can be called before T-1 day or at the end of T-1 day). At this time, the task of data synchronization and merging is started. The goal of this task is to ensure that the data changes of the two clusters can be synchronized in real time or regularly from T-1 day to T day, and finally merged into a unified data set.

[0061] Among them, the current storage data of the two clusters can be the merged table data of T-1 day, that is, at the end of T-1 day, the two clusters have exactly the same merged table data, which serves as the starting point for data synchronization and merging on T day. The incremental change table data can be the incremental change table data of T day, that is, from T-1 day to T day, the two clusters capture the same incremental change table data, and then merge the corresponding incremental change table data into the corresponding merged table data of T-1 day, to obtain two merged tables of T day. The data in these two merged tables are the same, thereby ensuring data consistency and integrity.

[0062] In one embodiment, the data verification method further includes:

[0063] If the current storage data of the original data cluster and the current storage data of the target data cluster are different in data volume, the target data cluster fetches the incremental change table data from the business source library.

[0064] In this embodiment, since the original data cluster and the target data cluster both capture the incremental change table data from the business source library when no migration or synchronization is performed, when the current storage data of the original data cluster and the current storage data of the target data cluster are different data volumes, it indicates that the synchronization task cannot be performed at this time. Therefore, the target data cluster still captures the incremental change table data from the business source library. The synchronization task can only be started when the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume, and the target data cluster can be switched from the business source library to the original data cluster to capture the incremental change table data.

[0065] S50: Verify the data consistency of the two merged tables.

[0066] In this step, in order to determine whether the data migration between the two clusters is successful, the data consistency of the two merged tables needs to be verified. During the implementation process, if the ODS downstream programs of the two clusters are normal, the data synchronized by the ODS synchronization task is the same, and the data of the two merged tables obtained are consistent (the same). If the ODS downstream program is abnormal, it means that the data synchronized by the ODS synchronization task is different. At this time, the problem can be quickly discovered and handled accordingly.

[0067] In one embodiment, if Figure 6 As shown, the data consistency of the two merged tables is verified as follows:

[0068] S51, determining whether the data amounts of the two merged tables are consistent;

[0069] S52, if they are consistent, respectively calculating the hash values ​​of each row and column in the two merged tables;

[0070] S53, comparing all corresponding hash values ​​in the two merged tables to see if they are consistent. If they are consistent, it indicates that the data in the two merged tables are consistent.

[0071] When the data in the two merged tables are verified to be consistent, the migration is successful.

[0072] In one embodiment, if Figure 7 As shown, the data verification method also includes:

[0073] S60: Unmark the ODS synchronization task, switch the target data cluster back to the business source database, and capture the incremental change table data.

[0074] In this embodiment, after the verification is completed, it is necessary to ensure that the data of the two merged tables are consistent, that is, it is necessary to ensure that the downstream program is normal, and then cancel the mark of the ODS synchronization task and stop the migration. At this time, the original data cluster is offline, the target data cluster switches back to the business source library, and captures the incremental change table data. If the data of the two merged tables are consistent, that is, the downstream program is abnormal, manual verification is required. If the manual verification is still abnormal, all repairs are performed. If the manual verification believes that it is caused by normal circumstances, it is regarded as normal. The mark of the ODS synchronization task will be canceled when the downstream program is normal.

[0075] It can be seen that the data verification method implemented by the present invention does not require task changes to the ODS synchronization task, which greatly saves the manpower required for transformation and ensures the stability of the cluster program. Moreover, the incremental change table data imported by the import command can remain consistent, which greatly reduces the large amount of data verification manpower required due to data inconsistency. In addition, the amount of data in the imported incremental change table is smaller than the full amount of data after the merger. At this time, it is only necessary to capture the incremental change table data from the original data cluster, and there is no need to copy the full amount of data every day, which greatly reduces the cross-city bandwidth bottleneck. In addition, by marking the ODS synchronization task, the location of the captured data can be quickly switched.

[0076] The embodiment of the present invention also provides a data verification device for big data cluster migration, which is used to execute any embodiment of the data verification method for big data cluster migration. Figure 8 , Figure 8 1 is a schematic block diagram of a data verification device for big data cluster migration provided by an embodiment of the present invention. The data verification device 700 for big data cluster migration includes:

[0077] A preprocessing unit 710 is used to create a target data cluster based on the original data cluster and preprocess the target data cluster;

[0078] A marking unit 720 is used to mark the pre-processed target data cluster to obtain an ODS synchronization task;

[0079] The identification unit 730 is used to identify the import command in the ODS synchronization task, and if the identification is successful, to capture the incremental change table data from the original data cluster to the target data cluster;

[0080] The merging unit 740 is used to determine whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume. If they are the same data volume, the two storage data and the incremental change table data are synchronized and merged on the corresponding data clusters to obtain two merged tables.

[0081] The verification unit 750 is used to verify the data consistency of the two merged tables.

[0082] In one embodiment, the preprocessing unit 710 is specifically configured to:

[0083] Obtain the environment information, configuration information, and data information of the original data cluster;

[0084] Based on the environment information, a target data cluster is created, and the configuration information and data information are copied to the target data cluster.

[0085] In one embodiment, the marking unit 720 is specifically configured to:

[0086] Obtain tasks in the preprocessed target data cluster;

[0087] Add the configuration record of the task to the configuration table to obtain the ODS synchronization task.

[0088] In one embodiment, the identification unit 730 is specifically configured to:

[0089] Convert import commands into Java commands;

[0090] Using a recognition function to identify whether the Java command contains an import keyword;

[0091] If the import keyword is identified, it is determined that the identification is successful, and the incremental change table data is captured from the original data cluster to the target data cluster.

[0092] In one embodiment, the verification unit 750 is specifically configured to:

[0093] Determine whether the data amounts of the two merged tables are consistent;

[0094] If they are consistent, the hash values ​​of each row and column in the two merged tables are calculated respectively;

[0095] Compare all corresponding hash values ​​in the two merged tables to see if they are consistent. If they are consistent, it indicates that the data in the two merged tables are consistent.

[0096] The verification device also includes:

[0097] The fetching unit is configured to, if the current storage data of the original data cluster and the current storage data of the target data cluster are different in data volume, cause the target data cluster to fetch the incremental change table data from the business source library.

[0098] The verification device also includes:

[0099] The mark cancellation unit is used to cancel the mark of the ODS synchronization task, switch the target data cluster back to the business source database, and capture the incremental change table data.

[0100] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the server side of a data verification method for migration of a large data cluster.

[0101] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a data verification method for big data cluster migration.

[0102] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0103] Based on the original data cluster, create a target data cluster, and preprocess the target data cluster;

[0104] Mark the preprocessed target data cluster to obtain the ODS synchronization task;

[0105] Identify the import command in the ODS synchronization task, and if the identification is successful, fetch the incremental change table data from the original data cluster to the target data cluster;

[0106] Determine whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume. If they are the same data volume, synchronize and merge the two storage data and the incremental change table data on the corresponding data clusters to obtain two merged tables.

[0107] Verify the data consistency of the two merged tables.

[0108] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0109] Mark the preprocessed target data cluster to obtain the ODS synchronization task;

[0110] Identify the import command in the ODS synchronization task, and if the identification is successful, fetch the incremental change table data from the original data cluster to the target data cluster;

[0111] Determine whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume. If they are the same data volume, synchronize and merge the two storage data and the incremental change table data on the corresponding data clusters to obtain two merged tables.

[0112] Verify the data consistency of the two merged tables.

[0113] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0114] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0115] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0116] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A data verification method for big data cluster migration, characterized in that: include: Based on the original data cluster, create a target data cluster, and preprocess the target data cluster; Mark the preprocessed target data cluster to obtain the ODS synchronization task; Identify the import command in the ODS synchronization task, and if the identification is successful, fetch the incremental change table data from the original data cluster to the target data cluster; Determine whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume. If they are the same data volume, synchronize and merge the two storage data and the incremental change table data on the corresponding data clusters to obtain two merged tables. Verify the data consistency of the two merged tables.

2. The data verification method according to claim 1, characterized in that: The step of creating a target data cluster based on the original data cluster and preprocessing the target data cluster includes: Obtain the environment information, configuration information, and data information of the original data cluster; Based on the environment information, a target data cluster is created, and the configuration information and data information are copied to the target data cluster.

3. The data verification method according to claim 1, characterized in that: The step of marking the preprocessed target data cluster to obtain an ODS synchronization task includes: Obtain tasks in the preprocessed target data cluster; Add the configuration record of the task to the configuration table to obtain the ODS synchronization task.

4. The data verification method according to claim 1, characterized in that: The identifying the import command in the ODS synchronization task, if the identification is successful, fetching the incremental change table data from the original data cluster to the target data cluster, includes: Convert import commands into Java commands; Using a recognition function to identify whether the Java command contains an import keyword; If the import keyword is identified, it is determined that the identification is successful, and the incremental change table data is captured from the original data cluster to the target data cluster.

5. The data verification method according to claim 1, characterized in that: Also includes: If the current storage data of the original data cluster and the current storage data of the target data cluster are different in data volume, the target data cluster fetches the incremental change table data from the business source library.

6. The data verification method according to claim 1, characterized in that: The verifying the data consistency of the two merged tables includes: Determine whether the data amounts of the two merged tables are consistent; If they are consistent, the hash values ​​of each row and column in the two merged tables are calculated respectively; Compare all corresponding hash values ​​in the two merged tables to see if they are consistent. If they are consistent, it indicates that the data in the two merged tables are consistent.

7. The data verification method according to claim 1, characterized in that: Also includes: The ODS synchronization task is unmarked, the target data cluster is switched back to the business source database, and the incremental change table data is captured.

8. A data verification device for big data cluster migration, used to implement the data verification method according to any one of claims 1 to 7, characterized in that: include: A preprocessing unit, used to create a target data cluster based on the original data cluster, and preprocess the target data cluster; A marking unit is used to mark the preprocessed target data cluster to obtain an ODS synchronization task; an identification unit, used for identifying the import command in the ODS synchronization task, and if the identification is successful, fetching the incremental change table data from the original data cluster to the target data cluster; A merging unit, used to determine whether the current storage data of the original data cluster and the current storage data of the target data cluster are the same data volume, and if they are the same data volume, synchronize and merge the two storage data and the incremental change table data on the corresponding data clusters to obtain two merged tables; A verification unit is used to verify the data consistency of the two merged tables.

9. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data verification method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data verification method according to any one of claims 1 to 7 is implemented.

Images