ETL Processing Requirement Migration and Data Verification Method and System for Cross-Cluster Versions

Abstract

The present invention provides a method and system for migrating ETL processing requirements and validating data across cluster versions, which transforms the logic code of the original requirements on an old version of a big data cluster into logic code that can be executed on a new version of a big data cluster with new features; performs MD5 transcoding on the result data obtained from running the logic code of the transformed requirements on the new version of the big data cluster, and performs MD5 transcoding on the result data obtained from running the requirement logic code on the old version of the big data cluster as comparison data; conducts cross-platform communication for the comparison job, starts an automatic transmission job flow to transmit the MD5 data to be compared; compares the MD5 data of the new and old clusters, automatically validates the total number of data and the data content, and outputs the differential data to complete the field-level comparison analysis. The present invention efficiently and labor-savingly completes the transformation and migration requirements across big data clusters and versions, reducing the cycle of interface migration development and verification and the time for parallel operation of interface services.

Description

Technical Field

[0001] The present invention relates to the technical field of requirement transformation and migration, and specifically, to a method and system for ETL processing requirement migration and data verification across cluster versions, specifically a method and system for ETL processing requirement migration and data verification across big data clusters and across versions. In particular, it is a method and system for ETL processing requirement migration and data verification based on different big data cluster versions. Background Art

[0002] Patent document CN107704597A discloses a method for creating an ETL script from a relational database to Hive. The method includes the following steps: obtaining database metadata information; quickly identifying the feature classification of each data table in the relational database; automatically creating an ETL script for each data table according to the classification to which it belongs; and registering the generated ETL script into the scheduling platform through a simulation access technology or a service call method.

[0003] What this patent document solves is the scenario of automatically mapping the data of a relational database (such as mysql) and the most basic simple data transformations (insertion, deletion, modification) into the data inserted into the Hive database and performing ETL batch scheduling to complete data processing, which is only a simple SQL transformation.

[0004] However, this patent document does not solve the technical problems of the need to overall migrate the complex processing script logic code of the low version (Hive library) to the complex processing logic of the high version (Hive library) during cluster upgrade, and a series of verification process methods for verifying each row and each field of the data output by the ETL script after migration and transformation. Summary of the Invention

[0005] Aiming at the deficiencies in the prior art, the purpose of the present invention is to provide a method and system for ETL processing requirement migration and data verification across cluster versions.

[0006] According to a method for ETL processing requirement migration and data verification across cluster versions provided by the present invention, it includes:

[0007] Step S1: Transforming the logic code of the original requirements on the old version big data cluster into logic code that can be executed on the new version big data cluster with new features;

[0008] Step S2: Performing MD5 transcoding on the result data obtained by running the logic code of the transformed requirements on the new version big data cluster, and performing MD5 transcoding on the result data obtained by running the requirement logic code on the old version big data cluster as comparison data;

[0009] Step S3: Performing cross-platform communication for the comparison job, starting an automatic transmission job flow, and transmitting the MD5 data to be compared;

[0010] Step S4: Compare the MD5 data of the new cluster and the old cluster, automatically verify the total number of data and the data content, and output the differential data to complete the field-level comparison and analysis.

[0011] Preferably, the step S2 includes the following steps:

[0012] Step S2.1: Generate an MD5 transcoding generation script for the ETL result table data corresponding to the old version of the big data cluster, and configure the scheduling tool CTM scheduling job dependencies and job flows;

[0013] Step S2.2: Generate an MD5 transcoding generation script for the ETL result table data corresponding to the new version of the big data cluster, and configure the scheduling system scheduling job dependencies and job flows corresponding to the new version of the big data cluster.

[0014] Preferably, the step 3 includes the following steps:

[0015] Step S3.1: Configure the transmission job flow and dependencies in the data transmission source cluster. The transmission job flow includes: transmitting the MD5 code value corresponding to the source cluster to the target cluster. After detecting the completion of the transmission, automatically generate a notification file indicating the completion of the MD5 code value transmission of the source cluster in the target cluster;

[0016] Step S3.2: Configure the job for monitoring the notification file in the target cluster where the MD5 code value data is transmitted. If the notification file already exists, notify the comparison job, and trigger step S4 for comparison when the comparison preconditions are met.

[0017] Preferably, optimize the content of the with as statement for the old platform, and customize the UDF function of SparkSQL on the new platform;

[0018] At the same time, process the business data on the old and new platforms respectively, and then perform cross-platform data transmission and data consistency comparison; among them, the business processing script of the old platform outputs the report data D1, and the business processing script of the new platform outputs the report data D2, that is, the ETL result table data. Generate the corresponding MD5 table data, that is, the MD5 code value, on the old platform and the new platform to be migrated respectively. Each row of the MD5 table on the old platform corresponds to all fields of each row in the report data D1, and each row of the MD5 table on the new platform corresponds to all fields of each row in the report data D2; in the old and new platforms, transmit the MD5 table data generated daily by one of the big data cluster platforms to the other big data cluster platform for consistency matching calculation, obtain the matching result, and identify the content of the table data with inconsistent comparison as the basis for evaluating the quality of the migration script.

[0019] According to an ETL processing requirement migration and data verification system with cross-cluster versions provided by the present invention, it includes:

[0020] Module M1: Transform the logic code of the original requirements on the old - version big - data cluster into logic code that can be executed on the new - version big - data cluster with new features;

[0021] Module M2: Perform MD5 transcoding on the result data obtained from the logic code of the transformation requirements on the new - version big - data cluster, and perform MD5 transcoding on the result data obtained from the logic code of the requirements on the old - version big - data cluster as comparison data;

[0022] Module M3: Conduct cross - platform communication for the comparison job, start the automatic transmission job flow, and transmit the MD5 data to be compared;

[0023] Module M4: Compare the MD5 data of the new cluster and the old cluster, automatically verify the total number of data records and the data content, and output the differential data to complete the field - level comparison analysis.

[0024] Preferably, the module M2 includes the following modules:

[0025] Module M2.1: Generate an MD5 transcoding generation script for the ETL result table data corresponding to the old - version big - data cluster, and configure the scheduling tool CTM to schedule job dependencies and job flows;

[0026] Module M2.2: Generate an MD5 transcoding generation script for the ETL result table data corresponding to the new - version big - data cluster, and configure the scheduling system corresponding to the new - version big - data cluster to schedule job dependencies and job flows.

[0027] Preferably, the module M3 includes the following modules:

[0028] Module M3.1: Configure the transmission job flow and dependencies in the data transmission source cluster. The transmission job flow includes: transmitting the MD5 code value corresponding to the source cluster to the target cluster, and after detecting the completion of the transmission, automatically generating a notification file indicating the completion of the MD5 code value transmission of the source cluster in the target cluster;

[0029] Module M3.2: Configure the job for monitoring the notification file in the target cluster of the MD5 code value data transmission. If the notification file already exists, notify the comparison job, and trigger module M4 to perform the comparison when the pre - comparison conditions are met.

[0030] Preferably, optimize the content using the with as statement on the old platform and customize the UDF function of SparkSQL on the new platform;

[0031] Process business data on both the old and new platforms simultaneously, and then perform cross-platform data transmission and data consistency comparison. Among them, generate report data D1 from the business processing script of the old platform and report data D2 from the business processing script of the new platform, that is, the ETL result table data, and generate corresponding MD5 table data, that is, MD5 code values, on the old platform and the new platform to be migrated respectively. Each row of the MD5 table on the old platform corresponds to all fields of each row in the report data D1, and each row of the MD5 table on the new platform corresponds to all fields of each row in the report data D2. In the old and new platforms, transfer the MD5 table data generated daily by one of the big data cluster platforms to the other big data cluster platform for consistency matching calculation, obtain the matching results, and identify the inconsistent table data content as the basis for evaluating the quality of the migration script.

[0032] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the cross-cluster version ETL processing requirement migration and data verification method are implemented.

[0033] According to an electronic device provided by the present invention, including a memory, a processor, and a computer program stored on the memory and executable on the processor, when the computer program is executed by the processor, the steps of the cross-cluster version ETL processing requirement migration and data verification method are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. Compared with many traditional data warehouses that do not use big data to implement and are not built on a big data cluster, the present invention provides a cross-platform ETL batch processing requirement transformation, ETL processing result data MD5 transcoding, transmission, communication, and automatic comparison and verification solution based on different hive versions of the big data cluster (hive is a data warehouse tool based on Hadoop).

[0036] 2. The present invention can efficiently and labor-savingly complete the cross-cluster version requirement transformation and migration, reduce the time required for parallel verification of service switching in the business scenario during the migration of big data between the old and new platforms, and improve the migration efficiency.

[0037] 3. By automatically comparing the consistency of the processed data, the present invention can save manpower to quickly locate problems, ensure the integrity and accuracy of business functions, and ensure the quality of application service data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] By reading the following detailed description of non-limiting embodiments with reference to the accompanying drawings, other features, objects, and advantages of the present invention will become more apparent:

[0039] Figure 1Schematic diagram of the method flow steps of the present invention. Detailed implementation manners

[0040] The present invention will be described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several changes and improvements can still be made. These all belong to the protection scope of the present invention.

[0041] The present invention is a solution for transforming ETL batch processing requirements across platforms, MD5 transcoding of result data, transmission, communication, and automated comparison and verification based on different hive versions of big data clusters, so as to efficiently and labor-savingly complete the transformation and migration of cross-cluster version requirements, reduce the switching time between old and new platforms, and ensure the data quality of application services. ETL refers to Extract-Transform-Load, that is, data extraction, transformation, and loading.

[0042] Through the migration and comparison methods, the present invention can automatically verify the quality of each migrated requirement transformation, greatly reducing the labor cost. At the same time, it can help R & D personnel discover problem points and iterate and modify, ensuring that the business data run on two different platforms is 100% exactly the same, enabling the business to switch without any perception.

[0043] According to an ETL processing requirement migration and data verification method for cross-cluster versions provided by the present invention, it includes:

[0044] Step S1: Transformation of SQL for cross-platform and cross-version ETL batch processing;

[0045] In the step S1, the old version of the big data cluster uses hive to execute jobs, with hive version 1.1.0, and the new version of the big data cluster uses spark2.12.10 to execute jobs, with hive version 3.1.3. By using the summarized transformation method, the logical code of the original requirements of the old version cluster is transformed into the logical code that can be executed on the new version cluster. Specifically, the content using the with as statement on the old platform is optimized, and the UDF function of SparkSQL is customized on the new platform to prevent the phenomenon of inconsistent string formats between the old and new platforms caused by the conversion of numeric types to character types into scientific notation.

[0046] Step S2: Perform MD5 transcoding on the result data obtained from the logical codes of the old and new cluster requirements for the transmission and comparison of the transcoded data;

[0047] The step S2 includes the following steps:

[0048] Step S2.1: Use the summarized old platform method to generate an MD5 transcoding generation script for the data in the old version ETL result table, and configure the CTM scheduling job dependencies and job flow. Among them, all fields of types such as decimal, int, and timestamp in the old platform are uniformly defined as varchar type, and uniformly expanded to three times the original file field type before warehousing. When creating this table, it needs to be specified as an external table. Perform default value conversion processing on all fields. If a numeric field is null, set it to '0' or '0.0'. If a character field is null, set it to "". Uniformly convert the content with inconsistent format definitions between the new and old platforms. The MD5 field generation uniformly adopts MD5(CONCAT(TRIM(field 1), TRIM(field 2),...)), and the fields cannot have NULL values. Among them, for varchar type fields, the leading and trailing spaces need to be removed. For decimal type, the rpad function is uniformly used to supplement 0 after the field to fill to a length of 32 bits.

[0049] Step S2.2: Use the summarized new platform method to generate an MD5 transcoding generation script for the data in the new version ETL result table, and configure the ETL batch processing scheduling system scheduling job dependencies and job flow. Among them, all field types in the new platform are uniformly defined as varchar type, and the length type is consistent with the MD5 transcoding script of the old platform. This table is an external table (because it is running data with the spark engine). Process null values. When inserting data from the business table into the MD5 transcoding table, if a numeric field in the business table is null, set it to '0' or '0.0'. If a character field is null, set it to "". Customize the UDF function of SparkSQL to convert the decimal field to a string and perform right zero-padding operation to supplement it to a string with a length of 32 bits, preventing the string from becoming a scientific notation string during direct conversion and ensuring the consistency of the string formats between the new and old platforms. The MD5 field generation uniformly adopts MD5(CONCAT(TRIM(field 1), TRIM(field 2),...)). Timely transfer the partition data of the MD5 transcoding table to the old platform every day. Transfer the partition data of the MD5 transcoding table generated every day to the old platform.

[0050] Step S3: Add an MD5 transcoding job flow completion notification method to conduct cross-platform communication for the daily comparison job, start the automatic transfer job flow, and transfer the data to be compared on the same day. Considering that due to the different schedulers used in the new and old platforms, it is impossible to achieve communication between different scheduler versions, the present invention adopts a communication method of asynchronous communication by transmitting notification files.

[0051] The said Step 3 includes the following steps:

[0052] Step S3.1: Configure a transfer job flow and dependencies in the data transfer source cluster. The transfer job flow includes: transferring the MD5 code value corresponding to the source cluster to the target cluster. After detecting the completion of the transfer, an notification file indicating the completion of the MD5 code value transfer from the source cluster is automatically generated in the target cluster. The step S3.1 includes:

[0053] Step S3.1.1: Wait for all the dependent data of the new platform processing job flow to arrive at the cluster on the same day, and then perform processing to generate business data D1;

[0054] Step S3.1.2: Start the MD5 code value generation job flow to convert the D1 business data into corresponding MD5 table data;

[0055] Step S3.1.3: Trigger the transfer job flow to transfer the MD5 table data and the notification file to a pre-set path in the old platform HDFS; after the transfer is completed, load the MD5 table data into the current day's partition in the old platform MD5 table;

[0056] Step S3.1.4: Trigger the notification file for comparing the corresponding MD5 table data. The file is named with the MD5 table name + business date. After the old platform receives the notification file, trigger the CTM scheduling to automatically compare the MD5 table data;

[0057] Step S3.2: Configure the notification file monitoring method in the data transfer target cluster, configure the dependency notification for the completion of the MD5 code value calculation corresponding to the target cluster, and perform the comparison in step S4 when the comparison preconditions are met.

[0058] Step S4: Use a general comparison method to automatically verify the total number of data records, the data content, and output the different data to complete the field-level comparison analysis. Among them, the automatic verification of the total number of data records and the data content includes:

[0059] Step S4.1: Write a template SQL for MD5 comparison (the comparison work is carried out in the old platform), set the values of the table name, partition date, and limit number of records as variables, and automatically generate daily comparison variable parameters by CTM scheduling and pass them into the comparison job flow;

[0060] Step S4.2: The comparison script counts the total number of data records in the same partition of the MD5 transcoding tables in the new and old platforms;

[0061] Step S4.3: The comparison script counts how many records do not match;

[0062] Step S4.4: If the "number of unmatched records" is 0, it indicates that after migration, the business data of the old and new platforms are consistent, and the comparison ends. If the "number of unmatched records" is not 0, the differential data is printed and provided to developers for accurate positioning to troubleshoot the reasons for the inconsistent matching (the process of finding the reasons is not regular and is performed manually).

[0063] By adopting the cross-platform data automatic comparison method with MD5 transcoding, when the cluster is upgraded and faces batch requirement migration and transformation, the problems of rapid verification of processing logic and data quality guarantee are solved. By automatically comparing the consistency of the processed data, it is possible to save manpower to quickly locate problems, thereby reducing the time for switching between the old and new platforms to provide services in parallel and ensuring the integrity and accuracy of business functions.

[0064] The present invention summarizes the differences in implementing the same logic between the old and new platforms and the differences in data display and conversion, and develops scripts corresponding to each complex processing business requirement manually according to this summary method. In the summary method, in addition to modifying the differences in table names, database names, and partitions between the old and new platforms, one of the main innovations of the present invention is to optimize the content using the with as statement for the old platform and customize the UDF function of SparkSQL for the new platform to prevent the phenomenon of inconsistent string formats between the old and new platforms caused by the conversion of numerical types to character types into scientific notation.

[0065] Considering that only by comparing the running results of the old and new platforms to be exactly the same can it be proved that the logic of the migrated script is consistent with the original requirement, it is necessary to process business data on both platforms at the same time, and then perform cross-platform data transmission and data consistency comparison. That is to say, the present invention performs automatic running and automatic comparison of the two platforms in production, thereby solving the problem that the test environment cannot provide real data consistent with the production scenario to cover the business scope of production data due to the confidentiality of production data. Among them, the comparison days can be adjusted manually, for example, the comparison days are 30 days.

[0066] The present invention generates corresponding MD5 table data, that is, MD5 hash values (each row of the MD5 table corresponds to all fields of each row in D1), for the report data D1, that is, the ETL result table data, produced by the business processing scripts in the two environments according to the method given in the present invention on the old platform and the new platform to be migrated respectively. The MD5 table data generated daily by one party is transmitted to the big data cluster platform of the other party for consistency matching calculation (here, the new platform is transmitted to the old platform), and the matching result is obtained, and the content of the table data with inconsistent comparison is marked. The process of using this as the basis for evaluating the quality of the migration script is automated. In the case of numerous migration scripts, huge data volume, and it is impossible for humans to accurately verify each piece of data in each table, the automatic comparison method of the present invention saves a large amount of manpower and material resources for migration and finally obtains a satisfactory result.

[0067] The present invention also provides a cross-cluster version of the ETL processing requirement migration and data verification system. Those skilled in the art can implement the cross-cluster version of the ETL processing requirement migration and data verification system by executing the process steps of the cross-cluster version of the ETL processing requirement migration and data verification method. That is, the cross-cluster version of the ETL processing requirement migration and data verification method can be understood as the preferred implementation manner of the cross-cluster version of the ETL processing requirement migration and data verification system. Specifically, according to a cross-cluster version of the ETL processing requirement migration and data verification system provided by the present invention, it includes:

[0068] Module M1: Transform the logic code of the original requirements on the old version of the big data cluster into logic code that can be executed on the new version of the big data cluster with new features;

[0069] Module M2: Perform MD5 transcoding on the result data obtained by running the logic code of the transformed requirements on the new version of the big data cluster, and perform MD5 transcoding on the result data obtained by running the requirement logic code on the old version of the big data cluster as comparison data;

[0070] Module M3: Perform cross-platform communication for the comparison job, start the automatic transmission job flow, and transmit the MD5 data to be compared;

[0071] Module M4: Compare the MD5 data of the new cluster and the old cluster, automatically verify the total number of data and the data content, and output the differential data to complete the field-level comparison analysis.

[0072] The module M2 includes the following modules:

[0073] Module M2.1: Generate an MD5 transcoding generation script for the ETL result table data corresponding to the old version of the big data cluster, and configure the scheduling tool CTM scheduling job dependencies and job flow;

[0074] Module M2.2: Generate an MD5 transcoding generation script for the ETL result table data corresponding to the new version of the big data cluster, and configure the scheduling system scheduling job dependencies and job flow corresponding to the new version of the big data cluster.

[0075] The module M3 includes the following modules:

[0076] Module M3.1: Configure the transmission job flow and dependencies in the data transmission source cluster. The transmission job flow includes: transmitting the MD5 code value corresponding to the source cluster to the target cluster, and after detecting the completion of the transmission, automatically generating a notification file indicating the completion of the MD5 code value transmission of the source cluster in the target cluster;

[0077] Module M3.2: For the job of configuring and monitoring notification files in the target cluster for MD5 code value data transmission, if the notification file already exists, notify the comparison job, and trigger Module M4 for comparison when the pre - comparison conditions are met.

[0078] Optimize the content using the with as statement on the old platform and customize the UDF function of SparkSQL on the new platform;

[0079] At the same time, process business data on both the old and new platforms respectively, and then perform cross - platform data transmission and data consistency comparison; among them, the business processing scripts on the old platform produce report data D1, and the business processing scripts on the new platform produce report data D2, that is, ETL result table data. Generate corresponding MD5 table data, that is, MD5 code values, on the old platform and the new platform to be migrated respectively. Each row in the MD5 table on the old platform corresponds to all fields in each row of report data D1, and each row in the MD5 table on the new platform corresponds to all fields in each row of report data D2. In both the old and new platforms, transfer the MD5 table data generated daily by one big data cluster platform to the other big data cluster platform for consistency matching calculation, obtain the matching results, and identify the content of the table data with inconsistent comparison as the basis for evaluating the quality of the migration script.

[0080] Those skilled in the art know that in addition to implementing the systems, devices, and their respective modules provided by the present invention in the form of pure computer - readable program code, the method steps can be logically programmed to enable the systems, devices, and their respective modules provided by the present invention to be implemented in the form of logic gates, switches, application - specific integrated circuits, programmable logic controllers, and embedded microcontrollers, etc. to achieve the same program. Therefore, the systems, devices, and their respective modules provided by the present invention can be regarded as a kind of hardware component, and the modules included therein for implementing various programs can also be regarded as the structures within the hardware component; the modules for implementing various functions can also be regarded as either software programs for implementing the method or structures within the hardware component.

[0081] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above - mentioned specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. Without conflict, the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily.

Claims

1. A method for migrating ETL processing requirements and validating data across cluster versions, characterized in that, Including: Step S1: Transform the logic code of the original requirements on the old version big data cluster into logic code that can be executed on the new version big data cluster with new features; Step S2: Perform MD5 transcoding on the result data obtained from running the logic code of the transformed requirements on the new version big data cluster, and perform MD5 transcoding on the result data obtained from running the requirements logic code on the old version big data cluster as comparison data; Step S3: Conduct cross-platform communication for the comparison job, start the automatic transmission job flow, and transmit the MD5 data to be compared; Step S4: Compare the MD5 data of the new cluster and the old cluster, automatically verify the total number of data and the data content, and output the differential data to complete the field-level comparison analysis; The said Step 3 includes the following steps: Step S3.1: Configure the transmission job flow and dependencies in the data transmission source cluster. The transmission job flow includes: transmitting the MD5 code value corresponding to the source cluster to the target cluster, and after detecting the completion of the transmission, automatically generating a notification file in the target cluster indicating the completion of the transmission of the MD5 code value of the source cluster; Step S3.2: Configure the job for monitoring the notification file in the target cluster for the MD5 code value data transmission. If the notification file already exists, notify the comparison job and trigger Step S4 for comparison when the preconditions for comparison are met; Optimize the content using the with as statement on the old platform, and customize the UDF function of SparkSQL on the new platform; At the same time, process the business data on the old and new platforms respectively, and then perform cross-platform data transmission and data consistency comparison; among them, generate report data D1 from the business processing script of the old platform and report data D2 from the business processing script of the new platform, that is, the ETL result table data, and generate the corresponding MD5 table data, that is, the MD5 code value, on the old platform and the new platform to be migrated respectively. Each row of the old platform MD5 table corresponds to all fields of each row in the report data D1, and each row of the new platform MD5 table corresponds to all fields of each row in the report data D2; in the old and new platforms, transmit the MD5 table data generated daily by one of the big data cluster platforms to the other big data cluster platform for consistency matching calculation, obtain the matching result, and mark the content of the table data with inconsistent comparison as the basis for evaluating the quality of the migration script.

2. The method for migrating ETL processing requirements across clusters and validating data according to claim 1, wherein The said Step S2 includes the following steps: Step S2.1: Generate an MD5 transcoding generation script for the ETL result table data corresponding to the old version big data cluster, and configure the scheduling tool CTM scheduling job dependencies and job flow; Step S2.2: Generate an MD5 transcoding generation script for the ETL result table data corresponding to the new version big data cluster, and configure the scheduling system scheduling job dependencies and job flow corresponding to the new version big data cluster.

3. An ETL processing requirement migration and data verification system for cross-cluster versions, characterized in that, Including: Module M1: Transform the logic code of the original requirements on the old version big data cluster into logic code that can be executed on the new version big data cluster with new features; Module M2: Perform MD5 transcoding on the result data obtained from running the logic code of the transformed requirements on the new version big data cluster, and perform MD5 transcoding on the result data obtained from running the requirements logic code on the old version big data cluster as comparison data; Module M3: Perform cross-platform communication for comparison operations, start the automatic transmission job flow, and transmit the MD5 data to be compared; Module M4: Compare the MD5 data of the new cluster and the old cluster, automatically verify the total number of data and the data content, output the differential data, and complete the field-level comparison analysis; The said Module M3 includes the following modules: Module M3.1: Configure the transmission job flow and dependencies in the data transmission source cluster. The transmission job flow includes: transmitting the MD5 code value corresponding to the source cluster to the target cluster. After detecting the completion of the transmission, automatically generate a notification file in the target cluster indicating the completion of the MD5 code value transmission of the source cluster; Module M3.2: Configure the job for monitoring the notification file in the target cluster for the MD5 code value data transmission. If the notification file already exists, notify the comparison operation, and trigger Module M4 to perform the comparison when the preconditions for comparison are met; Optimize the content using the with as statement for the old platform, and customize the UDF function of SparkSQL on the new platform; At the same time, process the business data on the old and new platforms respectively, and then perform cross-platform data transmission and data consistency comparison; among them, the business processing script of the old platform outputs the report data D1, and the business processing script of the new platform outputs the report data D2, that is, the ETL result table data. Generate the corresponding MD5 table data, that is, the MD5 code value, on the old platform and the new platform to be migrated respectively. Each row of the MD5 table on the old platform corresponds to all fields of each row in the report data D1, and each row of the MD5 table on the new platform corresponds to all fields of each row in the report data D2; in the old and new platforms, transmit the MD5 table data generated daily by one of the big data cluster platforms to the other big data cluster platform for consistency matching calculation, obtain the matching result, and mark the content of the table data with inconsistent comparison as the basis for evaluating the quality of the migration script.

4. The ETL processing requirement migration and data verification system for cross-cluster versions according to claim 3, wherein The said Module M2 includes the following modules: Module M2.1: Generate the MD5 transcoding generation script for the ETL result table data corresponding to the old version big data cluster, and configure the scheduling tool CTM scheduling job dependencies and job flow; Module M2.2: Generate the MD5 transcoding generation script for the ETL result table data corresponding to the new version big data cluster, and configure the scheduling system scheduling job dependencies and job flow corresponding to the new version big data cluster.

5. A computer-readable storage medium storing a computer program, characterized in that, When the said computer program is executed by a processor, it implements the steps of the cross-cluster version ETL processing requirement migration and data verification method described in any one of claims 1 to 2.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the said computer program is executed by a processor, it implements the steps of the cross-cluster version ETL processing requirement migration and data verification method described in any one of claims 1 to 2.

Images