Data migration method and device, electronic equipment and readable storage medium
By selecting target verification rules based on data category weights, the data to be verified is verified and migrated, solving the problem of low data verification quality caused by human experience selection and achieving more efficient data migration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTHERN POWER GRID DIGITAL GRID RESEARCH INSTITUTE CO LTD
- Filing Date
- 2023-03-07
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, the selection of verification rules based on human experience leads to low data verification quality and reduces data migration efficiency.
By selecting the target inspection rule from the preset inspection rules according to the category weight of each sub-data to be inspected, the data to be inspected is inspected, and the data is sent to the target system after meeting the preset data migration standards.
It improved the accuracy and quality of data verification, reduced the number of data verifications, and increased data migration efficiency.
Smart Images
Figure CN116166638B_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of data processing, and in particular relates to a data migration method, apparatus, electronic device, and readable storage medium. Background Technology
[0002] With the development of information technology, enterprises face challenges such as updating and iterating old and new systems, deploying to the cloud, or integrating business processes, requiring the migration of data from old systems to new ones. During the migration process, intermediate processing of historical data is performed to ensure that the processed data meets the requirements of the new system.
[0003] Currently, Extract-Transform-Load (ETL) technology is used to inspect historical data using tools or pre-defined rule scripts to identify and address issues, ensuring the processed data meets the standards of the new system. However, ETL tools and pre-defined rule scripts are limited by computing power and cannot execute all inspection rules within a limited timeframe. In existing technologies, inspection rules are typically selected based on the human experience of testers.
[0004] However, due to limitations imposed by human experience, the verification rules selected by testers often do not match historical data, resulting in quality issues in historical data after data verification and processing. This leads to low verification quality, requiring multiple verifications and reducing data migration efficiency. Summary of the Invention
[0005] This application provides a data migration method, apparatus, electronic device, and readable storage medium to solve the problem that operations based on human experience to select verification rules result in low verification quality and reduced data migration efficiency.
[0006] To solve the above-mentioned technical problems, this application is implemented as follows:
[0007] In a first aspect, this application provides a data migration method, the method comprising:
[0008] The system obtains the data to be checked from the source system and classifies the data according to preset classification conditions to obtain sub-data to be checked.
[0009] Based on the category weights corresponding to each sub-data to be checked, the target check rule for this data check is selected from the preset check rules;
[0010] The data to be checked is checked according to the target check rules, and the data to be migrated is generated based on the data check results and the data to be checked.
[0011] If the data to be migrated meets the preset data migration criteria, the data to be migrated is sent to the target system to migrate the data from the source system to the target system.
[0012] Optionally, the step of selecting the target inspection rule for this data inspection from preset inspection rules based on the category weights corresponding to each sub-data to be inspected includes:
[0013] Based on the category weights corresponding to each sub-data to be inspected, the rule score of each inspection rule in the preset inspection rules is determined;
[0014] Select a check rule whose score is not less than a preset score threshold from the preset check rules, and use it as the target check rule for this data check.
[0015] Optionally, before sending the data to be migrated to the target system if the data meets the preset data migration criteria, the method further includes:
[0016] The data to be migrated is tested according to preset test cases to obtain test results; the test cases are determined according to the data migration standards.
[0017] The step of sending the data to be migrated to the target system when the data meets the preset data migration criteria includes:
[0018] If the test results indicate that the data to be migrated meets the preset data migration standards, the data to be migrated will be sent to the target system.
[0019] Optionally, after verifying the data to be migrated according to preset test cases to obtain the verification results, the method further includes:
[0020] If the test results indicate that the data to be migrated does not meet the preset data migration standards, the target check rule is corrected according to the test results to obtain the corrected target check rule.
[0021] The data to be checked is re-checked according to the corrected target check rules, and data to be migrated is generated based on the data check results and the data to be checked.
[0022] Optionally, the step of correcting the target inspection rule based on the inspection result includes:
[0023] Based on the quality issues found in the data in the inspection results, select the inspection rule corresponding to the quality issues from the preset inspection rules as the inspection rule to be supplemented.
[0024] The missing check rules are added to the target check rules to correct the target check rules.
[0025] Optionally, the target verification rules include basic verification rules and transformation verification rules. The step of performing data verification on the data to be verified according to the target verification rules, and generating data to be migrated based on the data verification results and the data to be verified, includes:
[0026] The data to be inspected is pre-inspected according to the basic inspection rules to determine the data to be inspected that has original quality problems.
[0027] The data to be inspected is converted according to the original quality problem and the preset data conversion standard in order to process the original quality problem and obtain the converted data to be inspected.
[0028] The converted data to be checked is checked according to the conversion check rules, and the data to be migrated is generated based on the conversion check results and the converted data to be checked.
[0029] Optionally, obtaining the data to be checked from the source system includes:
[0030] The data in the source system is unloaded according to the preset data unloading format to obtain the source data;
[0031] Based on the source data and the preset data completion standard, determine the supplementary data corresponding to the source data;
[0032] Data to be checked is generated based on the source data and the supplementary data.
[0033] Secondly, this application provides a data migration apparatus, the apparatus comprising:
[0034] The acquisition module is used to acquire the data to be inspected from the source system, and classify the data to be inspected according to preset classification conditions to obtain sub-data to be inspected.
[0035] The selection module is used to select the target inspection rule for this data inspection from the preset inspection rules based on the category weights corresponding to each sub-data to be inspected.
[0036] The inspection module is used to perform data inspection on the data to be inspected according to the target inspection rules, and generate data to be migrated based on the data inspection results and the data to be inspected.
[0037] The sending module is used to send the data to be migrated to the target system when the data to be migrated meets the preset data migration criteria, so as to migrate the data from the source system to the target system.
[0038] Thirdly, this application provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described data migration method.
[0039] Fourthly, this application provides a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the above-described data migration method.
[0040] In this embodiment, the process involves obtaining data to be checked from the source system, classifying the data according to preset classification conditions to obtain sub-data to be checked, selecting a target check rule from preset check rules based on the category weights corresponding to each sub-data to be checked, performing data check on the data to be checked according to the target check rule, and generating data to be migrated based on the data check results and the data to be checked, and sending the data to be migrated to the target system if it meets preset data migration standards, thereby migrating the data from the source system to the target system. In this way, since the target check rule is determined based on the category weights corresponding to each sub-data to be checked, the target check rule for this data check can be matched with the data to be checked, improving the rationality of the check rule. Furthermore, by performing data verification on the data to be verified according to the target verification rules, more accurate data verification results can be obtained. Since the data to be migrated is generated based on more accurate data verification results and the data to be verified, the quality of the data to be migrated obtained in this data verification can be improved, making the data to be migrated conform to the preset data migration standards, reducing the number of data verifications, and thus improving data migration efficiency. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is a flowchart illustrating the steps of a data migration method provided in an embodiment of this application;
[0043] Figure 2 This is a flowchart illustrating a data migration method provided in an embodiment of this application;
[0044] Figure 3 This is a structural diagram of a data migration device provided in an embodiment of this application;
[0045] Figure 4This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation
[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0047] Figure 1 This is a flowchart illustrating the steps of a data migration method provided in an embodiment of this application, as follows: Figure 1 As shown, the method may include:
[0048] Step 101: Obtain the data to be checked from the source system, and classify the data to be checked according to preset classification conditions to obtain sub-data to be checked.
[0049] In this embodiment, the source system can be the system where the migrated data resides. The source system includes source data, and the data to be checked can be obtained based on the source data. The preset classification conditions can be classification conditions set according to the source, attributes, or uses of the data. The data to be checked is classified according to the preset classification conditions to obtain the data to be checked for each category, and the data to be checked for any category is taken as a sub-data to be checked. It is understood that, according to the preset classification conditions, there can be one or more sub-data to be checked, and this embodiment does not limit this.
[0050] For example, the data to be inspected can be categorized according to preset classification criteria: transaction data, customer data, log data, management data, and system data. The preset classification criteria can be shown in Table 1.
[0051] Table 1 Preset classification conditions
[0052] Trading Amount, unit price, quantity, transaction number, product number, transaction status, etc. Customer Class Customer attribute information, customer classification information, customer indicator information, related customers, etc. Log class Automatic business process logs, transaction details, and transaction product lists, etc. Management Sales targets, customer allocation, sales activity records, sales performance evaluation, etc. System Class System logs, system-generated IDs, standard data dictionary, timestamps, etc.
[0053] Step 102: Based on the category weights corresponding to each sub-data to be checked, select the target check rule for this data check from the preset check rules.
[0054] In this embodiment, category weights can be pre-set for each category. The category weight of the category to which any sub-data to be checked belongs is used as the category weight corresponding to that sub-data to be checked, thereby determining the category weights corresponding to each sub-data to be checked. For any selected check rule among the preset check rules, it can be determined whether the check rule can be pruned based on the relationship between the check rule and the category weights corresponding to each sub-data to be checked. If the check rule cannot be pruned, it is used as the target check rule; otherwise, it cannot be used as the target check rule. Specifically, the rule parameters corresponding to the check rule in any category can be determined, and then it can be determined whether the check rule can be pruned based on all the rule parameters. For example, if Rule 1 = Transaction parameters + Customer parameters + Log parameters + Management parameters + System parameters ≥ Unpruned parameters, then Rule 1 cannot be pruned. Furthermore, the check rules that cannot be pruned among the preset check rules are used together as the target check rules for this data check.
[0055] Step 103: Perform data verification on the data to be verified according to the target verification rules, and generate data to be migrated based on the data verification results and the data to be verified.
[0056] In this embodiment, data checks are performed on the data to be checked according to the target check rules to identify quality problems and obtain data check results. Based on the data check results, the quality problems in the data to be checked are processed to resolve them, and the processed data is then used as the data to be migrated.
[0057] Step 104: If the data to be migrated meets the preset data migration criteria, the data to be migrated is sent to the target system to migrate the data from the source system to the target system.
[0058] In this embodiment, the preset data migration standard can be determined in advance based on the data structure standards and data quality requirements of the target system, as well as data standards proposed for the data to be migrated to the target system. The preset data migration standard can be used to examine the data to be migrated to determine whether it conforms to the preset standard. Specifically, the preset data migration standard can be used to examine whether there are data quality problems in the data to be migrated to determine whether it conforms to the preset standard. If there are data quality problems in the data to be migrated, and these problems must be addressed, then the data to be migrated is determined to not conform to the preset standard. If there are no data quality problems in the data to be migrated, or even if there are data quality problems, but they are uncorrectable and do not affect other data, then the data to be migrated is determined to conform to the preset standard.
[0059] Optionally, for data with such quality issues that cannot be corrected and do not affect other data, a "Migration Issue Remarks" field or a "tag" can be added to the data for reference, so that the system users can be notified of the data quality issues when migrating to the new system.
[0060] In this embodiment, if the data to be migrated meets the preset data migration criteria, it is identified as the target data, which can then be migrated to the target system. The target data inherits data from the source system while also meeting the structural standards and quality requirements of the target system. The target data can be sent to the target system so that it can store the target data in its own database, thereby achieving the migration of data from the source system to the target system.
[0061] In this embodiment, the process involves obtaining data to be checked from the source system, classifying the data according to preset classification conditions to obtain sub-data to be checked, selecting a target check rule from preset check rules based on the category weights corresponding to each sub-data to be checked, performing data check on the data to be checked according to the target check rule, and generating data to be migrated based on the data check results and the data to be checked, and sending the data to be migrated to the target system if it meets preset data migration standards, thereby migrating the data from the source system to the target system. In this way, since the target check rule is determined based on the category weights corresponding to each sub-data to be checked, the target check rule for this data check can be matched with the data to be checked, improving the rationality of the check rule. Furthermore, by performing data verification on the data to be verified according to the target verification rules, more accurate data verification results can be obtained. Since the data to be migrated is generated based on more accurate data verification results and the data to be verified, the quality of the data to be migrated obtained in this data verification can be improved, making the data to be migrated conform to the preset data migration standards, reducing the number of data verifications, and thus improving data migration efficiency.
[0062] Optionally, step 102 may include the following steps:
[0063] Step 1021: Determine the rule score of each inspection rule in the preset inspection rules based on the category weight corresponding to each sub-data to be inspected.
[0064] In this embodiment, for any selected inspection rule among the preset inspection rules, the score obtained by the inspection rule in that category can be determined based on the category weight corresponding to the inspection rule and any sub-data to be inspected. Then, the rule score corresponding to the inspection rule is determined based on the scores obtained by the inspection rule in each category. For example, the rule score of rule 1 = transaction score + customer score + log score + management score + system score. Accordingly, the rule score of each inspection rule in the preset inspection rules is determined respectively.
[0065] Step 1022: Select an inspection rule whose rule score is not less than a preset score threshold from the preset inspection rules, and use it as the target inspection rule for this data inspection.
[0066] In this embodiment, for any selected inspection rule among the preset inspection rules, the rule score corresponding to the inspection rule can be compared with a preset score threshold to determine whether the inspection rule can be pruned. If the rule score corresponding to the inspection rule is not less than the preset score threshold, then the inspection rule cannot be pruned and can be used as the target inspection rule. Further, inspection rules among the preset inspection rules whose rule scores are not less than the preset score threshold are collectively used as the target inspection rules for this data inspection. For example, as shown in Table 2, it is determined whether the inspection rules among the preset inspection rules can be pruned, and the rules that cannot be pruned are used as the target inspection rules for this data inspection.
[0067] Table 2 Target Inspection Rules
[0068]
[0069]
[0070] It should be noted that in Table 2, the preset scoring threshold is 75. A score greater than or equal to 75 indicates that the inspection rule cannot be trimmed and can be used as the target inspection rule.
[0071] In this embodiment, the rule score of each pre-set inspection rule is determined based on the category weight corresponding to each sub-data to be inspected. Inspection rules with a rule score not less than a pre-set score threshold are selected from the pre-set inspection rules as the target inspection rule for this data inspection. This allows for convenient selection of the target inspection rule based on the rule score, and since the rule score is determined according to the category weight corresponding to each sub-data to be inspected, the target inspection rule can be matched with the data to be inspected, improving the rationality of the target inspection rule for this data inspection.
[0072] Optionally, prior to step 104, the method further includes:
[0073] Step 201: The data to be migrated is tested according to preset test cases to obtain test results; the test cases are determined according to the data migration standard.
[0074] In this embodiment, test cases are used to check for quality issues in the data to be migrated, in order to verify whether the data meets the preset data migration standards. Test cases can be generated based on specific indicators in the data migration standards, which include indicators corresponding to the target system's data structure standards and data quality requirements. Optionally, test cases may include a test case title, preconditions, test steps, and expected results. Inspection code can be written in advance based on the test cases, and the test steps can be implemented by running the code, thereby detecting quality issues in the data to be migrated.
[0075] Optionally, the data to be migrated can be initialized in a User Acceptance Testing (UAT) environment or a pre-production environment. During initialization, data transmission should be performed according to security requirements, and sensitive data should be anonymized. Alternatively, a sample of the full dataset can be taken, and then the sampled data can be initialized in a UAT testing environment or a pre-production environment. The data to be migrated can then be verified in the UAT testing environment or pre-production environment according to pre-defined test cases.
[0076] In this embodiment, the data to be migrated is tested according to preset test cases to determine whether the data meets the expected results, i.e., the preset data migration standards, and test results are generated. The test results may include a final conclusion on whether the data to be migrated meets the preset data migration standards, and, if the data does not meet the preset data migration standards, the data with quality problems and the corresponding quality problems.
[0077] Optionally, step 104 may include the following steps:
[0078] Step 1041: If the test results indicate that the data to be migrated meets the preset data migration standards, the data to be migrated is sent to the target system.
[0079] In this embodiment of the application, if the test results indicate that the data to be migrated meets the preset data migration standard, the data to be migrated is determined as the target data, and then the target data is sent to the target system so that the target system can store the target data in its own database, thereby realizing the migration of data from the source system to the target system.
[0080] In this embodiment, the data to be migrated is tested according to preset test cases to obtain test results. If the test results indicate that the data to be migrated meets preset data migration standards, the data to be migrated is sent to the target system. Since the test cases are determined based on data migration standards, it is convenient to determine whether the data to be migrated meets the preset data migration standards. Sending the data to be migrated to the target system when it meets the preset data migration standards makes the data to be migrated more compatible with the target system and achieves better data migration results.
[0081] Optionally, after step 201, the method further includes:
[0082] Step 301: If the inspection result indicates that the data to be migrated does not meet the preset data migration standard, the target inspection rule is corrected according to the inspection result to obtain the corrected target inspection rule.
[0083] In this embodiment of the application, when the test results indicate that the data to be migrated does not meet the preset data migration standard, it is possible to determine whether there are any missing test rules in the target test rules based on the data with quality problems and the corresponding quality problems in the data to be migrated included in the test results. If there are any missing test rules in the target test rules, the target test rules are supplemented based on the data with quality problems and the corresponding quality problems to obtain new target test rules, thereby correcting the target test rules, and the new test rules are used as the corrected target test rules.
[0084] Step 302: Re-check the data to be checked according to the corrected target check rules, and generate data to be migrated based on the data check results and the data to be checked.
[0085] In this embodiment, the data to be checked is re-checked according to the corrected target check rules, thereby discovering quality problems in the data to be checked, especially quality problems that were not found in the previous data check, and obtaining new data check results. Based on the new data check results, the quality problems in the data to be checked are processed to solve the quality problems of the data to be checked, and the processed data is used as the data to be migrated.
[0086] In this embodiment, when the inspection results indicate that the data to be migrated does not meet the preset data migration standards, the target verification rules are corrected based on the inspection results to obtain corrected target verification rules. This improves the accuracy of the target verification rules used for re-verification of the data. The data to be verified is then re-verified according to the corrected target verification rules, and the data to be migrated is generated based on the data verification results and the data to be verified. Thus, by using more accurate target verification rules to verify the data to be migrated, the quality of data verification is improved, resulting in higher-quality data to be migrated.
[0087] Optionally, step 301 may include the following steps:
[0088] Step 3011: Based on the quality problems existing in the data in the inspection results, select the inspection rule corresponding to the quality problem from the preset inspection rules as the inspection rule to be supplemented.
[0089] In this embodiment, the quality issues found in the data during the inspection results can be verified using test cases to identify data in the data to be migrated that does not conform to the data structure standards and / or data quality requirements of the target system. The differences between the structure and / or content of the data to be migrated and the data structure standards and / or data quality requirements of the target system are then considered as quality issues in the data to be migrated. The target verification rules can be supplemented based on the data with quality issues in the data to be migrated included in the inspection results and the corresponding quality issues. Specifically, verification rules that can detect the quality issue can be selected from preset verification rules, i.e., the verification rule corresponding to the quality issue can be selected as the verification rule to be supplemented.
[0090] Step 3012: Add the supplementary inspection rules to the target inspection rules to correct the target inspection rules.
[0091] In this embodiment of the application, the inspection rules to be supplemented can be added to the target inspection rules to obtain new target inspection rules, thereby correcting the target inspection rules and using the new inspection rules as the corrected target inspection rules.
[0092] In this embodiment, based on the quality issues found in the test results, a pre-defined inspection rule corresponding to the quality issue is selected as a supplementary inspection rule. This supplementary rule is then added to the target inspection rule to correct it. This facilitates the correction of the target inspection rule and improves its accuracy.
[0093] Optionally, the target inspection rule includes basic inspection rules and transformed inspection rules, and step 103 may include the following steps:
[0094] Step 1031: Perform a pre-transfer inspection on the data to be inspected according to the basic inspection rules to determine the data in the data to be inspected that has original quality problems.
[0095] In this embodiment, the basic check rules are used to check the data quality of the unprocessed raw data. Examples include the total number of rows in a table, the number of records with empty data in a field, the number of records with incorrect formatting in a field, and the number of records with strings shorter than a certain length in a field. These are merely illustrative examples, and this embodiment does not impose any limitations on them.
[0096] In this embodiment, an ETL inspection tool or a preset basic inspection script can be used to perform a pre-transformation inspection on the data to be inspected according to basic inspection rules. This inspection aims to identify data quality issues in the data to be inspected, such as null values in non-empty fields, garbled characters in strings, inconsistent time formats, and / or inconsistent numerical representations. This is merely an example and is not intended to limit the scope of the application. Based on the identified data quality issues in the data to be inspected, data with original quality problems is determined.
[0097] Step 1032: Perform data conversion on the data to be inspected according to the original quality problem and the preset data conversion standard, so as to process the original quality problem and obtain the converted data to be inspected.
[0098] In this embodiment, the preset data migration standard may include the target system's data structure standard and data quality requirements. Based on the original quality issues and the data quality requirements in the data conversion standard, data processing is performed on the data to be checked. Furthermore, the data structure of the data to be checked is adjusted according to the data conversion standard to perform data conversion, thus addressing the original quality issues and obtaining the converted data to be checked.
[0099] Step 1033: Perform post-conversion verification on the converted data to be verified according to the conversion verification rules, and generate data to be migrated based on the post-conversion verification results and the converted data to be verified.
[0100] In this embodiment, the conversion check rules are used to check the data quality of the data to be migrated after data conversion. For example, whether table A in the data table is split into tables B and C, whether field a is mapped to field b, and whether the old 15-digit ID card number is supplemented to the new 18-digit ID card number, etc.
[0101] In this embodiment, the converted data to be checked is checked according to the conversion check rules to identify quality problems and obtain the post-conversion check results. Based on the post-conversion check results, the quality problems in the converted data to be checked are processed to resolve them, and the processed data is used as the data to be migrated.
[0102] In this embodiment, by performing a pre-transfer check on the data to be checked according to basic check rules, data with original quality problems can be easily identified. Data transformation is then performed on the data to be checked based on the original quality problems and preset data transformation standards, which can address the original quality problems and improve the data quality of the transformed data. Furthermore, a post-transfer check is performed on the transformed data according to transformation check rules, and data to be migrated is generated based on the post-transfer check results and the transformed data. This improves the quality of the data to be migrated obtained from this data check.
[0103] Optionally, step 101 may include the following steps:
[0104] Step 1011: Unload the data in the source system according to the preset data unloading format to obtain the source data.
[0105] In this embodiment, the preset data unloading format can be determined based on the format of the database data file of the target system. For example, the preset data unloading format can be .cscv format. Data in the source system can be unloaded and saved as a source data file corresponding to the preset data unloading format to obtain the source data.
[0106] Step 1012: Determine the supplementary data corresponding to the source data based on the source data and the preset data completion standard.
[0107] In this embodiment, the preset data completion standard can be determined based on the target system's data standard. For example, if the target system's data standard specifies mandatory fields, the data completion standard can include those mandatory fields. The source data can be analyzed according to the preset data completion standard to identify missing data, and the corresponding supplementary content can be determined based on the data completion standard. A supplementary data file corresponding to the data loading format can be generated based on the supplementary content to obtain the supplementary data.
[0108] Step 1013: Generate the data to be checked based on the source data and the supplementary data.
[0109] In this embodiment, source data and supplementary data can be generated into a data file, and security measures such as file encryption can be used to store it in a preset storage area, such as a Direct-attached Storage (DAS) data disk. Data can be obtained from the data file as data to be checked. It should be noted that, generally, all historical data in the data file can be retrieved, also known as the full data. However, for cases where the amount of historical data is large and the subsequent checking and conversion processing time is too long, it can be processed by time-sharing, that is, the existing data and incremental data can be checked and converted separately. Here, the existing data refers to data that will not be updated again, and data that is easy to change can be used as incremental data.
[0110] In this embodiment, source data is obtained by unloading data from the source system according to a preset data unloading format; supplementary data corresponding to the source data is determined based on the source data and a preset data completion standard; and data to be checked is generated based on the source data and the supplementary data. This allows for the acquisition of data to be checked that conforms to the preset format and has been completed, thus improving the data migration efficiency of the data migration method in this application.
[0111] Figure 2 This is a flowchart illustrating the data migration method provided in an embodiment of this application, as shown below. Figure 2 As shown, the data migration architecture includes a data area, a check area, and a rules area. The data migration process includes: unloading source data from the source system, obtaining supplementary data corresponding to the source data, generating data files, and retrieving the data to be checked from the data files. The data to be checked can be obtained either as existing data or incremental data. The check program performs a pre-transfer check on the data to be checked according to basic check rules, followed by data transformation, also known as data cleaning, to obtain the transformed data to be checked. Finally, the check program performs a post-transfer check on the transformed data to be checked according to transformation check rules to obtain the data to be migrated.
[0112] The data to be migrated is provided to business acceptance (testing) personnel, who can then verify the data against business acceptance standards using test cases. If acceptance is successful, the data to be migrated is used as the target data and its version is finalized. Data initialization is then performed on the target system according to the new deployment plan. If acceptance fails, the data to be migrated is returned for re-verification and processing until the quality of the acquired data meets the target system's standards. Business personnel can check the data's accuracy through the target system's functional pages or statistical reports. When the page functionality is not intuitive or insufficient, developers can write acceptance programs or scripts and confirm the results with business personnel to determine acceptance success. If acceptance fails, business and technical personnel can analyze the data together, tracing back to whether the problem arose from the cleaning or transformation steps, or to quality issues in the source data, and address these issues during data verification and processing.
[0113] In the rules area, based on the results of business acceptance, the basic and transformed verification rules in the target verification rules can be supplemented. Verification and processing are then re-performed according to the supplemented rules, improving the quality of data verification. Optionally, for verification rule versions with multiple rounds of data verification, the method for selecting target verification rules can be modified by analyzing the patterns of supplementary or pruning rules to obtain more accurate target verification rules. In this way, through the division of labor and collaboration between technical and business personnel, verification rules can be continuously supplemented and iterated, standardizing the data migration process, thereby improving data migration efficiency, saving computing power, and ensuring the data quality of the target data after the target system goes live.
[0114] Figure 3 This is a structural diagram of a data migration apparatus provided in an embodiment of this application. The apparatus 40 may include:
[0115] The acquisition module 401 is used to acquire the data to be inspected from the source system and classify the data to be inspected according to preset classification conditions to obtain sub-data to be inspected.
[0116] The selection module 402 is used to select the target inspection rule for this data inspection from the preset inspection rules according to the category weights corresponding to each sub-data to be inspected.
[0117] The inspection module 403 is used to perform data inspection on the data to be inspected according to the target inspection rules, and generate data to be migrated based on the data inspection results and the data to be inspected.
[0118] The sending module 404 is used to send the data to be migrated to the target system when the data to be migrated meets the preset data migration standard, so as to migrate the data in the source system to the target system.
[0119] Optionally, the selection module 402 is specifically used for:
[0120] Based on the category weights corresponding to each sub-data to be inspected, the rule score of each inspection rule in the preset inspection rules is determined;
[0121] Select a check rule whose score is not less than a preset score threshold from the preset check rules, and use it as the target check rule for this data check.
[0122] Optionally, the device 40 further includes:
[0123] The verification module is used by the sending module 404 to verify the data to be migrated according to preset test cases before sending the data to be migrated to the target system if the data to be migrated meets the preset data migration standards, so as to obtain the verification results; the test cases are determined according to the data migration standards.
[0124] The sending module 404 is specifically used for:
[0125] If the test results indicate that the data to be migrated meets the preset data migration standards, the data to be migrated will be sent to the target system.
[0126] Optionally, the device 40 further includes:
[0127] The correction module is used to verify the data to be migrated according to the preset test cases, and after obtaining the verification results, if the verification results indicate that the data to be migrated does not meet the preset data migration standards, the correction module corrects the target verification rules according to the verification results to obtain the corrected target verification rules.
[0128] The generation module is used to re-check the data to be checked according to the corrected target check rules, and generate data to be migrated based on the data check results and the data to be checked.
[0129] Optionally, the correction module is specifically used for:
[0130] Based on the data showing quality problems in the inspection results, select the inspection rule corresponding to the quality problem from the preset inspection rules as the inspection rule to be supplemented;
[0131] The missing check rules are added to the target check rules to correct the target check rules.
[0132] Optionally, the target verification rules include basic verification rules and transformation verification rules, and the verification module 403 is specifically used for:
[0133] The data to be inspected is pre-inspected according to the basic inspection rules to determine the data to be inspected that has original quality problems.
[0134] The data to be inspected is converted according to the original quality problem and the preset data conversion standard in order to process the original quality problem and obtain the converted data to be inspected.
[0135] The converted data to be checked is checked according to the conversion check rules, and the data to be migrated is generated based on the conversion check results and the converted data to be checked.
[0136] Optionally, the acquisition module 401 is specifically used for:
[0137] The data in the source system is unloaded according to the preset data unloading format to obtain the source data;
[0138] Based on the source data and the preset data completion standard, determine the supplementary data corresponding to the source data;
[0139] Data to be checked is generated based on the source data and the supplementary data.
[0140] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.
[0141] The data migration device and the data migration method described above have the same advantages over the prior art, and will not be repeated here.
[0142] This application also provides an electronic device, see [link to document]. Figure 4 It includes: a processor 501, a memory 502, and a computer program 5021 stored in the memory and executable on the processor. When the processor executes the program, it implements the data migration method of the foregoing embodiments.
[0143] This application also provides a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform the data migration method of the foregoing embodiments.
[0144] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. The structure required to construct such a system is obvious from the above description. Furthermore, this application is not directed to any particular programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of specific languages is for the purpose of disclosing the best mode of implementation of this application.
[0145] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.
[0146] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.
[0147] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.
[0148] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some or all of the components in the sequencing device according to this application. This application can also be implemented as a device or apparatus program for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can take the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.
[0149] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.
[0150] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.
[0151] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.
[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0153] It should be noted that the various data-related processes in the embodiments of this application are carried out in compliance with the relevant data protection laws and policies of the country where the location is located, and with the authorization granted by the owner of the corresponding device.
Claims
1. A data migration method, characterized by, The method includes: The system obtains the data to be checked from the source system and classifies the data according to preset classification conditions to obtain sub-data to be checked. Pre-set category weights for each category, and use the category weight of the category to which any sub-data to be checked belongs as the category weight corresponding to that sub-data to be checked; Based on the category weights corresponding to each sub-data to be checked, the target check rule for this data check is selected from the preset check rules. Specifically, this includes: determining the rule parameters corresponding to the transaction category, the log category, the management category, and the customer category; determining whether the check rule can be pruned based on all the rule parameters of the check rule; and selecting the check rules that cannot be pruned from the preset check rules as the target check rules for this data check. If the sum of all the rule parameters of a check rule is greater than or equal to the number of parameters that cannot be pruned, then the check rule cannot be pruned. The data to be checked is checked according to the target check rules, and the data to be migrated is generated based on the data check results and the data to be checked. If the data to be migrated meets the preset data migration criteria, the data to be migrated is sent to the target system to migrate the data from the source system to the target system. The target verification rules include basic verification rules and transformation verification rules. The step of performing data verification on the data to be verified according to the target verification rules, and generating data to be migrated based on the data verification results and the data to be verified, includes: The data to be inspected is pre-inspected according to the basic inspection rules to determine the data to be inspected that has original quality problems. The data to be inspected is converted according to the original quality problem and the preset data conversion standard in order to process the original quality problem and obtain the converted data to be inspected. The transformed data to be checked is checked according to the transformation check rules, and the data to be migrated is generated based on the transformation check results and the transformed data to be checked. The data to be migrated is provided to the business acceptance team, and the business personnel use test cases to verify the data to be migrated according to the business acceptance standards. Based on the results of business acceptance, the basic and transformed verification rules in the target verification rules are supplemented, and the verification and processing are carried out again according to the supplemented verification rules to improve the quality of data verification. For the verification rule versions of multi-round data verification, the method of selecting target verification rules is modified by analyzing the patterns of supplementary or pruning rules in order to obtain more accurate target verification rules.
2. The method according to claim 1, characterized in that, The step of selecting the target inspection rule for this data inspection from the preset inspection rules based on the category weights corresponding to each sub-data to be inspected includes: Based on the category weights corresponding to each sub-data to be inspected, the rule score of each inspection rule in the preset inspection rules is determined; Select a check rule whose score is not less than a preset score threshold from the preset check rules, and use it as the target check rule for this data check.
3. The method according to claim 1, characterized in that, Before sending the data to be migrated to the target system when the data meets the preset data migration criteria, the method further includes: The data to be migrated is tested according to preset test cases to obtain test results; the test cases are determined according to the data migration standards. The step of sending the data to be migrated to the target system when the data meets the preset data migration criteria includes: If the test results indicate that the data to be migrated meets the preset data migration standards, the data to be migrated will be sent to the target system.
4. The method according to claim 3, characterized in that, After verifying the data to be migrated according to preset test cases to obtain the verification results, the method further includes: If the test results indicate that the data to be migrated does not meet the preset data migration standards, the target check rule is corrected according to the test results to obtain the corrected target check rule. The data to be checked is re-checked according to the corrected target check rules, and data to be migrated is generated based on the data check results and the data to be checked.
5. The method according to claim 4, characterized in that, The step of correcting the target inspection rule based on the inspection results includes: Based on the quality issues found in the data in the inspection results, select the inspection rule corresponding to the quality issues from the preset inspection rules as the inspection rule to be supplemented. The missing check rules are added to the target check rules to correct the target check rules.
6. The method according to claim 1, characterized in that, The process of obtaining the data to be checked from the source system includes: The data in the source system is unloaded according to the preset data unloading format to obtain the source data; Based on the source data and the preset data completion standard, determine the supplementary data corresponding to the source data; Data to be checked is generated based on the source data and the supplementary data.
7. A data migration device, characterized in that, The device includes: The acquisition module is used to acquire the data to be inspected from the source system, and classify the data to be inspected according to preset classification conditions to obtain sub-data to be inspected. The weight determination module is used to pre-set category weights for each category, and to take the category weight of the category to which any sub-data to be checked belongs as the category weight corresponding to that sub-data to be checked; The selection module is used to select the target verification rule for this data verification from the preset verification rules based on the category weights corresponding to each sub-data to be verified. Specifically, this includes: determining the rule parameters corresponding to the transaction category, the log category, the management category, and the customer category; determining whether the verification rule can be pruned based on all the rule parameters of the verification rule; and selecting the verification rules that cannot be pruned from the preset verification rules as the target verification rules for this data verification. If the sum of all the rule parameters of a verification rule is greater than or equal to the number of non-pruning parameters, then the verification rule cannot be pruned. The inspection module is used to perform data inspection on the data to be inspected according to the target inspection rules, and generate data to be migrated based on the data inspection results and the data to be inspected. The sending module is used to send the data to be migrated to the target system when the data to be migrated meets the preset data migration criteria, so as to migrate the data in the source system to the target system; The target verification rules include basic verification rules and transformed verification rules, and the verification module is specifically used for: The data to be inspected is pre-inspected according to the basic inspection rules to determine the data to be inspected that has original quality problems. The data to be inspected is converted according to the original quality problem and the preset data conversion standard in order to process the original quality problem and obtain the converted data to be inspected. The transformed data to be checked is checked according to the transformation check rules, and the data to be migrated is generated based on the transformation check results and the transformed data to be checked. The data to be migrated is provided to the business acceptance team, and the business personnel use test cases to verify the data to be migrated according to the business acceptance standards. Based on the results of business acceptance, the basic and transformed verification rules in the target verification rules are supplemented, and the verification and processing are carried out again according to the supplemented verification rules to improve the quality of data verification. For the verification rule versions of multi-round data verification, the method of selecting target verification rules is modified by analyzing the patterns of supplementary or pruning rules in order to obtain more accurate target verification rules.
8. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the data migration method as described in any one of claims 1-6.
9. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the data migration method according to any one of claims 1-6.