Data migration method in mode of combining total amount with increment

Through the full-scale combined incremental data migration method, the problem of difficult to ensure data accuracy and business continuity in enterprise-level system switching in the financial field is solved, and rapid construction and low-cost system production are achieved, reducing the risk of system switching.

CN118939603APending Publication Date: 2024-11-12LONGYING ZHIDA (BEIJING) TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410990003.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-23
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

During the switching of important enterprise-level systems in the financial field, the existing technology is difficult to ensure data accuracy and business continuity, resulting in long construction process, high cost, high system operation risks, and easy production accidents, affecting business continuity and causing Economic losses.

Method used

Using a full-scale combined incremental data migration method, through the combination of full-scale data migration and incremental data migration, including data preparation, migration source data, import and migration data files, data cleaning, data verification and data backfill, etc., to ensure that Data accuracy and business continuity.

Benefits of technology

It accelerates the construction of enterprise-level systems, reduces construction costs, ensures the stable production of the system, and reduces data problems and risks in the switching process of new and old systems, ensuring business continuity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939603A_ABST
    Figure CN118939603A_ABST
Patent Text Reader

Abstract

The invention discloses a data migration method in a mode of combining full quantity with increment. The data migration method comprises the following steps: S1, preparing data migration; s2, emigrating the source data; s3, importing an emigration data file; s4, importing a cleaning data file; s5, detecting and checking the migration data; s6, exporting a standard data file of the target system; and S7, carrying out data backfilling operation. The construction speed of an enterprise-level system is increased, the construction cost is reduced, the data accuracy and the business connectivity are ensured, and stable production of the system is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data migration, and in particular to a data migration method combining full and incremental methods. Background Art

[0002] The customer information management system (hereinafter referred to as the ECIF system) is an information system for unified management of customer information of the entire bank. After the ECIF system is put into operation, the original customer information scattered in various business systems will be offline, and ECIF customer information will be used uniformly. Due to the large number of business systems, it is difficult to ensure data accuracy and business continuity through a full migration. Therefore, an incremental migration method is adopted, using the parallel stage of the new and old systems to initialize the production stock data to the ECIF system with specific overwriting rules.

[0003] Here, we take an example of a business scenario to illustrate the functions implemented. According to the data and business requirements of the ECIF system, the data and business of the three main data source systems of core, credit, and credit card are sorted and analyzed, and the specific data mapping conversion rules that need to be initialized are confirmed. On the day the ECIF system goes online, a one-time migration is performed. Incremental migration begins after the daily end-of-day batch is completed. The main data source systems for incremental migration are the core system, credit system, and credit card system. After the migration is completed, an incremental data backfill file is generated for the backfill system to verify the data. Data migration on the production day begins after the daily end-of-day batch is completed. The main data source systems for data migration on the production day are the core system, credit system, credit card system, and peer system. After the migration is completed, a daily incremental data summary backfill file after the full data migration is generated for the backfill system to backfill data.

[0004] (1) There is no case in the industry where incremental migration is used to migrate data when switching important enterprise systems.

[0005] (2) There are generally two design principles for industry solutions. The first is to build a single system first to reduce the complexity of system switching. Data can be migrated using a full migration method or a combination of full and incremental migration methods. Other related systems are then gradually integrated to switch the system again, and gradually upgraded to an enterprise-level system. The second is to directly build an enterprise-level system and use a full migration method for data migration.

[0006] (3) The above-mentioned solutions of the same industry have the following problems: In the first way, the construction process of the enterprise-level system is long, the value is slow to be reflected, and it is not easy to see the economic and social benefits. At the same time, the system needs to be frequently switched and upgraded, which is relatively more labor-intensive and costly, and brings greater system operation risks. In the second way, it is very easy to cause production accidents, affect business continuity, cause public opinion risks for the enterprise, and directly cause economic losses to the enterprise. Based on these problems, we have designed a data migration method that combines full and incremental data for important enterprise-level systems in the financial field. It not only speeds up the construction of enterprise-level systems and reduces construction costs, but also ensures data accuracy and business connectivity, ensuring the stable commissioning of the system. Summary of the invention

[0007] In view of the above problems, the present invention is proposed to provide a data migration method combining full and incremental methods to overcome the above problems or at least partially solve the above problems.

[0008] According to one aspect of the present invention, a data migration method in a full and incremental manner is provided, the data migration method comprising: full data migration and incremental data migration;

[0009] The full data migration includes:

[0010] Step S1: prepare for data migration;

[0011] Step S2: Migrate source data;

[0012] Step S3: import the migration data file;

[0013] Step S4: importing the cleaning data file;

[0014] Step S5: Migration data verification;

[0015] Step S6: Exporting target system standard data files;

[0016] Step S7: Perform data backfill operation.

[0017] Optionally, the step S1: preparing for data migration specifically includes:

[0018] Before data migration begins, conduct research on business departments and peripheral systems related to the target system, analyze the current status of each system and the issues raised by each system, and finally determine the scope of migration;

[0019] Incorporate the information items involved in the target system into the target system logical model, analyze and classify the required information fields according to the logical model, and generate the target system physical model;

[0020] Now, according to the table structure document provided by the main data source system, the target system fields are mapped and analyzed, and the mapping relationship is issued by communicating with the source system.

[0021] Optionally, the step S2: migrating source data specifically includes:

[0022] Before data migration, the main data source system must complete the data governance of the problems that hinder the migration;

[0023] Perform initial data processing according to the value selection and mapping rules and intermediate format files in the document "Field Value Selection and Mapping Rules";

[0024] Code value field conversion principle: Code value field conversion is carried out according to Code Value Mapping. Illegal code values ​​are mapped to spaces or null values.

[0025] The conversion principle of non-code value fields shall be carried out in accordance with the "Field Value and Mapping Rules"; the data coverage principle shall be carried out in accordance with the "Field Value and Mapping Rules";

[0026] Push all customer information to the target system in the form of text files;

[0027] The generated data information must meet the uniqueness of the business primary key.

[0028] Optionally, the step S3: importing the migration data file specifically includes:

[0029] The data of the standard data file obtained by the target system includes the full data file of the primary data source system, and then the standard data file is loaded into the local intermediate table.

[0030] Optionally, the step S4: importing the cleaning data file specifically includes:

[0031] The data information provided in the data cleaning phase is associated based on the business primary key to update the data information in the intermediate table of the target system.

[0032] Optionally, the step S5: migration data verification specifically includes:

[0033] For data that does not conform to the physical model and data standard requirements of the target system, it needs to be processed according to the following rules;

[0034] For data records that trigger processing rules, data records will be recorded for subsequent data governance and problem handling.

[0035] Optionally, the step S6: exporting the target system standard data file specifically includes:

[0036] After the data is checked and verified, the target system generates a unified standard data file for the entire bank in a standard format based on the data model of this system and provides it to the main data system and peripheral systems that retain copies.

[0037] Optionally, the step S7: performing data backfilling operation specifically includes:

[0038] Data backfill includes: the main data source system and the peripheral reserved copy data system;

[0039] The main data source system confirms the customer information items that need to be backfilled, and then confirms the information item mapping that needs to be backfilled with the target system. After the target system is migrated, a backfill data file is generated according to the confirmed information item mapping for the main data source system to perform data backfilling.

[0040] The peripheral reserved copy data system confirms the customer information items that need to be backfilled, the target system summarizes the field information items, the peripheral reserved copy data system and the target system confirm the mapping of the customer information items that need to be backfilled, and after the target system is migrated, a backfill data file is generated according to the confirmed information item mapping for the peripheral reserved copy data system to backfill data;

[0041] Data backfilling and tagging require that the copy system be retained and the tagging be performed in each system according to the data standards of the target system.

[0042] Optionally, the incremental data migration includes:

[0043] The main data source system provides stock data;

[0044] The intermediate database performs incremental data migration and conversion;

[0045] The target system performs data migration and conversion;

[0046] The replica system backfills the incremental file. After the target system imports the incremental migration data file and overwrites the local data, it generates an incremental backfill file and sends it to each backfill system.

[0047] The present invention provides a data migration method combining full and incremental methods, the data migration method includes: step S1: preparing data migration; step S2: migrating source data; step S3: importing the migrated data file; step S4: importing the cleaned data file; step S5: checking the migrated data; step S6: exporting the target system standard data file; step S7: performing data backfilling operation. It can speed up the construction of enterprise-level systems, reduce construction costs, ensure data accuracy and business connectivity, and ensure the stable commissioning of the system.

[0048] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0050] Figure 1 A flowchart of full data migration provided by an embodiment of the present invention;

[0051] Figure 2 A flowchart of incremental data migration provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0053] The terms "comprises" and "having" and any variations thereof in the description embodiments, claims and drawings of the present invention are intended to cover non-exclusive inclusions, for example, including a series of steps or units.

[0054] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0055] A data migration method combining full and incremental methods, the data migration method comprising: full data migration and incremental data migration;

[0056] like Figure 1 As shown in the figure, full data migration includes:

[0057] 1) Data Migration Preparation

[0058] Before data migration begins, conduct research on business departments and peripheral systems related to the target system, analyze the current status of each system and the problems raised by each system, and finally determine the scope of migration. Include the information items involved in the target system into the logical model of the target system, analyze and classify the required information fields according to the logical model, and generate the physical model of the target system. Now, according to the table structure document provided by the main data source system, map and analyze the target system fields, and communicate with the source system to issue a mapping relationship.

[0059] 2) Source data migration

[0060] Before data migration, data governance of issues that hinder migration must be completed before data migration.

[0061] The data is processed initially according to the value selection and mapping rules and intermediate format files in the document "Field Value Selection and Mapping Rules"; the code value field conversion principle is carried out according to the "Code Value Mapping", and illegal code values ​​are mapped to spaces or null values; the non-code value field conversion principle is carried out according to the "Field Value Selection and Mapping Rules", such as: date format YYYY-MM-DD, field length, etc.; the data coverage principle is carried out according to the "Field Value Selection and Mapping Rules". Finally, the full amount of customer information is pushed to the target system in the form of a text file. The generated data information must meet the uniqueness of the business primary key.

[0062] 3) Import and export data files

[0063] The data of the standard data file obtained by the target system includes the full data file of the primary data source system, and then the standard data file is loaded into the local intermediate table.

[0064] 4) Import cleaning data file

[0065] The data information provided in the data cleaning phase is associated based on the business primary key to update the data information in the intermediate table of the target system.

[0066] 5) Migration data verification

[0067] Data that does not meet the requirements of the target system's physical model and data standards needs to be processed according to the following rules. Data records that trigger processing rules will be recorded for subsequent data governance and problem handling.

[0068] Field-level rules

[0069] (1) An illegal value appears in the code value field

[0070] In the case of illegal code values ​​during the data migration phase, the solution is: compare the code value mapping table, and the code values ​​that exceed the code value mapping range will be assigned an empty string.

[0071] (2) The text type field is too long

[0072] In the case of overlong fields during the data migration phase, the solution is to truncate the overlong part at the end of the data.

[0073] Sample data: The unit provident fund account field in the primary data source system customer contains overlong data: 330225470279007. The field type of the target system is VARCHAR2(12). After processing, it is 330225470279.

[0074] (3)Different precision of numeric type fields

[0075] Different precision is not allowed during the data migration phase. If the data precision of the primary data source system is more than that of the target system, rounding will be performed according to ECIF.

[0076] Sample data: The household income field type in the primary data source system is NUMBER(30,4), and the field type in the target system is NUMBER(19,2). When migrating data, it is found that the household income of a certain record is 10000.3455, and it will be processed as 10000.35 and migrated to the target system.

[0077] (4) Numeric field is too long

[0078] During the data migration phase, it is not allowed to have overlong numbers. If it does occur, the solution is to set the data to 0.

[0079] Sample data: The paid-in capital of a certain corporate customer in the primary data source system is too long. The overlong number is 91231224677455449622.0000. The field type in the target system is NUMBER(19,2). It needs to be processed when migrating to the target system. After processing, the field is set to 0.00.

[0080] (5) Non-numeric data exists in a numeric field

[0081] There are some numeric fields in the physical model of the target system, and the source system is character type. However, the source system stores non-numeric data in this field. If other non-numeric characters appear in the numeric field, they will be assigned 0.

[0082] Record-level rules

[0083] For record-level data processing, a discarded data list is generated after each migration, requiring the record of which data triggered the processing rules and resulted in not being migrated to the target system.

[0084] (1) Non-empty field processing rules

[0085] During the data migration phase, non-empty fields are not allowed to have empty values ​​(such as ID type, ID number, stock code). If this occurs, the solution is to discard the data and back up the data to a deleted file.

[0086] (2) Multiple ID numbers for the same customer and the same ID type

[0087] During the data migration phase, it is not allowed for the same customer to have multiple ID numbers of the same ID type. If this occurs, the following handling method will be used: retain the account opening ID number, discard the auxiliary ID number, and back up the data to a deleted file.

[0088] (3) Multiple auxiliary ID numbers appear for the same customer and the same ID type

[0089] During the data migration phase, it is not allowed for the same customer and the same document type to have multiple auxiliary document numbers. If this occurs, the following handling method should be used: If the document expiration date is empty, it should be processed as the current calendar day, and then the record with the latest document expiration date should be taken. Other records will not be migrated. This type of data should be available in the backup table.

[0090] 6) Export target system standard data files

[0091] After the data is checked and verified, the target system generates a unified standard data file for the entire bank in a standard format based on the data model of this system and provides it to the main data system and peripheral systems that retain copies.

[0092] 7) Data backfill

[0093] Data backfill mainly includes two parts: the main data source system and the peripheral system that retains copy data.

[0094] The main data source system confirms the customer information items (fields) that need to be backfilled in this system, and then confirms the information item mapping that needs to be backfilled with the target system. After the target system is migrated, a backfill data file is generated based on the confirmed information item mapping for the main data source system to backfill data.

[0095] The external system confirms the customer information items (fields) that need to be backfilled in the system. The target system summarizes the field information items. The external system must confirm the mapping of customer information items that need to be backfilled with the target system. After the target system is migrated, a backfill data file is generated based on the confirmed information item mapping for the external system to backfill data. Data backfilling requires that the copy system must perform backfilling in its own system according to the data standards of the target system.

[0096] like Figure 2 As shown, incremental data migration includes:

[0097] 1) The main data source system provides stock data

[0098] The main data source system provides full or incremental data for the production environment, accumulates full data in the intermediate database, and continuously maintains the full data of the source system.

[0099] 2) The intermediate database performs incremental data migration and conversion

[0100] (1) Use the full data of the primary data source system and convert it according to the data migration field mapping rules and code value mapping rules to generate the full data of the target system standard format;

[0101] (2) Generate an intermediate file of the current incremental data in the target system standard format using the current full data in the target system standard format and the previous full data in the target system standard format;

[0102] (3) The incremental data file in the target system standard format is exported from the intermediate library and provided to the target system. At the same time, the verification results are issued according to the full technical verification requirements.

[0103] 3) Data migration and conversion in the target system

[0104] The target system compares the incremental data with the full data in the database using the business unique key and performs the following processing:

[0105] (1) Update the matched full data with incremental data;

[0106] (2) For the incremental data that cannot be matched, convert and insert it according to the full migration rules;

[0107] At the same time, the verification results shall be issued according to the technical verification requirements.

[0108] 4) Backfill incremental files in the replica system

[0109] After the target system imports the incremental migration data file and overwrites the local data, it generates an incremental backfill file and sends it to each backfill system.

[0110] Beneficial effects: 1) It can reduce data problems during the switchover between the old and new systems and reduce the resistance to switching into production.

[0111] 2) It can reduce the risk of switching between the old and new systems, ensure a smooth transition between the old and new systems, and ensure business continuity.

[0112] The above specific implementation methods further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above are only specific implementation methods of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A data migration method combining full and incremental methods, characterized in that: The data migration method includes: full data migration and incremental data migration; The full data migration includes: Step S1: prepare for data migration; Step S2: Migrate source data; Step S3: import the migration data file; Step S4: importing the cleaning data file; Step S5: Migration data verification; Step S6: Exporting target system standard data files; Step S7: Perform data backfill operation.

2. The method for data migration combining full and incremental methods according to claim 1, characterized in that: The step S1: preparing for data migration specifically includes: Before data migration begins, conduct research on business departments and peripheral systems related to the target system, analyze the current status of each system and the issues raised by each system, and finally determine the scope of migration; Incorporate the information items involved in the target system into the target system logical model, analyze and classify the required information fields according to the logical model, and generate the target system physical model; Now, according to the table structure document provided by the main data source system, the target system fields are mapped and analyzed, and the mapping relationship is issued by communicating with the source system.

3. The method for data migration in a full and incremental manner according to claim 1, characterized in that: The step S2: migrating source data specifically includes: Before data migration, the main data source system must complete the data governance of the problems that hinder the migration; Perform initial data processing according to the value selection and mapping rules and intermediate format files in the document "Field Value Selection and Mapping Rules"; Code value field conversion principle: Code value field conversion is carried out according to Code Value Mapping. Illegal code values ​​are mapped to spaces or null values. The conversion principle of non-code value fields shall be carried out in accordance with the "Field Value and Mapping Rules"; the data coverage principle shall be carried out in accordance with the "Field Value and Mapping Rules"; Push all customer information to the target system in the form of text files; The generated data information must meet the uniqueness of the business primary key.

4. The method for data migration combining full and incremental methods according to claim 1, characterized in that: The step S3: importing the migration data file specifically includes: The data of the standard data file obtained by the target system includes the full data file of the primary data source system, and then the standard data file is loaded into the local intermediate table.

5. The method for data migration combining full and incremental methods according to claim 1, characterized in that: The step S4: importing the cleaning data file specifically includes: The data information provided in the data cleaning phase is associated based on the business primary key to update the data information in the intermediate table of the target system.

6. The method for data migration combining full and incremental methods according to claim 1, characterized in that: The step S5: migration data verification specifically includes: For data that does not conform to the physical model and data standard requirements of the target system, it needs to be processed according to the following rules; For data records that trigger processing rules, data records will be recorded for subsequent data governance and problem handling.

7. The method for data migration in a full and incremental manner according to claim 1, characterized in that: The step S6: exporting the target system standard data file specifically includes: After the data is checked and verified, the target system generates a unified standard data file for the entire bank in a standard format based on the data model of this system and provides it to the main data system and peripheral systems that retain copies.

8. The method for data migration combining full and incremental methods according to claim 1, characterized in that: The step S7: performing data backfilling operation specifically includes: Data backfill includes: the main data source system and the peripheral reserved copy data system; The main data source system confirms the customer information items that need to be backfilled, and then confirms the information item mapping that needs to be backfilled with the target system. After the target system is migrated, a backfill data file is generated according to the confirmed information item mapping for the main data source system to perform data backfilling. The peripheral reserved copy data system confirms the customer information items that need to be backfilled, the target system summarizes the field information items, the peripheral reserved copy data system and the target system confirm the mapping of the customer information items that need to be backfilled, and after the target system is migrated, a backfill data file is generated according to the confirmed information item mapping for the peripheral reserved copy data system to backfill data; Data backfilling and tagging require that the copy system be retained and the tagging be performed in each system according to the data standards of the target system.

9. The method for data migration combining full and incremental methods according to claim 1, characterized in that: The data migration method further includes: Incremental data migration, including: The main data source system provides stock data; The intermediate database performs incremental data migration and conversion; The target system performs data migration and conversion; The replica system backfills the incremental file. After the target system imports the incremental migration data file and overwrites the local data, it generates an incremental backfill file and sends it to each backfill system.