Business data migration method, device and system

Abstract

This invention provides a business data migration method, apparatus, and system, applicable to the financial sector or other fields. The method includes: generating a data migration task based on an acquired data migration request; acquiring the business data to be migrated according to the data migration task; copying the business data to be migrated to a data transfer table; performing data verification processing on the business data to be migrated according to preset data verification rules to obtain data verification results; performing data supplementation processing on the business data to be migrated according to the data verification results; migrating the business data to be migrated to a formal data table according to preset data mapping rules; and performing verification processing on the business data in the formal data table according to preset formal table verification rules to complete the business data migration. This invention avoids problems such as redundant development, low efficiency, time-consuming performance optimization and adjustment, and poor data quality during data migration, ensuring consistency in technical migration and business verification.

Description

Technical Field

[0001] This invention relates to the field of business data processing technology, and more particularly to a business data migration method, apparatus, and system. Background Technology

[0002] Currently, the upgrade of critical business systems in commercial banks involves migrating business data from old systems to new systems while ensuring data and business consistency. Each migration primarily achieves its migration goals by rewriting code. Repetitive projects can reuse relevant frameworks, but the mapping rules require extensive code development, and the data quality verification rules need to be redefined.

[0003] Specifically, the data migration accompanying the upgrade of commercial bank systems faces several challenges: (1) the migration time is relatively short; (2) the data migration is often large in scale, wide in scope, and has a large number of concurrent transactions, and there are many new data mappings due to differences in data models; (3) various modules are involved and there are many stakeholders; (4) many old business system data quality problems are identified during the migration process, resulting in a huge amount of related work. Summary of the Invention

[0004] In view of the problems existing in the prior art, the main objective of the embodiments of the present invention is to provide a business data migration method, apparatus and system to achieve the accuracy of business data migration and improve the efficiency of data migration.

[0005] To achieve the above objectives, embodiments of the present invention provide a business data migration method, the method comprising:

[0006] Based on the obtained data migration request, a data migration task is generated, and based on the data migration task, the business data to be migrated is obtained and copied to the data transfer table.

[0007] According to the preset data verification rules, the business data to be migrated in the data transfer table is processed for data verification to obtain the data verification results;

[0008] Based on the data verification results, data supplementation processing is performed on the business data to be migrated in the data transfer table;

[0009] According to the preset data mapping rules, the business data to be migrated after data supplementation is migrated to the formal data table, and the business data in the formal table is verified according to the preset formal table verification rules to complete the business data migration.

[0010] Optionally, in one embodiment of the present invention, the method further includes:

[0011] The data migration task is decoupled to obtain multiple migration subtasks, and the task lineage relationship between each migration subtask is determined.

[0012] Based on task lineage, the migration subtasks are scheduled.

[0013] Optionally, in one embodiment of the present invention, obtaining the business data to be migrated according to the data migration task and copying the business data to be migrated to the data transfer table includes:

[0014] Based on the migration subtasks after task scheduling, obtain the business data to be migrated and copy it to the data transfer table.

[0015] Optionally, in one embodiment of the present invention, data verification processing is performed on the business data to be migrated in the data transfer table according to preset data verification rules, and the data verification results include:

[0016] According to the preset data verification rules, the business data to be migrated in the data transfer table is checked for data format, data length, range, default value, integrity and consistency, and the data verification results are obtained.

[0017] This invention also provides a business data migration apparatus, the apparatus comprising:

[0018] The data transfer table module is used to generate data migration tasks based on the obtained data migration requests, obtain the business data to be migrated based on the data migration tasks, and copy the business data to be migrated to the data transfer table.

[0019] The data verification module is used to perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules, and obtain the data verification results.

[0020] The data entry module is used to perform data entry processing on the business data to be migrated in the data transfer table based on the data verification results.

[0021] The data migration module is used to migrate the business data to be migrated after data supplementation processing to the formal data table according to the preset data mapping rules, and to perform verification processing on the business data in the formal table according to the preset formal table verification rules to complete the business data migration.

[0022] Optionally, in one embodiment of the present invention, the apparatus further includes:

[0023] The task decoupling module is used to decouple the data migration task, obtain multiple migration subtasks, and determine the task lineage relationship between each migration subtask.

[0024] The task scheduling module is used to schedule migration subtasks based on task lineage.

[0025] Optionally, in one embodiment of the present invention, the transfer table module is further configured to obtain the business data to be migrated according to the migration subtasks after task scheduling, and copy the business data to be migrated to the data transfer table.

[0026] Optionally, in one embodiment of the present invention, the data verification module is further configured to perform data format checks, data length checks, range checks, default value checks, integrity checks, and consistency checks on the business data to be migrated in the data transfer table according to preset data verification rules, so as to obtain data verification results.

[0027] This invention also provides a business data migration system, which includes: a data access module, a data transfer module, and a data output module;

[0028] The data access module is used to obtain the business data to be migrated according to the data migration task;

[0029] The data transfer module is used to generate data migration tasks based on the acquired data migration requests; copy the business data to be migrated to the data transfer table; perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules to obtain the data verification results; and perform data supplementation processing on the business data to be migrated in the data transfer table based on the data verification results.

[0030] The data output module is used to migrate the business data to be migrated after data supplementation processing to the formal data table according to the preset data mapping rules, and to perform verification processing on the business data in the formal table according to the preset formal table verification rules to complete the business data migration.

[0031] Optionally, in one embodiment of the present invention, the data transfer module includes: a data transfer table submodule, an original data verification submodule, a business data supplementation submodule, a migration task scheduling submodule, and a data mapping configuration submodule;

[0032] The data transfer table submodule is used to copy the business data to be migrated to the data transfer table;

[0033] The original data verification submodule is used to perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules, and obtain the data verification results;

[0034] The business data supplementation submodule is used to supplement the business data to be migrated in the data transfer table based on the data verification results.

[0035] The migration task scheduling submodule is used to generate data migration tasks based on the acquired data migration requests; decouple the data migration tasks to obtain multiple migration subtasks and determine the task lineage between each migration subtask; and schedule the migration subtasks according to the task lineage.

[0036] The data mapping configuration submodule is used to store the data mapping rules.

[0037] Optionally, in one embodiment of the present invention, the system further includes a monitoring and statistics module for monitoring the business data migration process and the execution status of data migration tasks, and storing data verification results.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0039] The present invention also provides a computer-readable storage medium storing a computer program that performs the above-described methods by a computer.

[0040] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described method.

[0041] This invention utilizes data migration tasks, combined with data verification rules, to improve the accuracy and efficiency of business data migration. It avoids problems such as repetitive development, low efficiency, time-consuming performance optimization and adjustment, and poor data quality caused by business system upgrades and replacements during the data migration process, ensuring consistency in technology migration and business verification. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a flowchart of a business data migration method according to an embodiment of the present invention;

[0044] Figure 2 This is a flowchart of task scheduling in an embodiment of the present invention;

[0045] Figure 3 This is a schematic diagram of the structure of a business data migration system according to an embodiment of the present invention;

[0046] Figure 4This is a schematic diagram of the data transfer module in an embodiment of the present invention;

[0047] Figure 5 This is a schematic diagram of task decoupling in an embodiment of the present invention;

[0048] Figure 6 This is a schematic diagram of the structure of a business data migration device according to an embodiment of the present invention;

[0049] Figure 7 This is a schematic diagram of the structure of a business data migration device in another embodiment of the present invention;

[0050] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0051] This invention provides a business data migration method, apparatus, and system that can be used in the financial field and other fields. It should be noted that the business data migration method, apparatus, and system of this invention can be used in the financial field, or in any field other than the financial field. The application fields of the business data migration method, apparatus, and system of this invention are not limited.

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] like Figure 1 The diagram shows a flowchart of a business data migration method according to an embodiment of the present invention. The execution subject of the business data migration method provided in this embodiment includes, but is not limited to, a computer. The present invention utilizes data migration tasks, combined with data verification rules, to improve the data accuracy and migration efficiency of the business data migration process. It avoids problems such as redundant development, low efficiency, time-consuming performance optimization and adjustment, untimely migration statistics, and poor data quality caused by business system upgrades and replacements, ensuring consistency in technical migration and business verification. The method shown in the diagram includes:

[0054] Step S1: Generate a data migration task based on the obtained data migration request, and obtain the business data to be migrated based on the data migration task, and copy the business data to be migrated to the data transfer table.

[0055] Data migration requests are initiated by business personnel or the business system itself when a business system undergoes replacement or upgrade. Specifically, a data migration request may include basic information about the old business system to be migrated and basic information about the target system (i.e., the new business system), such as system name and address. Therefore, upon receiving a data migration request, based on the basic information of the old and new business systems to be migrated, the information of the data to be migrated, such as data size and format, can be determined. Consequently, in response to the data migration request, a corresponding data migration task is generated.

[0056] Furthermore, the data migration task includes multiple migration subtasks, which have a task lineage relationship. Specifically, the task lineage relationship represents the association between tasks, and the task lineage relationship can be preset or automatically generated based on the data migration request.

[0057] Furthermore, after generating the data migration task, its execution is achieved by decoupling and scheduling the task. Specifically, executing the data migration task first requires acquiring the business data to be migrated and copying it to a pre-defined data transfer table.

[0058] Specifically, the data transfer table is stored in a database; ideally, a TIDB database could be used.

[0059] Step S2: According to the preset data verification rules, perform data verification processing on the business data to be migrated in the data transfer table to obtain the data verification result.

[0060] Among them, the data migration task is used to perform data verification processing on the business data to be migrated in the data transfer table according to the preset verification rules.

[0061] Furthermore, the verification rules include data format checks, data length checks, range checks, default value / null value checks, integrity checks, and consistency checks. Specifically, based on the requirements of each subtask in the data migration task, such as what types of data need to be checked for range, the business data to be migrated is verified, and the corresponding data verification results are obtained.

[0062] Step S3: Based on the data verification results, perform data supplementation processing on the business data to be migrated in the data transfer table.

[0063] If the data verification results indicate that the business data to be migrated is missing, then the business data to be migrated in the data transfer table will be supplemented. Specifically, during the data migration process, business data may have missing fields, and the data supplementation process can be automatically completed according to the preset data supplementation template.

[0064] Furthermore, if the data verification results indicate that there is no missing data in the business data to be migrated, then the data entry process will be no data entry required.

[0065] Furthermore, if the data validation results include other issues, such as incorrect data length or format, these can be handled according to pre-defined processing rules. Specifically, if the errors in the data validation results are common errors, data format conversion can be performed directly to resolve such issues. If the errors are not common errors, the results can be sent to relevant personnel for manual resolution.

[0066] Step S4: According to the preset data mapping rules, the business data to be migrated after data supplementation is migrated to the formal data table, and the business data in the formal table is verified according to the preset formal table verification rules to complete the business data migration.

[0067] After the business data has undergone data verification and data entry, the business data to be migrated from the data transfer table to the formal data table is migrated according to the preset data mapping rules. Specifically, the data mapping rules are the specific rules for migrating from the old business system to the new business system, and the specific rules for migration can be set through preset configuration.

[0068] Furthermore, the business data in the official data table is validated according to the preset official table validation rules.

[0069] Specifically, the validation rules for the official tables are similar to those for the data validation rules, including data format checks and range checks. However, the validation rules for the official tables are set based on the business requirements of the new business system, therefore the specific rule requirements differ from those of the data validation rules for the old business system. After completing the business data validation process in the official data tables, if the validation result is successful, the data migration process is complete. If the validation result is unsuccessful, a detailed reason for the failure is generated and sent to the relevant business personnel for timely processing.

[0070] As one embodiment of the present invention, such as Figure 2 As shown, the method also includes:

[0071] Step S21: Decouple the data migration task to obtain multiple migration subtasks and determine the task lineage relationship between each migration subtask.

[0072] Step S22: Schedule the migration subtasks according to their task lineage.

[0073] The data migration task comprises multiple migration subtasks, which are related by a task lineage. Specifically, the task lineage represents the relationship between tasks, and can be preset or automatically generated based on the data migration request.

[0074] Furthermore, the scheduling of migration subtasks can be based on load conditions, specific task requirements, and other factors.

[0075] In this embodiment, obtaining the business data to be migrated according to the data migration task and copying the business data to be migrated to the data transfer table includes: obtaining the business data to be migrated according to the migration sub-tasks after task scheduling and copying the business data to be migrated to the data transfer table.

[0076] Furthermore, after generating the data migration task, its execution is achieved by decoupling and scheduling the task. Specifically, executing the data migration task first requires acquiring the business data to be migrated and copying it to a pre-defined data transfer table.

[0077] As an embodiment of the present invention, according to preset data verification rules, data verification processing is performed on the business data to be migrated in the data transfer table to obtain data verification results. This includes: according to preset data verification rules, data format checks, data length checks, range checks, default value checks, integrity checks, and consistency checks are performed on the business data to be migrated in the data transfer table to obtain data verification results.

[0078] The verification rules include data format checks, data length checks, range checks, default value / null value checks, integrity checks, and consistency checks. Specifically, based on the requirements of each subtask in the data migration task, such as what types of data require range checks, the business data to be migrated is verified, and the corresponding data verification results are obtained.

[0079] This invention utilizes data migration tasks, combined with data verification rules, to improve the accuracy and efficiency of business data migration. It avoids problems such as redundant development, low efficiency, time-consuming performance optimization and adjustment, untimely migration statistics, and poor data quality caused by business system upgrades and replacements during the data migration process, thus ensuring consistency in technical migration and business verification.

[0080] like Figure 3 The figure shows a schematic diagram of a business data migration system according to an embodiment of the present invention. The system shown in the figure includes: a data access module, a data transfer module and a data output module.

[0081] The data access module is used to obtain the business data to be migrated according to the data migration task;

[0082] The data transfer module is used to generate a data migration task based on the acquired data migration request; copy the business data to be migrated to the data transfer table, and perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules to obtain the data verification result; and perform data supplementation processing on the business data to be migrated in the data transfer table based on the data verification result.

[0083] The data output module is used to migrate the business data to be migrated after data supplementation processing to the formal data table according to the preset data mapping rules, and to perform verification processing on the business data in the formal table according to the preset formal table verification rules to complete the business data migration.

[0084] This invention's business data migration system primarily addresses issues encountered during data migration following system upgrades in commercial banks, such as redundant development, low efficiency, time-consuming performance optimization and adjustments, untimely migration statistics, and difficulty in identifying data quality problems. It modularizes and productizes data migration, avoiding excessive focus on framework selection and instead emphasizing data migration rules and the rectification of identified data quality issues. This ensures data migration is completed within project timelines, guaranteeing consistency in technical migration and business data verification.

[0085] As one embodiment of the present invention, such as Figure 4 As shown, the data transfer module includes: a data transfer table submodule, a raw data verification submodule, a business data supplementation submodule, a migration task scheduling submodule, and a data mapping configuration submodule;

[0086] The data transfer table submodule is used to copy the business data to be migrated into the data transfer table;

[0087] The original data verification submodule is used to perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules, and obtain the data verification results;

[0088] The business data supplementation submodule is used to supplement the business data to be migrated in the data transfer table based on the data verification results.

[0089] The migration task scheduling submodule is used to generate data migration tasks based on the acquired data migration requests; decouple the data migration tasks to obtain multiple migration subtasks and determine the task lineage between each migration subtask; and schedule the migration subtasks according to the task lineage.

[0090] The data mapping configuration submodule is used to store data mapping rules.

[0091] As an embodiment of the present invention, the system further includes a monitoring and statistics module for monitoring the business data migration process and the execution status of data migration tasks, and storing data verification results.

[0092] In this embodiment, as Figure 3 The data migration system shown includes a data access module, a data transfer module, a data output module, and a monitoring and statistics module, which are described in detail below.

[0093] 1. Data Access Module: The main function of data access is to import relevant data from the old business system, preparing the data foundation for subsequent mapping configurations. The database used by the data access module is consistent with the database type used by the old credit business system. Data access is configured in two ways:

[0094] 1) Dump package loading: This method primarily involves exporting old credit business data into a dump package for rapid loading. This enables snapshot recovery of relevant point-in-time data from the old credit system.

[0095] Here, "dump" refers to the backup file system. Most mainstream databases have this function to back up existing data and copy existing database-related storage from primary or secondary storage to external storage.

[0096] 2) Loading files exported from old business systems: This mainly involves loading text files generated by scripts from old credit systems and related data.

[0097] Neither of the above two methods is particularly necessary; the choice depends on the specific hardware resources available for the project implementation. The data access module facilitates the data migration preparation for the old business system. Consequently, the relevant data has been imported into the source tables of the old credit system.

[0098] 2. Data Transfer Module: The system uses the data transfer module to perform functions such as verification of business data from old business systems, remapping and matching of heterogeneous data, addition of supplementary data, migration scheduling, monitoring and statistics.

[0099] 1) Data Transfer Table Submodule: This submodule copies the source tables from the old credit system to the database specific to this invention's system. It primarily uses domestic databases (TIDB or DM Database), and all related data storage and processing can be performed based on this domestic database. Simultaneously, the Data Transfer Table submodule unifies the collection of data to be transferred, enabling the unified database to handle heterogeneous data sources from the business systems. Thus, the relevant business data has been imported into the Data Transfer Table.

[0100] 2) Original Data Validation Submodule: The original data validation submodule is used to validate the business data of the old credit system in the data transfer table. The validation rules are mainly reflected in data format check, data length check, range check, null / default value check, integrity check, and consistency check.

[0101] Furthermore, the system of this invention completes the collection of relevant business-independent verification rules, forming a verification rule base (fields such as date and ID card). Simultaneously, the verification rule base implements rigid / flexible control marking. Data that does not conform to rigid control rules is not migrated; data that does not conform to flexible control rules is marked but allowed to migrate.

[0102] It also has an automatic recognition function, which can perform preliminary recognition of initial data based on relevant templates (ID card length, mobile phone number length, name, etc.), confirm the verification range, provide it to operators for judgment, avoid missing verification data, and save manual configuration of data verification, thus improving efficiency.

[0103] Integrity checks refer to verifying the integrity of data relationships, such as whether a record references a code value. It's particularly important to note that some systems remove foreign key constraints after a period of use to improve efficiency. Consistency checks refer to verifying whether there is any logically inconsistent data.

[0104] 3) Business Data Completion Submodule: Addressing the potential for missing fields in the old business system during data migration between new and old systems, this system enables separate data completion. It supports both Excel and text loading. Missing fields are linked using primary key fields during the data migration process. Based on data validation, it identifies the data to be completed, automatically generates an Excel template (with a primary key), and provides this template for business personnel to complete the data entry. The system loads the Excel template, automatically links fields based on the primary key, and completes the business data completion. Versioning can be configured to record changes in the completed data.

[0105] 4) Migration Task Scheduling Submodule: The migration task scheduling submodule mainly implements task configuration, resource allocation, and task scheduling. The task scheduling design considers scenarios such as task decoupling, task retries, and breakpoint continuation.

[0106] A. Task Configuration Section: Based on a four-layer design, namely, job name (unique), module name, stage name, and task name, such as... Figure 5 As shown.

[0107] Among them, prerequisite dependencies can be set between modules, stages, and tasks to realize task lineage; and priorities (high, medium, low) can be set.

[0108] Furthermore, the task scheduling design considers scenarios such as task decoupling, task retries, and breakpoint continuation.

[0109] (1) Once each task is decoupled, after a single task is re-executed, subsequent dependent tasks will be updated synchronously based on task lineage.

[0110] (2) Supports single task and multiple tasks to be executed repeatedly, and related data is automatically cleaned up to ensure that no dirty data is missed under non-automatic scheduling.

[0111] (3) Supports the retention of the abnormal status of the interrupted task after the scheduling execution stops abnormally. After restarting, the abnormal task is rescheduled and the subsequent dependent tasks are automatically executed based on the task lineage, instead of starting the scheduling execution from scratch after the scheduling stops abnormally.

[0112] B. Task scheduling

[0113] a) Based on the concurrent scheduling task settings, the database can support the maximum performance evaluation, with the maximum concurrent task tentatively set at 10;

[0114] b) Based on task sorting, when a task is invoked, first determine if the current task is within the maximum concurrency of 10 or if there are tasks with higher priority waiting. If so, set it to a waiting state and poll it after 1 minute. If not, proceed. For waiting tasks, high-priority tasks maintain their high-priority order, and low-priority tasks maintain their low-priority order in the queue.

[0115] C. Resource allocation

[0116] The system is linked to a database monitoring tool. In concurrent task scenarios, if database performance exceeds a set warning threshold, the number of concurrent tasks is reduced in a two-step manner. This adjustment is performed every minute.

[0117] 5) Data Mapping Configuration Submodule: This submodule implements batch remapping of mapping rules for heterogeneous modules between old and new business systems. It mainly includes mapping rule configuration functionality, metadata dictionary viewing functionality, and version management functionality. The mapping rule configuration specifies the rules for migrating from the old business system to the new business system. All rule configurations are based on visual operations.

[0118] Metadata is data that describes data, mainly information that describes data attributes, and is used to support things like indicating data storage location, historical data, and resource lookup.

[0119] 6) Database optimization submodule: Implements migration table-level analysis to ensure the frequency of table transfers in the old business system and supports table copying for tables with high dependencies.

[0120] 3. Data Output Module

[0121] 1) Official table insertion: Based on migration preparation, after the data transfer table completes data verification and passes, the data is inserted into the official table (new business system table) based on the mapping rules.

[0122] 2) Formal table validation: For the migrated data, a rule base is formed according to the field requirements of the new business system, validation is completed, and relevant data that does not conform to the operation of the business system is identified.

[0123] 4. Monitoring and Statistics Module

[0124] (1) Verification results: The verification results of the transit form / formal form can be viewed online.

[0125] (2) Data migration volume verification: Based on the mapping rules, the data migration volume verification is completed.

[0126] (3) Business verification: Based on the configuration script, the data business fields of the old and new systems are verified (summary of amount, summary of interest, etc.).

[0127] (4) Performance monitoring: Implement overall monitoring of database performance and server performance during the data migration task scheduling process.

[0128] (5) Task lineage view: Based on the task lineage, you can view the overall relationship and view the overall task execution status on the task lineage chart.

[0129] In addition, the system of the present invention also includes a system management module for user management: to implement user access control, and to allow development teams and business personnel supported by data migration to log in to this device to view relevant information, and to allow specific developers to configure development scheduling for data migration.

[0130] This invention relates to a business data migration system. Through the encapsulation and assembly of relevant modules, it enables online configuration and development for data migration, completing the platform setup. This allows developers to focus on verifying the validity of relevant mapping rules and identifying data quality issues during the business system data migration process, avoiding excessive time spent on repetitive framework building. The establishment of this platform can significantly reduce the cost of repetitive investment in multiple projects, efficiently and quickly achieving business objectives.

[0131] like Figure 6 The figure shows a schematic diagram of a business data migration device according to an embodiment of the present invention. The device shown in the figure includes:

[0132] The transfer table module 10 is used to generate a data migration task based on the obtained data migration request, and to obtain the business data to be migrated based on the data migration task, and copy the business data to be migrated to the data transfer table.

[0133] The data verification module 20 is used to perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules, and obtain the data verification result.

[0134] The data entry module 30 is used to perform data entry processing on the business data to be migrated in the data transfer table based on the data verification results.

[0135] The data migration module 40 is used to migrate the business data to be migrated after data supplementation processing to the formal data table according to the preset data mapping rules, and to perform verification processing on the business data in the formal table according to the preset formal table verification rules to complete the business data migration.

[0136] As one embodiment of the present invention, such as Figure 7 As shown, the device also includes:

[0137] The task decoupling module 50 is used to decouple the data migration task, obtain multiple migration sub-tasks, and determine the task lineage relationship between each migration sub-task.

[0138] The task scheduling module 60 is used to schedule migration subtasks according to task lineage.

[0139] In this embodiment, the transfer table module is also used to obtain the business data to be migrated according to the migration sub-tasks after task scheduling, and copy the business data to be migrated to the data transfer table.

[0140] As an embodiment of the present invention, the data verification module is also used to perform data format checks, data length checks, range checks, default value checks, integrity checks, and consistency checks on the business data to be migrated in the data transfer table according to preset data verification rules, so as to obtain data verification results.

[0141] Based on the same concept as the aforementioned business data migration method and system, this invention also provides a business data migration apparatus. Since the principle by which this business data migration apparatus solves the problem is similar to that of the aforementioned business data migration method and system, the implementation of this business data migration apparatus can refer to the implementation of the aforementioned business data migration method and system; details that are repeated will not be elaborated upon.

[0142] This invention utilizes data migration tasks, combined with data verification rules, to improve the accuracy and efficiency of business data migration. It avoids problems such as redundant development, low efficiency, time-consuming performance optimization and adjustment, untimely migration statistics, and poor data quality caused by business system upgrades and replacements during the data migration process, thus ensuring consistency in technical migration and business verification.

[0143] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method.

[0144] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described method.

[0145] The present invention also provides a computer-readable storage medium storing a computer program that performs the above-described methods by a computer.

[0146] like Figure 8 As shown, the electronic device 600 may also include: a communication module 110, an input unit 120, an audio processor 130, a display 160, and a power supply 170. It is worth noting that the electronic device 600 does not necessarily need to include these components. Figure 8 All components shown; in addition, the electronic device 600 may also include Figure 8 For components not shown, please refer to existing technologies.

[0147] like Figure 8 As shown, the central processing unit 100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device. The central processing unit 100 receives inputs and controls the operation of various components of the electronic device 600.

[0148] The memory 140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 100 may execute the program stored in the memory 140 to perform information storage or processing, etc.

[0149] Input unit 120 provides input to central processing unit 100. Input unit 120 may be, for example, a keypad or touch input device. Power supply 170 provides power to electronic device 600. Display 160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0150] The memory 140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 140 can also be some other type of device. The memory 140 includes a buffer memory 141 (sometimes referred to as a buffer). The memory 140 may include an application / function storage unit 142 for storing application programs and function programs or processes for executing the operation of the electronic device 600 via the central processing unit 100.

[0151] The memory 140 may also include a data storage unit 143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 144 of the memory 140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0152] The communication module 110 is a transmitter / receiver 110 that transmits and receives signals via antenna 111. The communication module (transmitter / receiver) 110 is coupled to the central processing unit 100 to provide input signals and receive output signals, which can be the same as in a conventional mobile communication terminal.

[0153] Based on different communication technologies, multiple communication modules 110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module (transmitter / receiver) 110 is also coupled to a speaker 131 and a microphone 132 via an audio processor 130 to provide audio output via the speaker 131 and receive audio input from the microphone 132, thereby enabling typical telecommunications functions. The audio processor 130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 130 is coupled to a central processing unit 100, enabling on-device recording via the microphone 132 and on-device playback of stored audio via the speaker 131.

[0154] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0156] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0157] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0158] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method of service data migration, characterized by, The method includes: Based on the obtained data migration request, a data migration task is generated, and based on the data migration task, the business data to be migrated is obtained and copied to the data transfer table. According to the preset data verification rules, the business data to be migrated in the data transfer table is subjected to data verification processing to obtain the data verification result; Based on the data verification results, the data to be migrated in the data transfer table is supplemented. According to the preset data mapping rules, the business data to be migrated after data supplementation is migrated to the formal data table, and the business data in the formal table is verified according to the preset formal table verification rules to complete the business data migration. The method further includes: decoupling the data migration task to obtain multiple migration sub-tasks, and determining the task lineage relationship between each migration sub-task; and scheduling the migration sub-tasks according to the task lineage relationship. Based on the data verification results, data supplementation processing is performed on the business data to be migrated in the data transfer table, including: if the data verification results indicate that the business data to be migrated is missing, then data supplementation processing is performed on the business data to be migrated in the data transfer table; if the data verification results also include other situations besides the missing data, then processing is performed according to preset processing rules.

2. The method of claim 1, wherein, According to the data migration task, obtaining the business data to be migrated and copying the business data to be migrated to the data transfer table includes: Based on the migration subtasks after task scheduling, obtain the business data to be migrated and copy the business data to be migrated to the data transfer table.

3. The method according to claim 1, characterized in that, The step of performing data verification processing on the business data to be migrated in the data transfer table according to preset data verification rules, and obtaining the data verification results includes: According to the preset data verification rules, the business data to be migrated in the data transfer table is checked for data format, data length, range, default value, integrity and consistency, and the data verification results are obtained.

4. A service data migration apparatus characterized by comprising: The device includes: The data transfer table module is used to generate a data migration task based on the obtained data migration request, obtain the business data to be migrated based on the data migration task, and copy the business data to be migrated to the data transfer table. The data verification module is used to perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules, and obtain the data verification result. The data entry module is used to perform data entry processing on the business data to be migrated in the data transfer table based on the data verification results. The data migration module is used to migrate the business data to be migrated after data supplementation processing to the formal data table according to the preset data mapping rules, and to perform verification processing on the business data in the formal table according to the preset formal table verification rules to complete the business data migration. The device further includes: decoupling the data migration task to obtain multiple migration sub-tasks, and determining the task lineage relationship between each migration sub-task; and scheduling the migration sub-tasks according to the task lineage relationship. Based on the data verification results, data supplementation processing is performed on the business data to be migrated in the data transfer table, including: if the data verification results indicate that the business data to be migrated is missing, then data supplementation processing is performed on the business data to be migrated in the data transfer table; if the data verification results also include other situations besides the missing data, then processing is performed according to preset processing rules.

5. A service data migration system, characterized by, The system includes: a data access module, a data transfer module, and a data output module; The data access module is used to obtain the business data to be migrated according to the data migration task; The data transfer module is used to generate a data migration task based on the acquired data migration request; copy the business data to be migrated to the data transfer table, and perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules to obtain the data verification result; and perform data supplementation processing on the business data to be migrated in the data transfer table based on the data verification result. The data output module is used to migrate the business data to be migrated after data supplementation processing to the formal data table according to the preset data mapping rules, and to perform verification processing on the business data in the formal table according to the preset formal table verification rules to complete the business data migration. The data transfer module includes a migration task scheduling submodule, which is used to decouple the data migration task to obtain multiple migration subtasks and determine the task lineage relationship between each migration subtask; and to schedule the migration subtasks according to the task lineage relationship. Based on the data verification results, data supplementation processing is performed on the business data to be migrated in the data transfer table, including: if the data verification results indicate that the business data to be migrated is missing, then data supplementation processing is performed on the business data to be migrated in the data transfer table; if the data verification results also include other situations besides the missing data, then processing is performed according to preset processing rules.

6. The system of claim 5, wherein, The data transfer module includes: a data transfer table submodule, a raw data verification submodule, a business data supplementation submodule, and a data mapping configuration submodule; The data transfer table submodule is used to copy the business data to be migrated into the data transfer table; The original data verification submodule is used to perform data verification processing on the business data to be migrated in the data transfer table according to the preset data verification rules, and obtain the data verification result; The business data supplementation submodule is used to perform data supplementation processing on the business data to be migrated in the data transfer table according to the data verification result; The migration task scheduling submodule is also used to generate data migration tasks based on the acquired data migration requests; The data mapping configuration submodule is used to store the data mapping rules.

7. The system of claim 5, wherein, The system also includes a monitoring and statistics module, which is used to monitor the business data migration process and the execution status of data migration tasks, and to store the data verification results.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 3.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that enables a computer to execute the method according to any one of claims 1 to 3.

Images