Data migration processing method, device and computer-readable storage medium

Abstract

The present invention discloses a data migration processing method, device and computer-readable storage medium. The method includes: obtaining a migration customer list, wherein the migration customer list includes at least the account information of the migration customer; initiating a locking process based on the migration customer list, wherein the locking process includes at least: locking the customer information platform, global routing and source system in sequence; processing the source data exported from the source system according to the account information of the migration customer to obtain target data, and importing the target data into the target system, wherein the target data is data that conforms to the data model structure of the target system; initiating an unlocking process based on the migration customer list, wherein the unlocking process includes at least: unlocking the source system, global routing and customer information platform in sequence. The present invention solves the technical problem in the prior art that it is impossible to achieve switching between the old and new systems and data migration without stopping the external service of the real-time transaction system.

Description

Technical Field

[0001] The present invention relates to the field of data migration, and in particular to a data migration processing method, device and computer-readable storage medium. Background Art

[0002] As customer numbers and transaction volumes continue to climb, outdated business systems will become a bottleneck restricting business development. Not only do the old and new systems involve numerous interconnected systems, requiring extensive data migration and voluminous data volumes, but their construction at different times also results in significant differences in data models and complex mapping relationships. The rollout of the new system requires migrating massive amounts of data while ensuring that real-time customer transactions are not impacted.

[0003] Existing data migration techniques using Oracle's OWB (Oracle Warehouse Builder) product component can load data from other databases into the data warehouse after cleansing and conversion. However, OWB cannot achieve real-time data migration. Furthermore, by recording customer information that has changed, data synchronization transactions are performed through the source database's transaction system based on this recorded customer information. This data synchronization is then used to synchronize data in real time with the target database's transaction system. However, this approach cannot achieve the migration of massive amounts of data.

[0004] Therefore, existing technologies cannot achieve switching between the old and new systems and data migration without stopping the real-time trading system's external services.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0006] Embodiments of the present invention provide a data migration processing method, device, and computer-readable storage medium to at least solve the technical problem in the prior art of being unable to implement switching between old and new systems and data migration without stopping the real-time trading system from providing external services.

[0007] According to one aspect of an embodiment of the present invention, a method for processing data migration is provided, comprising: obtaining a migration customer list, wherein the migration customer list includes at least the account information of the migration customer; initiating a locking process based on the migration customer list, wherein the locking process includes at least: locking the customer information platform, the global routing, and the source system in sequence; processing the source data exported from the source system according to the account information of the migration customer to obtain target data, and importing the target data into the target system, wherein the target data is data that conforms to the data model structure of the target system; initiating an unlocking process based on the migration customer list, wherein the unlocking process includes at least: unlocking the source system, the global routing, and the customer information platform in sequence.

[0008] Optionally, obtaining the migration customer list includes: obtaining the migration customer list from a related system according to a customer activity list, wherein the customer activity list includes customers and activities associated with the customers.

[0009] Optionally, before obtaining the migrated customer list from the associated system based on the customer activity list, the method further includes: sending the non-migrated customer list to the big data platform to calculate the activity of the non-migrated customers to obtain a calculation result; obtaining the calculation result from the big data platform, and updating the customer activity list based on the calculation result.

[0010] Optionally, the data processing includes at least: data removal, data conversion, data verification and data import, wherein the data removal is to unload the source data of the source system and transfer the source data to the data conversion environment; the data conversion is to convert the source data exported from the source system into the target data of the target system according to the conversion relationship in the data mapping table; the data verification is to compare the target data with the source data for data accuracy and consistency, and verify them according to data quality standards; the data import is to load the verified target data into the target system in batch migration or online migration.

[0011] Optionally, in the process of processing the source data exported from the source system to obtain target data based on the account information of the migrating customer, and importing the target data into the target system, the method also includes: managing the data migration process, and / or monitoring the data migration process.

[0012] Optionally, managing the data migration process includes at least one of the following: managing migration batches and the rules for generating the migration batches; configuring the operating parameters required by the task scheduler and querying the task operation status; managing message interactions between systems; managing any one or more of operating users, role permissions, system logs, and user login / logout.

[0013] Optionally, monitoring the data migration process includes at least one of the following: monitoring the migration progress during the data migration process; and monitoring migration exception information during the data migration process.

[0014] According to another aspect of an embodiment of the present invention, a data migration processing device is provided, comprising: a first acquisition module for acquiring a migration customer list, wherein the migration customer list includes at least the account information of the migration customer; a locking module for initiating a locking process according to the migration customer list, wherein the locking process includes at least: locking the customer information platform, the global routing and the source system in sequence; a migration module for processing the source data exported from the source system according to the account information of the migration customer to obtain target data, and importing the target data into the target system, wherein the target data is data that conforms to the data model structure of the target system; an unlocking module for initiating an unlocking process according to the migration customer list, wherein the unlocking process includes at least: unlocking the source system, the global routing and the customer information platform in sequence.

[0015] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, including: the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the data migration processing methods.

[0016] According to yet another aspect of an embodiment of the present invention, a processor is provided, which is configured to run a program, wherein the program executes any one of the data migration processing methods when running.

[0017] In an embodiment of the present invention, in the data migration processing method, first, a migration customer list is obtained, wherein the migration customer list at least includes the account information of the migration customer; then, a locking process is initiated based on the obtained migration customer list, and thereafter, the source data exported from the source system is processed according to the account information of the migration customer to obtain target data, and the target data is imported into the target system; finally, an unlocking process is sent according to the migration customer list, wherein the locking process at least includes: locking the customer information platform, the global routing and the source system in sequence, and the unlocking process at least includes: unlocking the source system, the global routing and the customer information platform in sequence. In this solution, a locking process is initiated based on the obtained migration customer list, and then the source data exported from the source system is processed to obtain the target data based on the account information of the migration customer, and the target data is imported into the target system. Finally, an unlocking process is sent based on the migration customer list. That is, this solution ensures the consistency of the migration customer status and the consistency of the data during the data migration process through the locking and unlocking processes. In addition, this solution sends the locking process and the unlocking process based on the migration customer list, thereby realizing the migration of massive data without stopping the external service of the real-time trading system, thereby solving the technical problem in the existing technology that it is impossible to realize the switching of the old and new systems and data migration without stopping the external service of the real-time trading system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0019] Figure 1 is a flowchart of a data migration processing method according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a data migration system according to an embodiment of the present invention;

[0021] Figure 3 is a flowchart of migrating customer data according to an embodiment of the present invention;

[0022] Figure 4 is a flowchart of migration task scheduling according to an embodiment of the present invention;

[0023] Figure 5 FIG. 4 is a schematic diagram of a data migration processing device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0027] Data migration: Smoothly migrate the business and technical information carried by the data in the source system to the target system, and ensure that the business meaning of all customer service data before and after the migration remains unchanged;

[0028] Pre-migration: refers to migrating all data at once before going online;

[0029] Incremental migration: This involves migrating existing data in advance and then migrating incremental data generated daily during the transition period to reduce the pressure of online migration.

[0030] Online migration: Online migration of contracted customer data and non-final register data to the new system (i.e., target system);

[0031] Delayed migration: refers to migrating all data to the new system at once after the switch transition is completed;

[0032] Passive migration: This occurs during the transition period, when transactions such as changing the account name, changing real-name IDs, or migrating primary and secondary cardholders result in a change in the customer to whom the credentials belong, and one customer is currently in the old system while the other has been migrated to the new system. At this point, a passive migration is initiated, obtaining a list of customers to be migrated, grouping them into batches, and prioritizing their migration and importing them into the new system.

[0033] Task scheduler: used to load batch information generated by the control subsystem and schedule customer data;

[0034] Task Executor: Exports, converts, verifies, and imports customer data in migration batches.

[0035] According to an embodiment of the present invention, an embodiment of a data migration processing method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0036] Figure 1 This is a data migration processing method according to an embodiment of the present invention, such as Figure 1 As shown, the method includes the following steps:

[0037] Step S102: Acquire a list of migrated customers, wherein the list of migrated customers at least includes account information of the migrated customers;

[0038] Step S104: Initiate a locking process based on the migration customer list, wherein the locking process at least includes: sequentially locking the customer information platform, global routing, and source system;

[0039] Step S106 , processing the source data exported from the source system to obtain target data based on the account information of the migrating customer, and importing the target data into the target system, wherein the target data conforms to the data model structure of the target system;

[0040] Step S108 : Initiating an unlocking process according to the migration customer list, wherein the unlocking process at least includes: sequentially unlocking the source system, the global routing, and the customer information platform.

[0041] In the above-mentioned data migration processing method, first, a migration customer list is obtained, wherein the migration customer list at least includes the account information of the migration customer; then, a locking process is initiated based on the obtained migration customer list, and thereafter, the source data exported from the above-mentioned source system is processed according to the account information of the above-mentioned migration customer to obtain target data, and the above-mentioned target data is imported into the target system; finally, an unlocking process is sent according to the above-mentioned migration customer list, wherein the above-mentioned locking process at least includes: locking the customer information platform, the global routing and the source system in sequence, and the above-mentioned unlocking process at least includes: unlocking the above-mentioned source system, the above-mentioned global routing and the above-mentioned customer information platform in sequence. In this solution, a locking process is initiated based on the obtained migration customer list, and then the source data exported from the source system is processed to obtain the target data based on the account information of the migration customer, and the target data is imported into the target system. Finally, an unlocking process is sent based on the migration customer list. That is, this solution ensures the consistency of the migration customer status and the consistency of the data during the data migration process through the locking and unlocking processes. In addition, this solution sends the locking process and the unlocking process based on the migration customer list, thereby realizing the migration of massive data without stopping the external service of the real-time trading system, thereby solving the technical problem in the existing technology that it is impossible to realize the switching of the old and new systems and data migration without stopping the external service of the real-time trading system.

[0042] In actual application, after sending the unlocking process according to the migration customer list, the status of the migration customer can also be changed to "migrated".

[0043] In one embodiment of the present application, obtaining a migration customer list includes: obtaining the migration customer list from a related system based on a customer activity list, wherein the customer activity list includes customers and the activity levels associated with the customers. In this embodiment, obtaining the migration customer list from a related system based on the customer activity list ensures that the migration customers to be migrated can be more reasonably planned. Subsequently, a locking process can be initiated based on the migration customer list, further enabling batch migration of the migration customers and further ensuring high migration efficiency.

[0044] In order to ensure that the calculated activity of non-migrated customers is relatively accurate, and to further ensure that the migration customers that need to be migrated can be planned more reasonably based on the activity of the non-migrated customers in the future, in another embodiment of the present application, before obtaining the above-mentioned migration customer list from the above-mentioned associated system based on the customer activity list, the above-mentioned method also includes: sending the non-migrated customer list to the big data platform to calculate the activity of the non-migrated customers and obtain the calculation results; obtaining the above-mentioned calculation results from the above-mentioned big data platform, and updating the above-mentioned customer activity list based on the above-mentioned calculation results.

[0045] In actual application, the data migration system can regularly generate a list of non-migrated customers and send the non-migrated customer list to the big data platform to analyze customer transaction behavior and account status, so as to reasonably plan the customer migration window. The specific steps can be: the associated system can obtain the corresponding relationship between existing customer accounts and regularly maintain incremental data; the big data platform can regularly obtain the list of non-migrated customers to calculate the activity; the big data platform returns the calculated results of the activity of non-migrated customers; based on the customer activity list, the customer migration list is obtained again from the associated system, so that each customer is split into each migration scenario. The specific migration methods can be: early migration, online migration, incremental migration, delayed migration and passive migration.

[0046] Furthermore, customer data can be categorized into static and non-static data. Static data refers to data historically stored in the legacy system for query purposes. It has low access efficiency based on the customer dimension, large data volumes, and low transaction usage. Therefore, it is migrated in batches, using data as the non-customer dimension (pre- and post-transaction). Data is not converted during migration and is migrated according to the original data type. Non-static data refers to data that is subsequently updated after initial storage. It has high access efficiency, relatively small data volumes, and high transaction usage. It is migrated using customer-based locking and online migration and batch migration. Previous business closure migrations used one-to-one migrations based on database tables and data records. Online migrations use customer-based migrations. This reduces migration complexity and shortens customer lock-in periods, improving migration efficiency and customer experience. Furthermore, a migration list is generated based on each customer's activity level.

[0047] Specifically, in the actual application process, the activity level of the above-mentioned non-migrated customers can be an active time period when the customers go to handle business. Subsequently, the non-migrated customers in the non-migrated customer list can be grouped according to the active time period of the customers. In the process of migrating data for non-migrated customers, the active time period of the non-migrated customers can be avoided, and data migration can be performed in time periods other than the active time period of the non-migrated customers. That is, this solution fully considers the sensitivity of China UnionPay and China National NetPay to transaction processing time and transaction success rate, reduces the migration concurrency during transaction peaks and end-of-day batch processing windows, and reduces the risk of data migration to the smooth operation of the production system.

[0048] In order to further ensure that more accurate target data can be obtained subsequently, and further ensure that the consistency of the data after subsequent migration is high, in another embodiment of the present application, the above-mentioned data processing at least includes: data removal, data conversion, data verification and data import, wherein the above-mentioned data removal is to unload the source data of the above-mentioned source system and transfer the above-mentioned source data to the data conversion environment; the above-mentioned data conversion is to convert the above-mentioned source data exported from the above-mentioned source system into the above-mentioned target data of the above-mentioned target system according to the conversion relationship in the data mapping table; the above-mentioned data verification is to compare the data accuracy and data consistency of the above-mentioned target data with the above-mentioned source data, and verify them according to data quality standards; the above-mentioned data import is to load the verified above-mentioned target data into the above-mentioned target system in the form of batch migration or online migration.

[0049] In another embodiment of the present application, in the process of processing the source data exported from the source system according to the account information of the migration customer to obtain the target data, and importing the target data into the target system, the method further includes: managing the data migration process and / or monitoring the data migration process. In this embodiment, in the process of importing the target data into the target system, the data migration process is managed, or the data migration process is monitored, or the data migration process is managed and monitored, that is, the present solution realizes control over the entire data migration process, and can also detect anomalies through monitoring and promptly handle the discovered anomalies.

[0050] In actual application, after data migration, daily report and query functions can be provided to facilitate understanding of the data migration status.

[0051] In order to further ensure that the data migration process can be executed more stably and further ensure the correctness of all data before and after migration, in one embodiment of the present application, the management of the data migration process includes at least one of the following: managing migration batches and the rules for generating the above-mentioned migration batches; configuring the operating parameters required by the task scheduler and querying the task operation status; managing the message interaction between systems; managing any one or more of the operating users, role permissions, system logs, and user login / logout.

[0052] Specifically, before the migration begins, migration batches can be generated based on the migration scenario and migration rules, and the migration batches can be managed, and the rules for generating migration batches can also be managed; the management of message interactions between systems includes: when interacting with external systems, after the local message is packaged, the protocol conversion WTC (WebLogic TuxedoConnector) protocol conversion interface is called to convert it into a tuxedo protocol message, which is sent to the target system, and the target system response is received to complete the unpacking process; when the external system calls the service of this system, the application programming interface (Application Programming Interface, API) exposed to the external system by the data migration system is called to convert the tuxedo message into an http message and perform the unpacking operation.

[0053] In another embodiment of the present application, monitoring the data migration process includes at least one of the following: monitoring the migration progress during the data migration process; and monitoring migration exception information during the data migration process. In this embodiment, monitoring the migration progress and migration exception information during the data migration process further ensures that exceptions that occur during the migration process can be obtained in a relatively timely manner, further ensuring that any exceptions that occur can be handled in a relatively timely manner.

[0054] Specifically, the entire process of data migration operation is monitored and displayed, and abnormal situations are promptly warned through the display interface or text messages. The aggregation and report generation functions of data within a certain range are supported. The processing process: the monitoring data capture interface captures data related to monitoring-related indicators from databases, files, and logs, and sends it to the monitoring data processing interface; after the monitoring data processing interface receives the intermediate data, it formats the data according to the monitoring display requirements and sends it to the presentation layer; the presentation layer parses the data and displays a visual interface related to the monitoring indicators.

[0055] The present application also provides a data migration processing device. It should be noted that the data migration processing device of the present application can be used to execute the data migration processing method provided in the present application. The data migration processing device provided in the present application is introduced below.

[0056] Figure 5 is a schematic diagram of a data migration processing device according to an embodiment of the present application, such as Figure 5 As shown, the device includes:

[0057] A first acquisition module 40 is configured to acquire a migration client list, wherein the migration client list includes at least account information of the migration client;

[0058] A locking module 42, connected to the first acquisition module 40, is configured to initiate a locking process based on the migration customer list, wherein the locking process at least includes: sequentially locking the customer information platform, global routing, and source system;

[0059] A migration module 44, connected to the locking module 42, is configured to process the source data exported from the source system according to the account information of the migrating client to obtain target data, and import the target data into the target system, wherein the target data conforms to the data model structure of the target system;

[0060] The unlocking module 46 is connected to the migration module 44 and is used to initiate an unlocking process according to the migration customer list, wherein the unlocking process at least includes: unlocking the source system, the global routing and the customer information platform in sequence.

[0061] In the above-mentioned data migration processing device, the first acquisition module is used to obtain a list of migration customers, wherein the above-mentioned migration customer list at least includes the account information of the migration customers; the locking module is used to initiate a locking process based on the above-mentioned migration customer list, wherein the above-mentioned locking process at least includes: locking the customer information platform, the global routing and the source system in sequence; the migration module is used to process the source data exported from the above-mentioned source system according to the account information of the above-mentioned migration customers to obtain target data, and import the above-mentioned target data into the target system, wherein the above-mentioned target data is data that conforms to the data model structure of the above-mentioned target system; the unlocking module is used to initiate an unlocking process based on the above-mentioned migration customer list, wherein the above-mentioned unlocking process at least includes: unlocking the above-mentioned source system, the above-mentioned global routing and the above-mentioned customer information platform in sequence. In this solution, a locking process is initiated based on the obtained migration customer list, and then the source data exported from the source system is processed to obtain the target data based on the account information of the migration customer, and the target data is imported into the target system. Finally, an unlocking process is sent based on the migration customer list. That is, this solution ensures the consistency of the migration customer status and the consistency of the data during the data migration process through the locking and unlocking processes. In addition, this solution sends the locking process and the unlocking process based on the migration customer list, thereby realizing the migration of massive data without stopping the external service of the real-time trading system, thereby solving the technical problem in the existing technology that it is impossible to realize the switching of the old and new systems and data migration without stopping the external service of the real-time trading system.

[0062] It should be noted that during the locking process, the customer information platform, global routing, and source system must be locked separately. Optionally, the customer information platform primarily queries customer information, changes customer lock status, and locks customer transaction restrictions. The global routing queries migration status, changes customer lock status, and queries the time of the customer's last transaction. The source system changes media status and locks account transaction restrictions. The data migration system maintains customer-media relationships, manages customer lock status, and schedules locking processes. Considering that the impact on customer transactions should be minimized during the locking process, there are sequential requirements for locking and unlocking these three platforms. During the locking process, the customer is first locked on the customer information platform, the media contained under the customer is queried, the customer's migration status is changed to "Migrating" in the global routing, and finally, the media lock status is changed to "Migrating" in the source system. After the locking process is completed, data is retrieved, converted, and imported.

[0063] During the unlocking process, the source system is unlocked first, followed by an update of the global routing status, and finally the customer information platform. The source system is locked last during the locking process and unlocked first at the unlocking timer. This minimizes the duration of the source system lockout and reduces the impact on customer transactions.

[0064] In actual application, after sending the unlocking process according to the migration customer list, the status of the migration customer can also be changed to "migrated".

[0065] In one embodiment of the present application, the first acquisition module includes an acquisition submodule configured to acquire the migration customer list from an associated system based on a customer activity list, wherein the customer activity list includes customers and the activity levels associated with the customers. In this embodiment, the migration customer list is acquired from the associated system based on the customer activity list, thereby ensuring that the migration customers to be migrated can be more reasonably planned. Subsequently, a locking process can be initiated based on the migration customer list, further enabling batch migration of the migration customers and ensuring higher migration efficiency.

[0066] In order to ensure that the calculated activity of non-migrated customers is relatively accurate, and to further ensure that the migration customers that need to be migrated can be planned more reasonably based on the activity of the non-migrated customers in the future, in another embodiment of the present application, the above-mentioned processing device also includes a calculation module and a second acquisition module, wherein the above-mentioned calculation module is used to send the non-migrated customer list to the big data platform to calculate the activity of the non-migrated customers and obtain the calculation results before obtaining the above-mentioned migration customer list from the above-mentioned associated system according to the customer activity list; the above-mentioned second acquisition module is used to obtain the above-mentioned calculation results from the above-mentioned big data platform, and update the above-mentioned customer activity list according to the above-mentioned calculation results.

[0067] In actual application, the data migration system can regularly generate a list of non-migrated customers and send the non-migrated customer list to the big data platform to analyze customer transaction behavior and account status, so as to reasonably plan the customer migration window. The specific steps can be: the associated system can obtain the corresponding relationship between existing customer accounts and regularly maintain incremental data; the big data platform can regularly obtain the list of non-migrated customers to calculate the activity; the big data platform returns the calculated results of the activity of non-migrated customers; based on the customer activity list, the customer migration list is obtained again from the associated system, so that each customer is split into each migration scenario. The specific migration methods can be: early migration, online migration, incremental migration, delayed migration and passive migration.

[0068] Furthermore, customer data can be categorized into static and non-static data. Static data refers to data historically stored in the legacy system for query purposes. It has low access efficiency based on the customer dimension, large data volumes, and low transaction usage. Therefore, it is migrated in batches, using data as the non-customer dimension (pre- and post-transaction). Data is not converted during migration and is migrated according to the original data type. Non-static data refers to data that is subsequently updated after initial storage. It has high access efficiency, relatively small data volumes, and high transaction usage. It is migrated using customer-based locking and online migration and batch migration. Previous business closure migrations used one-to-one migrations based on database tables and data records. Online migrations use customer-based migrations. This reduces migration complexity and shortens customer lock-in periods, improving migration efficiency and customer experience. Furthermore, a migration list is generated based on each customer's activity level.

[0069] Specifically, in the actual application process, the activity level of the above-mentioned non-migrated customers can be an active time period when the customers go to handle business. Subsequently, the non-migrated customers in the non-migrated customer list can be grouped according to the active time period of the customers. In the process of migrating data for non-migrated customers, the active time period of the non-migrated customers can be avoided, and data migration can be performed in time periods other than the active time period of the non-migrated customers. That is, this solution fully considers the sensitivity of China UnionPay and China National NetPay to transaction processing time and transaction success rate, reduces the migration concurrency during transaction peaks and end-of-day batch processing windows, and reduces the risk of data migration to the smooth operation of the production system.

[0070] In order to further ensure that more accurate target data can be obtained subsequently, and further ensure that the consistency of the data after subsequent migration is high, in another embodiment of the present application, the above-mentioned data processing at least includes: data removal, data conversion, data verification and data import, wherein the above-mentioned data removal is to unload the source data of the above-mentioned source system and transfer the above-mentioned source data to the data conversion environment; the above-mentioned data conversion is to convert the above-mentioned source data exported from the above-mentioned source system into the above-mentioned target data of the above-mentioned target system according to the conversion relationship in the data mapping table; the above-mentioned data verification is to compare the data accuracy and data consistency of the above-mentioned target data with the above-mentioned source data, and verify them according to data quality standards; the above-mentioned data import is to load the verified above-mentioned target data into the above-mentioned target system in the form of batch migration or online migration.

[0071] In another embodiment of the present application, the processing device further includes a management module for managing and / or monitoring the data migration process during the process of processing the source data exported from the source system to obtain target data based on the account information of the migration client and importing the target data into the target system. In this embodiment, the data migration process is managed, or monitored, or both managed and monitored during the process of importing the target data into the target system. That is, this solution achieves control over the entire data migration process and can also detect anomalies through monitoring and promptly handle the discovered anomalies.

[0072] In actual application, after data migration, daily report and query functions can be provided to facilitate understanding of the data migration status.

[0073] In order to further ensure that the data migration process can be executed more stably, and further ensure the correctness of all data before and after the migration, in one embodiment of the present application, the above-mentioned management module includes at least one of the following: a first management sub-module, a configuration sub-module, a second management sub-module, and a third management sub-module, wherein the above-mentioned first management sub-module is used to manage the migration batches and the rules for generating the above-mentioned migration batches; the above-mentioned configuration sub-module is used to configure the operating parameters required by the task scheduler and query the task operation status; the above-mentioned second management sub-module is used to manage the message interaction between systems; the above-mentioned third management sub-module is used to manage any one or more of the operating users, role permissions, system logs, and user login / logout.

[0074] Specifically, before the migration begins, migration batches can be generated based on the migration scenario and migration rules, and the migration batches can be managed, and the rules for generating migration batches can also be managed; the management of message interactions between systems includes: when interacting with external systems, after the local message is packaged, the protocol conversion WTC (WebLogic TuxedoConnector) protocol conversion interface is called to convert it into a tuxedo protocol message, which is sent to the target system, and the target system response is received to complete the unpacking process; when the external system calls the service of this system, the application programming interface (Application Programming Interface, API) exposed to the external system by the data migration system is called to convert the tuxedo message into an http message and perform the unpacking operation.

[0075] In another embodiment of the present application, the management module further includes at least one of the following: a first monitoring submodule and a second monitoring submodule, wherein the first monitoring submodule is configured to monitor the migration progress during the data migration process; and the second monitoring submodule is configured to monitor migration exception information during the data migration process. In this embodiment, monitoring the migration progress and migration exception information during the data migration process further ensures that exceptions occurring during the migration process can be obtained in a relatively timely manner, further ensuring that any exceptions occurring can be handled in a relatively timely manner.

[0076] Specifically, the entire process of data migration operation is monitored and displayed, and abnormal situations are promptly warned through the display interface or text messages. The aggregation and report generation functions of data within a certain range are supported. The processing process: the monitoring data capture interface captures data related to monitoring-related indicators from databases, files, and logs, and sends it to the monitoring data processing interface; after the monitoring data processing interface receives the intermediate data, it formats the data according to the monitoring display requirements and sends it to the presentation layer; the presentation layer parses the data and displays a visual interface related to the monitoring indicators.

[0077] In a typical embodiment of the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned data migration processing methods.

[0078] In another typical embodiment of the present application, a processor is further provided, and the processor is used to run a program, wherein the program executes any one of the above-mentioned data migration processing methods when running.

[0079] In order to make those skilled in the art more clearly understand the technical solution of this application, the following will be described with reference to specific embodiments: Example 1

[0080] like Figure 2 As shown in the figure, the online data migration processing of the new system (i.e., the target system) relies on the data migration system. Its core functions include control and migration functions. The control function includes management functions such as UI display, migration public management, arbitrary management, exception handling, monitoring, batch management, and inter-system interaction. The migration function includes data export from the source system, data conversion, data import into the target system, verification, and other migration functions. The data migration system is built on a distributed microservice framework and provides high-performance and easily scalable online data migration processing capabilities. Example 2

[0081] like Figure 3 As shown, the overall process of migrating customer data is as follows: the data migration system regularly generates a list of unmigrated customers, the big data platform analyzes customer transaction behavior and account status, and rationally plans the customer migration window. The specific steps are:

[0082] (1) Obtain the existing customer account mapping relationship from the related system and regularly maintain incremental data;

[0083] (2) The big data platform regularly obtains a list of non-migrated customers and calculates their activity.

[0084] (3) The big data platform returns the calculation results of the activity of non-migrated customers;

[0085] (4) Based on the customer activity list, the customer migration list is obtained from the related system again, so that each customer is divided into each migration scenario. The migration methods are divided into early migration, online migration, incremental migration, delayed migration and passive migration;

[0086] (5) In order to maintain data consistency during the migration process, lock the customer information platform, global routing and source system in accordance with the customer migration list.

[0087] (6) The data exported from the source system is converted into data that conforms to the data model structure of the new system through data processing during data migration, and then imported into the new system. The data processing process of data migration mainly includes: a. Data removal, which refers to the process of unloading data from the production environment; data transfer, which refers to transferring data from the source system production environment to the data conversion environment; b. Data conversion, which converts the source data exported from the source system into the target data of the new system according to the conversion relationship in the data mapping table; c. Data verification, which compares the converted data with the exported source data for data accuracy and consistency, and verifies it according to data quality standards; d. Data import, which is the process of loading the verified target data after conversion into the new system environment in batches or online migration, and verifies the correctness of the migrated data and prepares for emergency response;

[0088] (7) Import target data into the target system;

[0089] (8) Initiate the unlocking process according to the customer migration list, unlock the source system, global routing and customer information platform in turn, and finally complete the migration after reconciliation analysis of the source system and target system. Example 3

[0090] like Figure 4 As shown in the figure, the specific steps of the migration task scheduling process are: the control system generates a batch and locks it to start migration; calls the task scheduling service; queries the batch details to obtain customer data; the task scheduling service calls the task executor based on the obtained customer data to initiate customer data export processing. Example 4

[0091] The control function also provides an interface for human-computer interaction (i.e. Figure 2 The display UI (shown in the figure) is the command center for the data migration process. Before migration, you can flexibly configure the data migration method based on data characteristics. During migration, you can control the entire migration process through batch and task management. Monitoring functions can detect and resolve any anomalies. After migration, daily reports and query functions are provided to facilitate monitoring of the migration status. The following is an introduction to the functions of each module:

[0092] (1) Migrate the public management module

[0093] The migration public management module mainly handles: sending migration start / stop instructions to the task scheduler, and maintaining and managing migration parameters;

[0094] (2) Migration exception management module

[0095] The migration exception management module mainly handles: monitoring exceptions during the migration process and intervening and handling them;

[0096] (3) Migration batch management module

[0097] The migration batch management module mainly handles: before the migration begins, it generates migration batches based on the migration scenario and migration rules, manages the batches, and manages the rules for generating batches;

[0098] (4) Task configuration module

[0099] The task configuration module mainly processes: configuring the operating parameters required by the task scheduler and performing overall queries on the task operation status.

[0100] (5) Monitoring and management module

[0101] The monitoring and management module mainly handles: status monitoring and display of the entire process of data migration, timely alerts of abnormal situations through the display interface or SMS, and supports the aggregation and report generation of data within a certain range. Processing process:

[0102] a. Monitoring data capture interface, which captures data related to monitoring-related indicators from databases, files, and logs, and sends it to the monitoring data processing interface.

[0103] b. After receiving the intermediate data, the monitoring data processing interface formats the data according to the monitoring display requirements and sends it to the presentation layer.

[0104] c. The presentation layer parses the data and displays a visual interface related to monitoring indicators.

[0105] (6) Inter-system interaction module

[0106] The system interaction module mainly handles: when interacting with external systems such as locking customers, updating customer information, unlocking customers, sending alarm SMS, etc., it interacts with related systems through online message exchanges. Processing process:

[0107] a. When interacting with an external system, after the local message is packaged, the protocol conversion WTC protocol conversion interface is called to convert it into a Tuxedo protocol message, which is sent to the target system and receives a response from the target system to complete the unpacking process.

[0108] b. When an external system calls the service of this system, the communication API exposed to the external system by the data migration system is called to convert the Tuxedo message into an HTTP message and perform the unpacking operation.

[0109] (7) System management module

[0110] The system management module mainly handles: management of operating users, role permissions, system logs, and user login / logout of the control platform.

[0111] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0112] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0113] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0114] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0115] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0116] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk.

[0117] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data migration processing method, characterized in that: include: Send the list of non-migrated customers to the big data platform to calculate the activity of non-migrated customers and obtain the calculation results; Obtaining the calculation results from the big data platform, and updating the customer activity list according to the calculation results; Obtaining a migration customer list, comprising: obtaining the migration customer list from a related system according to a customer activity list, wherein the customer activity list includes customers and activities associated with the customers, and wherein the migration customer list includes at least account information of the migration customers; Initiating a locking process according to the migration customer list, wherein the locking process at least includes: locking the customer information platform, global routing and source system in sequence; Processing the source data exported from the source system according to the account information of the migrating customer to obtain target data, and importing the target data into the target system, wherein the target data is data that conforms to the data model structure of the target system; An unlocking process is initiated based on the migration customer list, wherein the unlocking process at least includes: unlocking the source system, the global routing and the customer information platform in sequence; the customer information platform mainly performs operations such as querying customer information, changing customer locking status and locking customer transaction restrictions; the global routing performs functions such as querying migration status, changing customer locking status and querying the time of the customer's last transaction; the source system performs medium status changes and locks account transaction restrictions; and the data migration system performs customer and medium relationship maintenance, customer locking status management and locking process scheduling. Considering that the impact on customer transactions should be minimized during the locking process, there is a sequential requirement in the process of locking and unlocking these three platforms.

2. The method according to claim 1, characterized in that The data processing at least includes: data removal, data conversion, data verification and data import, wherein the data removal is to unload the source data of the source system and transfer the source data to the data conversion environment; the data conversion is to convert the source data exported from the source system into the target data of the target system according to the conversion relationship in the data mapping table; the data verification is to compare the target data with the source data for data accuracy and data consistency, and verify them according to data quality standards; the data import is to load the verified target data into the target system in batch migration or online migration.

3. The method according to any one of claims 1 to 2, characterized in that In the process of processing the source data exported from the source system to obtain target data according to the account information of the migrating customer, and importing the target data into the target system, the method further includes: Manage and / or monitor the data migration process.

4. The method according to claim 3, characterized in that Managing the data migration process includes at least one of the following: Managing migration batches and rules for generating said migration batches; Configure the operating parameters required by the task scheduler and query the task running status; Manage message interactions between systems; Manage any one or more of the operating users, role permissions, system logs, and user login / logout.

5. The method according to claim 3, characterized in that Monitoring the data migration process includes at least one of the following: Monitor the progress of data migration; Monitor migration exception information during data migration.

6. A data migration processing device, characterized in that: include: A first acquisition module is configured to acquire a migration client list, wherein the migration client list at least includes account information of the migration client; The first acquisition module includes an acquisition submodule for acquiring the migration customer list from the associated system according to a customer activity list, wherein the customer activity list includes customers and activities associated with the customers; a calculation module and a second acquisition module, wherein the calculation module is used to send the non-migrated customer list to the big data platform to calculate the activity of the non-migrated customers and obtain a calculation result before obtaining the migrated customer list from the associated system based on the customer activity list; the second acquisition module is used to obtain the calculation result from the big data platform and update the customer activity list according to the calculation result; A locking module, configured to initiate a locking process according to the migration customer list, wherein the locking process at least includes: sequentially locking the customer information platform, global routing, and source system; a migration module, configured to process the source data exported from the source system to obtain target data according to the account information of the migrating customer, and import the target data into the target system, wherein the target data is data that conforms to the data model structure of the target system; An unlocking module is used to initiate an unlocking process based on the migration customer list, wherein the unlocking process at least includes: unlocking the source system, the global routing and the customer information platform in sequence, the customer information platform mainly performs operations such as querying customer information, changing customer locking status and locking customer transaction restrictions, the global routing performs functions such as querying migration status, changing customer locking status and querying the time of the customer's last transaction, the source system performs medium status changes and locks account transaction restrictions, and the data migration system performs customer and medium relationship maintenance, customer locking status management and locking process scheduling. Considering that the impact on customer transactions should be minimized during the locking process, there is a sequential requirement in the process of locking and unlocking these three platforms.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the data migration processing method according to any one of claims 1 to 5.

8. A processor, characterized in that: The processor is used to run a program, wherein the program executes the data migration processing method according to any one of claims 1 to 5 when running.

Images