Data migration method and device, equipment, storage medium and program product

Abstract

The invention discloses a data migration method and device, equipment, a storage medium and a program product. The invention relates to the technical field of data migration. Acquiring historical transaction information respectively corresponding to the plurality of customer identifiers; dividing the to-be-migrated data into a plurality of batches based on the historical transaction information; for the to-be-migrated data of the first batch, reading the to-be-migrated data of the first batch from the old system to a first data file, and importing the to-be-migrated data from the first data file to a first data table in the new system; and for each non-first batch of to-be-migrated data, reading the non-first batch of to-be-migrated data from the old system to one or more second data files, and importing the to-be-migrated data from the one or more second data files to a plurality of second data tables in the new system. The data migration speed and accuracy can be improved, and the loss caused by problems in the data migration process can be reduced to the minimum.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data migration, and in particular, to a data migration method, apparatus, device, storage medium, and program product. Background Art

[0002] After the system is reconstructed, the data of the old system needs to be migrated to the new system. In the traditional implementation method, the data of the old system is generally migrated to the new system as a whole. This full migration operation has great risks, and the switching results cannot be verified. If a serious error occurs in the new system, it will cause immeasurable and irrecoverable losses. Summary of the invention

[0003] The embodiments of the present invention provide a data migration method, apparatus, device, storage medium and program product, which can improve the speed and accuracy of data migration by migrating data from an old system to an old system in batches.

[0004] In a first aspect, an embodiment of the present invention provides a data migration method, including:

[0005] Acquire historical transaction information corresponding to multiple customer identifiers respectively; wherein the historical transaction information includes historical transaction volume and transaction category within a set period of time;

[0006] Based on the historical transaction information, the data to be migrated is divided into a plurality of batches; wherein the data to be migrated carries the customer identifier, and the data to be migrated with the same customer identifier is divided into the same batch;

[0007] For a first batch of data to be migrated, the first batch of data to be migrated is read from the old system into a first data file, and then the data to be migrated is imported from the first data file into a first data table in the new system;

[0008] For each non-first batch of data to be migrated, the non-first batch of data to be migrated is read from the old system into one or more second data files, and then the data to be migrated from the one or more second data files is imported into multiple second data tables in the new system.

[0009] In a second aspect, an embodiment of the present invention further provides a data migration device, including:

[0010] A historical transaction information acquisition module is used to acquire historical transaction information corresponding to multiple customer identifiers; wherein the historical transaction information includes historical transaction volume and transaction category within a set period of time;

[0011] A batch division module, used to divide the data to be migrated into multiple batches based on the historical transaction information; wherein the data to be migrated carries the customer identifier, and the data to be migrated with the same customer identifier is divided into the same batch;

[0012] A first batch data migration module, for reading the first batch of data to be migrated from the old system into a first data file, and then importing the data to be migrated from the first data file into a first data table in the new system;

[0013] The non-first batch data migration module is used to read the non-first batch of data to be migrated from the old system into one or more second data files, and then import the data to be migrated from the one or more second data files into multiple second data tables in the new system.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data migration method as described in the embodiment of the present invention is implemented.

[0015] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data migration method as described in the embodiment of the present invention.

[0016] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data migration method as described in the embodiment of the present invention.

[0017] The embodiment of the present invention discloses a data migration method, device, equipment, storage medium and program product, the method comprising: obtaining historical transaction information corresponding to multiple customer identifications respectively; wherein the historical transaction information includes historical transaction volume and transaction category within a set period; dividing the data to be migrated into multiple batches based on the historical transaction information; wherein the data to be migrated carries the customer identification, and the data to be migrated with the same customer identification is divided into the same batch; for the first batch of data to be migrated, the first batch of data to be migrated is read from the old system into a first data file, and then the data to be migrated is imported from the first data file into a first data table in the new system; for each non-first batch of data to be migrated, the non-first batch of data to be migrated is read from the old system into one or more second data files, and then the data to be migrated is imported from the one or more second data files into multiple second data tables in the new system. The data migration method provided by the embodiment of the present invention divides the data to be migrated into multiple batches, and migrates the data of the first batch and the data of the non-first batch from the old system to the new system according to different migration methods, which can not only improve the speed and accuracy of data migration, but also minimize the loss caused by problems in the data migration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of a data migration method provided by an embodiment of the present invention;

[0020] Figure 2 A flowchart of a data migration method provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of the structure of a data migration device provided by an embodiment of the present invention;

[0022] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0024] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of the present invention, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance. The acquisition, storage, use, processing, etc. of data in the technical solution of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. The purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or will necessarily use the solution.

[0025] Figure 1 A flow chart of a data migration method provided in an embodiment of the present invention is applicable to the case of migrating data from an old system to a new system. The method can be executed by a data migration device, which can be implemented in the form of software and / or hardware, and optionally, by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0026] In this embodiment, before data migration, the deployment and configuration of the new system needs to be completed. The specific process includes the following: first, network activation and verification; then deployment of basic components; then migration of table structure and basic technical data; then modification of application firewall (WAF) domain name resolution, pointing domain name resolution to the new system service, modification of routing configuration, and sending all business requests to the request dispatcher of the new system, which dispatches the business requests.

[0027] like Figure 1 As shown, the method comprises the following steps:

[0028] S110, obtaining historical transaction information corresponding to multiple customer identifications.

[0029] Among them, historical transaction information includes historical transaction volume and transaction category within a set time period. The customer identifier can be understood as the unique identification code of the customer who registers in the system to initiate a transaction service request to the system. For example, a certain enterprise is registered in the banking system and enters its unique identification code during registration, so that the system processes the transaction service initiated by the enterprise based on the unique identification code. The historical transaction volume within the set time period can be understood as the total business transaction volume corresponding to a certain customer identifier in the most recent set time period (such as the most recent week, the last month, etc.), and the transaction category within the set time period can be understood as the number of transaction categories corresponding to a certain customer identifier in the most recent set time period. Exemplarily, transaction categories may include: transfer business, deposit business, loan business, repayment business, purchase of financial products business, etc.

[0030] In this embodiment, when a customer initiates a transaction service request to the system, the transaction service request will carry the customer's customer identification and transaction category, and statistics will be collected on the transaction volume and transaction category corresponding to each customer identification within a set period of time, thereby obtaining historical transaction information corresponding to each customer identification.

[0031] S120: Divide the data to be migrated into multiple batches based on historical transaction information.

[0032] The data to be migrated carries a customer identifier, and the data to be migrated with the same customer identifier is divided into the same batch. After the data to be migrated is divided into multiple batches, the data to be migrated is migrated in sequence according to the batch order. Optionally, the multiple batches are three batches, namely the first batch, the second batch and the third batch. In this embodiment, the data to be migrated in the first batch is migrated first, and then the data to be migrated in the second batch is migrated after the migration is completed. After the migration is completed, the data to be migrated in the third batch is migrated last, thereby completing the migration of all data.

[0033] Optionally, the method of dividing the data to be migrated into multiple batches based on historical transaction information may be: dividing the data to be migrated of customer identifiers whose historical transaction volume is less than or equal to a first set threshold into a first batch; dividing the data to be migrated of customer identifiers whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories that exceeds a second set threshold into a second batch; dividing the data to be migrated of customer identifiers whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories that does not exceed the second set threshold into a third batch.

[0034] Among them, the first set threshold and the second set threshold can be preset. In order to make the new system handle as many transaction categories as possible in the second batch, the second set threshold can be set to any value between 80% and 100%. Among them, the proportion of transaction categories to total transaction categories can be understood as: the proportion of the number of transaction categories corresponding to the customer identification within the set time period to the total number of transaction categories that the system can handle. Exemplarily, assuming that the number of transaction categories corresponding to a certain customer identification within the set time period is M, and the total number of transaction categories that the system can handle is N, then the proportion of transaction categories to total transaction categories is: M / N.

[0035] In this embodiment, the data to be migrated of the customer identifiers whose historical transaction volumes are less than or equal to the first set threshold are divided into the first batch for migration. After the migration is completed, the new system can process the transaction business of the customer identifiers migrated in the first batch, so as to verify the stability of the new system by using the transaction business with smaller transaction volumes, thereby minimizing the losses caused by the instability of the new system.

[0036] In this embodiment, the data to be migrated of the customer identifiers whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories that exceeds the second set threshold are divided into the second batch for migration. After the migration is completed, the new system can process the transaction business of the customer identifiers migrated from the second batch, so that the new system can handle transaction business of more transaction categories, thereby verifying the generalization ability of the new system in handling transaction business of different transaction categories.

[0037] In this embodiment, the data to be migrated of the customer identifiers whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories that does not exceed the second set threshold are divided into the third batch for migration. That is, the remaining data to be migrated is migrated in the third batch.

[0038] S130, for a first batch of data to be migrated, read the first batch of data to be migrated from the old system into a first data file, and then import the data to be migrated from the first data file into a first data table in the new system.

[0039] Among them, the first data file is a newly created file used to store data read from the old system. The first data table is used in the new system to store the first batch of data migrated from the old system. In this embodiment, since the first batch of data to be migrated is the data to be migrated of the customer identification whose historical transaction volume is less than or equal to the first set threshold, that is, the number of data to be migrated in the first batch is small, the method of migrating the first batch of data to be migrated is: read it as a whole from the old system into a first data file, and then import the first batch of data to be migrated from the first data file as a whole into a first data table of the new system, that is, the method of overall unloading and overall loading is adopted, so as to improve the migration rate of the first batch of data.

[0040] Optionally, before reading the first batch of data to be migrated from the old system into the first data file, and then importing the data to be migrated from the first data file into the first data table in the new system, the following step is also included: switching the whitelist configuration of the first batch of customer identifiers in the old system from normal processing to stop processing, so that the old system stops processing the transaction business of the first batch of customer identifiers.

[0041] Among them, there are two types of whitelist configurations: one is normal processing, and the other is stop processing. If the whitelist configuration of a customer ID in the old system is normal processing, the old system will normally process the transaction business corresponding to the customer ID; if the whitelist configuration of a customer ID in the old system is stop processing, the old system will stop processing the transaction business corresponding to the customer ID. In this embodiment, in order to smoothly complete the migration of the first batch of data to be migrated, before the migration, the whitelist configuration of the first batch of customer IDs in the old system is switched from normal processing to stop processing, that is, during the migration process of the first batch of data to be migrated and after the migration is completed, the old system stops processing the transaction business corresponding to the customer ID.

[0042] Optionally, after reading the first batch of data to be migrated from the old system into the first data file, and then importing the data to be migrated from the first data file into the first data table in the new system, the following step is also included: setting the whitelist configuration of the first batch of customer identifiers in the new system to normal processing, so that the new system can normally process the transaction business of the first batch of customer identifiers.

[0043] Specifically, if the whitelist configuration of a certain customer ID in the new system is set to normal processing, the new system will normally process the transaction business corresponding to the customer ID; if the whitelist configuration of a certain customer ID in the new system is set to stop processing, the new system will stop processing the transaction business corresponding to the customer ID. In this embodiment, after the migration is completed, the whitelist configuration of the first batch of customer IDs in the new system is set to normal processing, that is, after the migration of the first batch of data to be migrated is completed, the new system will normally process the transaction business corresponding to the customer ID.

[0044] Optionally, after reading the first batch of data to be migrated from the old system into the first data file, and then importing the data to be migrated from the first data file into the first data table in the new system, the following steps are also included: forwarding the transaction business of the first batch of customer identifications to the new system for processing, and detecting the processing results of the new system on the transaction business of the first batch of customer identifications; for transaction business with abnormal processing results, migrating the data corresponding to the customer identification of the transaction business or the data of the first batch of customer identifications from the new system to the old system; switching the whitelist configuration of the migrated customer identifications in the old system from stop processing to normal processing, so that the old system can process the transaction business of the migrated customer identifications normally; switching the whitelist configuration of the migrated customer identifications in the new system from normal processing to stop processing, so that the information system stops processing the transaction business of the migrated customer identifications.

[0045] The processing result includes normal processing or abnormal processing.

[0046] In this embodiment, if the new system processes a transaction as abnormal, it means that the new system cannot process the transaction under the customer ID corresponding to the transaction normally. In order to ensure that the transaction under the customer ID is executed normally and does not affect the customer experience, the data corresponding to the customer ID of the transaction needs to be migrated from the new system to the old system, and the whitelist configuration of the migrated customer ID in the old system is switched from stop processing to normal processing, so that the old system can process the transaction of the migrated customer ID normally; the whitelist configuration of the migrated customer ID in the new system is switched from normal processing to stop processing, so that the information system stops processing the transaction of the migrated customer ID. The new system is tested to find any problems.

[0047] In this embodiment, if the new system processes a transaction as abnormal, it indicates that there may be problems in the deployment and configuration of the new system. In order to ensure that the transaction under the first batch of customer IDs is executed normally and does not affect the customer experience, all the data of the first batch of customer IDs are migrated from the new system to the old system; the whitelist configuration of the migrated customer IDs in the old system is switched from stop processing to normal processing, so that the old system can normally process the transaction of the migrated customer IDs; the whitelist configuration of the migrated customer IDs in the new system is switched from normal processing to stop processing, so that the information system stops processing the transaction of the migrated customer IDs. The new system is tested to find any existing problems.

[0048] Optionally, for transaction businesses with abnormal processing results, the method for migrating the data corresponding to the customer identifier of the transaction business or the data of the customer identifier of the first batch from the new system to the old system can be: for transaction businesses with abnormal processing results, the data corresponding to the customer identifier of the transaction business or the data of the customer identifier of the first batch is read from the new system to a third data file, and then the data in the third data file is imported into the old system.

[0049] Among them, the third data file is a newly created file used to store data read from the new system. In this embodiment, since the data of the first batch is the data of the customer identification whose historical transaction volume is less than or equal to the first set threshold, that is, the amount of data in the first batch is small, the method of relocating the data of the first batch is: reading it as a whole from the new system into a third data file, and then importing the data in the third data file as a whole into the old system, that is, using the method of overall unloading and overall loading, so as to improve the relocation rate of the data of the first batch.

[0050] S140, for each non-first batch of data to be migrated, read the non-first batch of data to be migrated from the old system into one or more second data files, and then import the data to be migrated from the one or more second data files into multiple second data tables in the new system.

[0051] The non-first batch is the second batch or the third batch. The second data file is a newly created file used to store data read from the old system. The number of the second data files is one or more.

[0052] Optionally, the method of reading the data to be migrated that is not the first batch from the old system into one or more second data files, and then importing the data to be migrated from the one or more second data files into multiple second data tables can be: for the second batch of data to be migrated, obtaining the correspondence between the customer identifier and the second data file and the correspondence between the second data file and the second data table, and determining them as the first correspondence and the second correspondence, respectively; reading the second batch of data to be migrated from the old system into multiple second data files based on the first correspondence; and importing the data to be migrated from each second data file into the corresponding second data table of the new system based on the second correspondence.

[0053] In this embodiment, there are multiple second data files, and the correspondence between the customer identification and the second data file can be a many-to-one relationship, that is, the data to be migrated of multiple customer identifications are read into the same second data file. The correspondence between the second data file and the second data table is a one-to-one relationship, that is, the data in one second data file is read into a corresponding second data table.

[0054] Specifically, the process of reading the second batch of data to be migrated from the old system into multiple second data files based on the first correspondence may be: for each second data file, determine at least one customer identifier corresponding to the second data file based on the first correspondence, then query the data to be migrated corresponding to the at least one customer identifier from the old system, and read the queried data to be migrated into the second data file. The method of importing the data to be migrated from each second data file into the second data table corresponding to the new system based on the second correspondence may be: for each second data file, determine its corresponding second data table based on the second correspondence, and then import the data in the second data file into the corresponding second data table. In this embodiment, the migration of the second batch of data to be migrated is achieved based on the correspondence between the customer identifier and the second data file and the correspondence between the second data file and the second data table, which can improve the accuracy of data migration.

[0055] In the above embodiment, since it is necessary to query the data to be migrated corresponding to each customer identifier from the old system based on the first corresponding relationship, and the amount of data to be migrated in the second batch is large, this process is very time-consuming. Therefore, in order to increase the migration rate, for data to be migrated that is not the first batch, the overall unloading method can be used to read it into a second data file.

[0056] Optionally, the method of reading the data to be migrated that is not the first batch from the old system into one or more second data files, and then importing the data to be migrated from the one or more second data files into multiple second data tables can be: for each non-first batch of data to be migrated, obtaining the correspondence between the customer identifier and the second data table, and determining it as a third correspondence; reading the data to be migrated that is not the first batch from the old system into a second data file; splitting the second data file into multiple sub-data files based on the third correspondence; and importing the data to be migrated from each sub-data file into the corresponding second data table of the new system.

[0057] Among them, the sub-data file corresponds to the second data table one by one. The correspondence between the customer identification and the second data table can be a many-to-one relationship, that is, the data of multiple customer identifications are migrated to the same second data table. Reading the non-first batch of data to be migrated from the old system into a second data file can be understood as: reading the non-first batch of data to be migrated from the old system as a whole into a second data file, without the need to query the data to be migrated, which can greatly save time.

[0058] Specifically, the method of splitting the second data file into multiple sub-data files based on the third corresponding relationship can be: divide the data to be migrated of the customer identification corresponding to the same second data table into the same sub-data file, so that the divided sub-data files correspond one-to-one to the second data table. That is, the number of divided sub-data files is the same as the number of second data tables. The process of importing the data to be migrated from each sub-data file into the second data table corresponding to the new system can be: for each sub-data file, obtain its corresponding second data table, and then import the data in the sub-data file into the corresponding second data table. In this embodiment, firstly, the data other than the first batch is read as a whole into a second data file, and then the second data file is split, and finally the data in the split sub-data files is imported into the corresponding second data table, which can improve the rate of data migration.

[0059] Optionally, before obtaining the correspondence between the customer identifier and the second data table, the following step is also included: switching the whitelist configuration of customer identifiers other than the first batch in the old system from normal processing to stop processing, so that the old system stops processing transaction business for customer identifiers other than the first batch.

[0060] Among them, there are two types of whitelist configurations: one is normal processing, and the other is stop processing. If the whitelist configuration of a customer ID in the old system is normal processing, the old system will normally process the transaction business corresponding to the customer ID; if the whitelist configuration of a customer ID in the old system is stop processing, the old system will stop processing the transaction business corresponding to the customer ID. In this embodiment, in order to smoothly complete the migration of data that is not the first batch to be migrated, before the migration, the whitelist configuration of customer IDs that are not the first batch in the old system is switched from normal processing to stop processing, that is, during the migration process of data that is not the first batch to be migrated and after the migration is completed, the old system stops processing the transaction business corresponding to the customer ID.

[0061] Optionally, after importing the data to be migrated from each sub-data file into the second data table corresponding to the new system, the following step is also included: setting the whitelist configuration of customer identifiers other than the first batch in the new system to normal processing, so that the new system can normally process transaction business of customer identifiers other than the first batch.

[0062] Specifically, if the whitelist configuration of a certain customer ID in the new system is set to normal processing, the new system will normally process the transaction business corresponding to the customer ID; if the whitelist configuration of a certain customer ID in the new system is set to stop processing, the new system will stop processing the transaction business corresponding to the customer ID. In this embodiment, after the migration is completed, the whitelist configuration of customer IDs other than the first batch in the new system is set to normal processing, that is, after the migration of data other than the first batch to be migrated is completed, the new system will normally process the transaction business corresponding to the customer ID.

[0063] Optionally, after importing the data to be migrated from each sub-data file into the second data table corresponding to the new system, the following steps are also included: forwarding the transaction business with customer identifiers other than the first batch to the new system for processing, and detecting the processing results of the new system for the transaction business with customer identifiers other than the first batch; for transaction business with abnormal processing results, migrating the data corresponding to the customer identifier of the transaction business from the new system to the old system; switching the whitelist configuration of the customer identifier migrated back from the old system from stop processing to normal processing, so that the old system can process the transaction business with the migrated customer identifier normally; switching the whitelist configuration of the customer identifier migrated back from the new system from normal processing to stop processing, so that the information system stops processing the transaction business with the migrated customer identifier.

[0064] The processing result includes normal processing or abnormal processing.

[0065] In this embodiment, if the new system processes a transaction as abnormal, it means that the new system cannot process the transaction under the customer ID corresponding to the transaction normally. In order to ensure that the transaction under the customer ID is executed normally and does not affect the customer experience, the data corresponding to the customer ID of the transaction needs to be migrated from the new system to the old system, and the whitelist configuration of the migrated customer ID in the old system is switched from stop processing to normal processing, so that the old system can process the transaction of the migrated customer ID normally; the whitelist configuration of the migrated customer ID in the new system is switched from normal processing to stop processing, so that the information system stops processing the transaction of the migrated customer ID. The new system is tested to find any problems.

[0066] Optionally, for transaction businesses with abnormal processing results, the method of migrating the data corresponding to the customer identification of the transaction businesses from the new system to the old system can be: for transaction businesses with abnormal processing results, the data corresponding to the customer identification of the transaction businesses or the data of the customer identification of the first batch are read from the new system to a fourth data file, and then the data in the fourth data file is imported into the old system.

[0067] The fourth data file is a newly created file used to store data read from the new system.

[0068] Based on the above embodiments, Figure 2 is a flow chart of a data migration method in this embodiment. Figure 2 As shown, the method comprises the following steps:

[0069] S201, obtaining historical transaction volumes and transaction categories corresponding to multiple customer identifications within a set period of time.

[0070] S202, classify the data to be migrated of the customer identifiers whose historical transaction volume is less than or equal to the first set threshold into the first batch; classify the data to be migrated of the customer identifiers whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories that exceeds the second set threshold into the second batch; classify the data to be migrated of the customer identifiers whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories that does not exceed the second set threshold into the third batch.

[0071] S203, reading a first batch of data to be migrated from the old system into a first data file, and then importing the data to be migrated from the first data file into a first data table in the new system.

[0072] S204: For each batch of data to be migrated that is not the first batch, obtain a correspondence between the customer identifier and the second data table, and determine it as a third correspondence.

[0073] S205, reading the data to be migrated that is not the first batch from the old system into a second data file; splitting the second data file into multiple sub-data files based on the third corresponding relationship; and importing the data to be migrated from each sub-data file into the second data table corresponding to the new system.

[0074] S206, forwarding the transaction business with the migrated customer identifier to the new system for processing, and detecting the processing result of the transaction business with the migrated customer identifier by the new system; for the transaction business with abnormal processing result, migrating the data corresponding to the customer identifier of the transaction business from the new system to the old system.

[0075] S207, switching the whitelist configuration of the customer IDs migrated back from the old system from stop processing to normal processing, so that the old system can normally process the transaction business of the customer IDs migrated back; switching the whitelist configuration of the customer IDs migrated back from the new system from normal processing to stop processing, so that the information system stops processing the transaction business of the customer IDs migrated back.

[0076] The technical solution of this embodiment is to obtain historical transaction information corresponding to multiple customer identifiers respectively; wherein the historical transaction information includes the historical transaction volume and transaction category within a set period; the data to be migrated is divided into multiple batches based on the historical transaction information; wherein the data to be migrated carries the customer identifier, and the data to be migrated with the same customer identifier is divided into the same batch; for the first batch of data to be migrated, the first batch of data to be migrated is read from the old system into a first data file, and then the data to be migrated from the first data file is imported into a first data table in the new system; for each non-first batch of data to be migrated, the non-first batch of data to be migrated is read from the old system into one or more second data files, and then the data to be migrated from the one or more second data files is imported into multiple second data tables in the new system. The data migration method provided by the embodiment of the present invention divides the data to be migrated into multiple batches, and migrates the first batch of data and the non-first batch of data from the old system to the new system according to different migration methods, which can not only improve the speed and accuracy of data migration, but also minimize the loss caused by problems in the data migration process.

[0077] Figure 3 is a structural diagram of a data migration device provided by an embodiment of the present invention, such as Figure 3 As shown, the device comprises:

[0078] The historical transaction information acquisition module 310 is used to acquire historical transaction information corresponding to multiple customer identifiers; wherein the historical transaction information includes historical transaction volume and transaction category within a set period of time;

[0079] The batch division module 320 is used to divide the data to be migrated into multiple batches based on the historical transaction information; wherein the data to be migrated carries a customer identifier, and the data to be migrated with the same customer identifier is divided into the same batch;

[0080] A first batch data migration module 330 is used to read the first batch of data to be migrated from the old system into a first data file, and then import the data to be migrated from the first data file into a first data table in the new system;

[0081] The non-first batch data migration module 340 is used to read the non-first batch of data to be migrated from the old system into one or more second data files, and then import the data to be migrated from the one or more second data files into multiple second data tables in the new system.

[0082] Optionally, the multiple batches are three batches, namely a first batch, a second batch and a third batch.

[0083] Optionally, the batch division module 320 is further used to:

[0084] The data to be migrated of the customer identifiers whose historical transaction volumes are less than or equal to the first set threshold are divided into a first batch;

[0085] The data to be migrated of the customer IDs whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories exceeding the second set threshold are divided into the second batch;

[0086] The data to be migrated of the customer identifiers whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories that does not exceed the second set threshold are divided into the third batch.

[0087] Optionally, it further includes: a first whitelist configuration switching module, used to:

[0088] The whitelist configuration of the first batch of customer IDs in the old system is switched from normal processing to stop processing, so that the old system stops processing the transaction business of the first batch of customer IDs.

[0089] Optionally, it further includes: a second whitelist configuration switching module, used to:

[0090] Set the whitelist configuration for the first batch of customer IDs in the new system to normal processing, so that the new system can process the transaction business of the first batch of customer IDs normally.

[0091] Optionally, the method further includes: a first data relocation module, configured to:

[0092] Forwarding the transaction business of the first batch of customer identifications to the new system for processing, and detecting the processing results of the transaction business of the first batch of customer identifications by the new system;

[0093] For transactions with abnormal processing results, the data corresponding to the customer ID of the transaction or the data of the first batch of customer IDs will be migrated back from the new system to the old system;

[0094] Switch the whitelist configuration of customer IDs migrated back from the old system from stop processing to normal processing, so that the old system can normally process the transaction business of the migrated customer IDs; switch the whitelist configuration of customer IDs migrated back from the new system from normal processing to stop processing, so that the information system stops processing the transaction business of the migrated customer IDs.

[0095] Optionally, the first data migration module is further used for:

[0096] For transactions with abnormal processing results, data corresponding to the customer identification of the transaction or data of the customer identification of the first batch is read from the new system into a third data file, and then the data in the third data file is imported into the old system.

[0097] Optionally, the data migration module 340 of the non-first batch is further used for:

[0098] For the second batch of data to be migrated, a correspondence between the customer identifier and the second data file and a correspondence between the second data file and the second data table are obtained, and are determined as a first correspondence and a second correspondence respectively;

[0099] Based on the first corresponding relationship, read the second batch of data to be migrated from the old system into multiple second data files;

[0100] Based on the second corresponding relationship, the data to be migrated is imported from each second data file into a second data table corresponding to the new system.

[0101] Optionally, the data migration module 340 of the non-first batch is further used for:

[0102] For each non-first batch of data to be migrated, a correspondence between the customer identifier and the second data table is obtained, and determined as a third correspondence; wherein the non-first batch is the second batch or the third batch;

[0103] Reading the data to be migrated that is not the first batch from the old system into a second data file;

[0104] Splitting the second data file into a plurality of sub-data files based on the third corresponding relationship; wherein the sub-data files correspond to the second data table one by one;

[0105] The data to be migrated is imported from each sub-data file into the second data table corresponding to the new system.

[0106] Optionally, it further includes: a third whitelist configuration switching module, used to:

[0107] The whitelist configuration of customer IDs other than the first batch in the old system is switched from normal processing to stop processing, so that the old system stops processing transactions for customer IDs other than the first batch.

[0108] Optionally, it further includes: a fourth whitelist configuration switching module, used to:

[0109] Set the whitelist configuration of customer IDs other than the first batch in the new system to normal processing, so that the new system can normally process transactions for customer IDs other than the first batch.

[0110] Optionally, the method further includes: a second data migration module, configured to:

[0111] Forwarding the transaction business of the customer identification that is not the first batch to the new system for processing, and detecting the processing results of the transaction business of the customer identification that is not the first batch by the new system;

[0112] For transactions with abnormal processing results, the data corresponding to the customer ID of the transaction will be migrated from the new system to the old system;

[0113] Switch the whitelist configuration of customer IDs migrated back from the old system from stop processing to normal processing, so that the old system can normally process the transaction business of the migrated customer IDs; switch the whitelist configuration of customer IDs migrated back from the new system from normal processing to stop processing, so that the information system stops processing the transaction business of the migrated customer IDs.

[0114] Optionally, the second data migration module is further used for:

[0115] For transactions with abnormal processing results, data corresponding to the customer identification of the transaction or data of the customer identification of the first batch is read from the new system into a fourth data file, and then the data in the fourth data file is imported into the old system.

[0116] The above device can execute the methods provided by all the above embodiments of the present invention, and has the corresponding functional modules and beneficial effects of executing the above methods. For technical details not described in detail in this embodiment, please refer to the methods provided by all the above embodiments of the present invention.

[0117] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0118] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0119] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0120] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the data migration method.

[0121] In some embodiments, the data migration method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the data migration method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the data migration method in any other appropriate manner (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0124] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0126] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0127] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0128] An embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the data migration method provided in any embodiment of the present application.

[0129] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0130] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data migration method, characterized in that: include: Acquire historical transaction information corresponding to multiple customer identifiers respectively; wherein the historical transaction information includes historical transaction volume and transaction category within a set period of time; Based on the historical transaction information, the data to be migrated is divided into a plurality of batches; wherein the data to be migrated carries the customer identifier, and the data to be migrated with the same customer identifier is divided into the same batch; For a first batch of data to be migrated, the first batch of data to be migrated is read from the old system into a first data file, and then the data to be migrated is imported from the first data file into a first data table in the new system; For each non-first batch of data to be migrated, the non-first batch of data to be migrated is read from the old system into one or more second data files, and then the data to be migrated from the one or more second data files is imported into multiple second data tables in the new system.

2. The method according to claim 1, characterized in that The multiple batches are three batches, namely a first batch, a second batch and a third batch.

3. The method according to claim 2, characterized in that The data to be migrated is divided into multiple batches based on the historical transaction information, including: Classify the to-be-migrated data of the customer identification whose historical transaction volume is less than or equal to the first set threshold into a first batch; Classify the data to be migrated of the customer identifier whose historical transaction volume is greater than the first set threshold and whose transaction category accounts for a proportion of the total transaction categories exceeding the second set threshold into a second batch; The data to be migrated of the customer identifiers whose historical transaction volume is greater than the first set threshold and whose transaction categories account for a proportion of the total transaction categories that does not exceed the second set threshold are divided into a third batch.

4. The method according to claim 1, characterized in that: Before reading the first batch of data to be migrated from the old system into a first data file and then importing the data to be migrated from the first data file into a first data table in the new system, the method further includes: The whitelist configuration of the first batch of customer identifiers in the old system is switched from normal processing to stop processing, so that the old system stops processing the transaction business of the first batch of customer identifiers.

5. The method according to claim 1, characterized in that After reading the first batch of data to be migrated from the old system into a first data file, and then importing the data to be migrated from the first data file into a first data table in the new system, the method further includes: The whitelist configuration of the first batch of customer identifiers in the new system is set to normal processing, so that the new system can normally process the transaction business of the first batch of customer identifiers.

6. The method according to claim 4 or 5, characterized in that: After reading the first batch of data to be migrated from the old system into a first data file, and then importing the data to be migrated from the first data file into a first data table in the new system, the method further includes: forwarding the transaction business of the first batch of customer identifications to the new system for processing, and detecting the processing results of the transaction business of the first batch of customer identifications by the new system; For the transaction business with abnormal processing results, the data corresponding to the customer identification of the transaction business or the data of the customer identification of the first batch is migrated back from the new system to the old system; The whitelist configuration of the customer identifications that have been migrated back from the old system is switched from stop processing to normal processing, so that the old system can normally process the transaction business of the customer identifications that have been migrated back; the whitelist configuration of the customer identifications that have been migrated back from the new system is switched from normal processing to stop processing, so that the information system stops processing the transaction business of the customer identifications that have been migrated back.

7. The method according to claim 6, characterized in that For a transaction business with an abnormal processing result, the data corresponding to the customer identifier of the transaction business or the data of the first batch of customer identifiers are migrated from the new system to the old system, including: For transactions with abnormal processing results, data corresponding to the customer identification of the transaction or data of the customer identification of the first batch is read from the new system into a third data file, and then the data in the third data file is imported into the old system.

8. The method according to claim 2, characterized in that: Reading the non-first batch of data to be migrated from the old system into one or more second data files, and then importing the data to be migrated from the one or more second data files into a plurality of second data tables, comprises: For the second batch of data to be migrated, obtain a correspondence between the customer identifier and the second data file and a correspondence between the second data file and the second data table, and determine them as a first correspondence and a second correspondence, respectively; reading the second batch of data to be migrated from the old system into a plurality of second data files based on the first corresponding relationship; The data to be migrated is imported from each of the second data files into a second data table corresponding to the new system based on the second corresponding relationship.

9. The method according to claim 2, characterized in that: Reading the data to be migrated that is not the first batch from the old system into one or more second data files, and then importing the data to be migrated from the one or more second data files into a plurality of second data tables, comprises: For each non-first batch of data to be migrated, obtaining a correspondence between the customer identifier and the second data table, and determining it as a third correspondence; wherein the non-first batch is the second batch or the third batch; Reading the non-first batch of data to be migrated from the old system into a second data file; Splitting the second data file into a plurality of sub-data files based on the third corresponding relationship; wherein the sub-data files correspond to the second data table one by one; Import the data to be migrated from each of the sub-data files into the second data table corresponding to the new system.

10. The method according to claim 9, characterized in that Before obtaining the correspondence between the customer identifier and the second data table, the method further includes: The whitelist configuration of the customer identifiers of the non-first batch in the old system is switched from normal processing to stop processing, so that the old system stops processing the transaction business of the customer identifiers of the non-first batch.

11. The method according to claim 9, characterized in that After importing the data to be migrated from each of the sub-data files into the second data table corresponding to the new system, the method further includes: The whitelist configuration of the customer identifiers of the non-first batch in the new system is set to normal processing, so that the new system can normally process the transaction business of the customer identifiers of the non-first batch.

12. The method according to claim 10 or 11, characterized in that: After importing the data to be migrated from each of the sub-data files into the second data table corresponding to the new system, the method further includes: forwarding the transaction business of the customer identifier of the non-first batch to the new system for processing, and detecting the processing result of the transaction business of the customer identifier of the non-first batch by the new system; For a transaction business with an abnormal processing result, the data corresponding to the customer identifier of the transaction business is migrated back from the new system to the old system; The whitelist configuration of the customer identifications that have been migrated back from the old system is switched from stop processing to normal processing, so that the old system can normally process the transaction business of the customer identifications that have been migrated back; the whitelist configuration of the customer identifications that have been migrated back from the new system is switched from normal processing to stop processing, so that the information system stops processing the transaction business of the customer identifications that have been migrated back.

13. The method according to claim 12, characterized in that For a transaction business with an abnormal processing result, the data corresponding to the customer identifier of the transaction business is migrated from the new system to the old system, including: For transactions with abnormal processing results, data corresponding to the customer identification of the transaction or data of the customer identification of the first batch is read from the new system into a fourth data file, and then the data in the fourth data file is imported into the old system.

14. A data migration device, characterized in that: include: A historical transaction information acquisition module is used to acquire historical transaction information corresponding to multiple customer identifiers; wherein the historical transaction information includes historical transaction volume and transaction category within a set period of time; A batch division module, used to divide the data to be migrated into multiple batches based on the historical transaction information; wherein the data to be migrated carries the customer identifier, and the data to be migrated with the same customer identifier is divided into the same batch; A first batch data migration module, for reading the first batch of data to be migrated from the old system into a first data file, and then importing the data to be migrated from the first data file into a first data table in the new system; The non-first batch data migration module is used to read the non-first batch of data to be migrated from the old system into one or more second data files, and then import the data to be migrated from the one or more second data files into multiple second data tables in the new system.

15. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data migration method according to any one of claims 1 to 13 when executing the computer program.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data migration method as described in any one of claims 1 to 13 is implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the data migration method according to any one of claims 1 to 13.

Images